\chapter{Numerical Results}

\section{Implementation}
We propose OFBench, an automatic testing framework integrated
with Ryu controller \footnote{Ryu, version 4.0, https://osrg.github.io/ryu/} and Ostinato
traffic generator \footnote{Ostinato, version 0.7.1, http://ostinato.org/}.
The architecture of OFBench, shown in Figure \ref{fig:OFBench_arch}, is a
controller-agent architecture. Controlled by the controller, each agent
is capable of generating traffic, capturing and parsing packets.
The test cases of OFBench are written in scripts for
basic OpenFlow operations and test results analysis.

\begin{figure}[htb]
    \begin{center}
        \includegraphics[width=0.8\textwidth]{OFBenchArchitecture.png}
        \caption{OFBench architecture}
        \label{fig:OFBench_arch}
    \end{center}
\end{figure}

\section{Experiment setup}
Five test cases are developed with test scripts for
the OFBench framework. The available configurations of the test cases consist of frame
size, transmission rate, and some specific values. The available frame sizes
are 64, 128, 256, 512, and 1024 bytes, and the available transmission
rates are ranging from 64 Kbps to 1Gbps. In our experiment, we use UDP traffic
for the testing of all test cases. We conduct the experiments with three physical PCs.
The operation systems and specifications of those PCs are shown in Table
\ref{table:component_information}.

\begin{center}
    \begin{table}[htb]
        \centering
        \begin{tabular}{| l | l | l | l | l |}
            \hline
            Component & Core & Clock rate & OS & Kernel \\ \hline
            Controller & 4 & 3.3GHz & Ubuntu 14.04 & 4.2.0-27 \\ \hline
            Host & 4 & 3.2GHz & Ubuntu 14.04 & 4.2.0-27 \\ \hline
            Switch & 2 & 3.1GHz & Ubuntu 14.04 & 3.13.0-24 \\ \hline
        \end{tabular}
        \caption{OS and hardware specifications for components}
        \label{table:component_information}
    \end{table}
\end{center}

We use the above environment to test four DUTs. The DUTs details are shown in
Table \ref{table:switch_spec}.

\begin{center}
    \begin{table}[htb]
        \centering
        \begin{tabular}{| P{2.5cm} | l | l | l | l | l | l |}
            \hline
            Switch & CPU &  Core & Clock rate & Memory & Buffer & OS version \\ \hline
            Pica8 P-3290 & MPC8541 & 1 & 1 GHz & 512 MB & 4 MB & v2.6.1 \\ \hline
            Pica8 P-3297 & P2020 & 2 & 1.33 GHz & 2 GB & 4 MB & v2.6.1 \\ \hline
            Edge-corE AS4610-30T & ARM Cortex A9 & 2 & 1 GHz & 2 GB & N/A & v2.6.4 \\ \hline
            Centec V350 & e500v2 & 1 & 533 MHz & 2 GB & N/A & v3.1(11), 1.alpha \\ \hline
        \end{tabular}
        \caption{Hardware switch specifications}
        \label{table:switch_spec}
    \end{table}
\end{center}

\section{Experimental Results}
In this section, we present ours five performance metrics: action time, pipeline
time, packet-in rate, packet-out rate, pipeline gain, and timeout accuracy. These
metrics try to distinguish the switches implementation for features.

Each metrics are evaluated based on the proposed five test cases. And the statistic results are
collected with five rounds of testing.

\subsection*{Action Time}
We try to find out the relationship of action time with different combinations of
actions. The tested actions are based on a set of packet modifications ranging from layer two to four.
The test case is conducted based on six different actions of packet modification.
The results of action time are shown in Figure \ref{fig:action_time_result}.
The six modifications of packet are ordered by source IP address,
destination IP address, source Ethernet address, destination Ethernet address,
UDP source port and UDP destination port respectively.
For each testing scenario, we generate traffic at the rate of 1,000 packet-per-second (pps) with a total of 10,000 packets.

Based on the results shown in Figure \ref{fig:action_time_result},
the result for hardware switch Centec V350 is nonavailable due to the mirror and action does not execute properly.
For the result, The software switch has the lower action time which ranging from 2 to 3 microseconds.
However, the action times for hardware switches have at least 100 microseconds.
If the implementation of action is pure hardware based, the action time possible be nano seconds.
But these results and the CPU specification are in direct proportion.
According to the specifications, the Open vSwitch has the highest clock rate, Pica8 P-3297 followed,
Pica8 P-3290 and Edge-corE AS4610 final.
Thus, we think the implementation of action is software based on switches.

Otherwise, the variation of action time is small due to the different number of modifications.
Thus, we conduct this test case on white box environment. We use the kernel trace tool (ftrace)
to record the execution time of set\_ip\_addr function for Open vSwitch. The result of white box measurement
approximates 0.6 microseconds. And the first modification action has the highest time cost.
This result explains the reason for the small variation.

%We think the timing of Apply-Action may be the reason.
%According to the specification, the Apply-Action should be applied immediately. However, it may not be applied
%immediately in the DUTs in this situation. There are two output actions and some modifications
%in one test. One of the output action is the mirror and carried by Apply-Action. If there
%are two modifications, we denote the timings of modification execution as $T_1$ and $T_2$.
%Furthermore, the outputs of Apply-Action and Write-Action are denoted as $\alpha$ and $\beta$.
%The action time is not affected by the number of modifications when the order of execution
%is $T_1$, $T_2$, $\alpha$, and $\beta$.

\begin{figure}[htb]
    \begin{center}
        \includegraphics[height=0.4\textheight]{ActionTimeResult.png}
        \caption{Action Time}
        \label{fig:action_time_result}
    \end{center}
\end{figure}

\subsection*{Pipeline Time}
For our pipeline time test case, the results are unavailable for Pica8
switches due to the Apply-Action is not working properly. The hardware switches can not
mirror packets between the pipeline process. Thus, we only have the result for
Open vSwitch with different frame sizes ranging
from 64 to 1024 bytes. The pipeline time measured is in the range of 1.1 to 2 microseconds
with the positive correlation of frame size.

\subsection*{Buffer Size and Packet-in/out}
For the buffer size test case, we try to measure the Packet-in rate,
Packet-out rate, and buffer size within one test. Therefore, we fix the
transmission rate of 1Gbps with different frame sizes for the measurement of Packet-in rate.
The results of Packet-in and Packet-out rate are shown in Figure \ref{fig:packet_in_out_rate_result}.
The results be collected at the beginning of the testing are discarded. This is mainly
because the traffic generator takes five to ten seconds for stabilizing the testing traffic.
According to our results, the rate of Packet-in for Open vSwitch is approximately $20$ times lower than that of the hardware switches under test. It indicates the software switch has poor handling capability with the small frame
size for Packet-in operation.

For the Packet-out rate, the Pica8 switches have poor handling capability with the small frame
sizes, especially the 64 bytes. However, the measurement results for Open vSwitch are different.
It is working well on the Packet-out operation with the frame size of 64 bytes. And the throughputs
of Packet-out for Open vSwitch are better than those observed in other switches under test.

The measurement results of buffer size for switches under test are shown in Table \ref{table:buffer_size_result}.
There are some error messages generated from the DUTs during the testing. The errors,
identified as buffer unknown for Packet-out
messages lead to the failure of Packet-out operation. The test results indicating packet loss occurred at the DUTs. Finally, we mark the buffer size of the DUT is not available (N/A) when
the number of errors is greater than the number of Packet-out messages sent from the controller.

In Table \ref{table:buffer_size_result}, only the buffer sizes tested with frame size of 64-byte are shown.
For other frame size, the buffer sizes of the switches are not available. We suspect the anomaly behavior of DUTs is due to the CVE-2016-2074 \cite{OVS_CVE_MPLS_buffer_overflow} issues for the
Open vSwitch implementation.

\begin{figure}[htb]
    \begin{center}
        \includegraphics[height=0.3\textheight]{PacketInOutRateResult.png}
        \caption{Packet-in and Packet-out rates}
        \label{fig:packet_in_out_rate_result}
    \end{center}
\end{figure}

%TODO: explain this table in more detail since it is confusing
\begin{center}
    \begin{table}[htb]
        \centering
        \begin{tabular}{| l | l | l | l | p{2.5cm} | l |}
            \hline
            Frame size(bytes) & Open vSwitch & Pica8 P-3290 & Pica8 P-3297 &
                Edge-corE AS4610 & Centec V350 \\ \hline

            64      & 13504 & 8256  & 16192 & 32768 & 342464 \\ \hline
            128     & N/A   & N/A   & N/A   & N/A   & N/A \\ \hline
            256     & N/A   & N/A   & N/A   & N/A   & N/A \\ \hline
            512     & N/A   & N/A   & N/A   & N/A   & N/A \\ \hline
            1024    & N/A   & N/A   & N/A   & N/A   & N/A \\ \hline
        \end{tabular}
        \caption{Buffer size with different frame size for 1Gbps traffic}
        \label{table:buffer_size_result}
    \end{table}
\end{center}

\subsection*{Pipeline Gain}
For the gain of table pipeline, the test case is conducted to go through from 20 to 254 tables.
with transmission rates 1024 Mbps and the duration is 10 seconds. Considering the warm up
for traffic generator, we use the masked entry to filter out the traffic from 0 to 5 seconds.
The results are shown in Figure \ref{fig:pipeline_performance_result}, the software switch
have poor performance on multiple table pipeline.
For hardware switches, the number of tables is inversely proportional to pipeline gain.
This indicates the hardware switches have the larger idle time when multiple tables pipelining.
Otherwise, all switches have the black-hole situation on 70 tables which indicates the number of available tables
for switch just only 60 tables. And the results for Centec V350 switch is not available due to the switch only
have a single table.

%the busy ratio increases at higher transmission rates.
%At the rate of 1024 Mbps, the Pica8 switches reach the busy ratio of $60\%$. However, the error messages occurred at the Open vSwitch while
%handling the large traffics in the same test. The sent and received packets mismatched as the
%Open vSwitch operated under heavy loading. This issue leads to the unavailable of busy ratio calculation.

\begin{figure}[htb]
    \begin{center}
        \includegraphics[height=0.4\textheight]{PipelinePerformanceResult.png}
        \caption{Pipeline gain with different number of tables}
        \label{fig:pipeline_performance_result}
    \end{center}
\end{figure}

\subsection*{Timeout Accuracy}
%TODO: need to discuss
In the testing of timeout accuracy, we set the idle-timeout to five seconds and
vary the hard-timeout from three to seven seconds. The traffic is generated at the
rate of 10K pps with 64 bytes frame size.
The results are shown in
Table \ref{table:timeout_accuracy_result}.
The deviation of idle-timeout for hardware switch
is around $\pm20\%$.
In our test case, we make the switch to swap idle-timeout and hard-timeout flow entries
This operation may cause the switch can not properly handle the idle-timeout properly,
and result in the larger deviation. Eventually, the Pica8 switches may not reset the timer
for idle-time properly when the idle-timeout flow entry matched.

\begin{center}
    \begin{table}[h]
        \centering
        \begin{tabular}{| l | r | r |}
            \hline
            Switch          & $Acc_{\var{idle-timeout}}$ & $Acc_{\var{hard-timeout}}$ \\ \hline
            Open vSwitch    & 2\%     & 0\% \\ \hline
            Pica8 P-3290    & -16.24\%   & 4.7\% \\ \hline
            Pica8 P-3297    & 24.64\%     & 3.3\% \\ \hline
            Edge-corE AS4610    & 10\%     & 0\% \\ \hline
            Centec V350    & 17.64\%     & 1.66\% \\ \hline
            \hline
        \end{tabular}
        \caption{Timeout accuracy}
        \label{table:timeout_accuracy_result}
    \end{table}
\end{center}
