\chapter{Introduction}
Software Defined Networking (SDN) is an emerging network architecture. The major
difference between SDN and the traditional network is the decoupling of the
control and data plane. Therefore, in the operation of SDN, the data plane needs
to communicate with the control plane through a specific communications
protocol. This change brings the operations of the control plane in the foreground as
part of the data packet processing. OpenFlow is a major standard for SDN which
is managed by Open Network

Foundation (ONF) \cite{ONF}, is a communications protocol used to manipulate the
data plane operations by the controller. Therefore,
another distinct difference between OpenFlow and the traditional network is the
stateful switching rather than the stateless counterpart.

As the packet arrives at the OpenFlow switch, selected tuples in the packet
header fields are compared with the rules in the flow table. If there is no
match, the packet is treated as a new flow and table-miss operation is
triggered. Depending on the configuration, the packet can be either simply
dropped or an OpenFlow message is sent to the controller waiting for further
action. The controller may insert a new flow entry in the flow table with action
for the matched packet. The flow entry is the record of the data packet, which
contains information such as packet headers, counters, and instructions
\cite{OpenFlow}. The process of the OpenFlow transaction can be categorized into
two parts, one is data-plane-trigger-control-plane (D2C), the other is
control-plane-trigger-data-plane (C2D). The whole process is abbreviated as
D2C2D. As such, OpenFlow adds some extra data structures and operations to the
data plane. These changes make the OpenFlow data plane (OFD) different from the
traditional data plane (TD) .

Currently, there is no performance test standard exist for the OpenFlow switch.
In the past, the performance test only focus on the traditional TD part.
However, we think the D2C2D process should also be included along with
the OFD as part of the testing suite for OpenFlow switch. Cause the packet
processing depends on the flow entry which including some actions. The D2C2D
process will affect the flow entry setup when the new type flow coming.
Moreover, the OFD process will affect the operation of flow entry in the
OpenFlow switch for the existed flows. Due to above mentioned, we think
the OpenFlow performance testing must be included TD, OFD, D2C, and C2D. The
main reason is that the D2C2D process is the core of the operation in both data
plane and control plane in the OpenFlow switch. Moreover, the D2C2D process and
OFD affect the packet processing in OpenFlow switch. So we think the OpenFlow
performance testing must be included TD, OFD, D2C, and C2D.

In our survey, there are some open-source tools such as OFTest \cite{OFTest} and
Ryu certification \cite{Ryu_cert} exist for the OpenFlow switch conformance
test. These tools only determine whether the switch fit the interoperability of
OpenFlow protocol or not. As for commercial tools, there is no one except the
white paper proposed by Spirent for OpenFlow performance testing
\cite{Spirent_whitepaper}. The white paper only addresses some test cases and
offers some concepts to evaluate the switch. There are many research works of literature
address the issues in this topic as well. However, most of them focus on TD
testing without addressing the others. The authors in OFLOPS \cite{OFLOPS}
discuss the test of OFD and C2D parts without covering all test matrices of these
two parts.

In this thesis, we propose thirteen test cases based on the white paper
\cite{Spirent_whitepaper} from Spirent . The test cases, including items such
as flow action, pipeline, Packet-in, Packet-out, and Flow-mod, address the most
critical parts of OFD, D2C, and C2D testing. Among these thirteen test cases,
there are seven test cases classified as non-trivial. And we are going to focus
on the development of test tools for these six out of seven non-trivial test
cases.

It is difficult to get the internal parameters from the device-under-test (DUT)
for evaluation in these non-trivial test cases due to black-box testing. For
example, the action time is hard to measure because the processes in OpenFlow
consist of many steps. Normally, we can only measure the end-to-end delay time
without the exact breakdown of individual action time where many variables involved.
And the idle timeout of flow entry is hard to measure the accuracy by traffic,
cause the idle timeout will be reset when packet coming and matched. Therefore,
we propose two methodologies to address these issues. The first one, named as
mirror-in-processing, is based on the Apply-Action instruction of OpenFlow to
mirror packets in the process. The system is capable of measuring the processing
time of each stage in the DUT based on the mirrored packets. So we can inference
some exact results. The second one, named as the masked entry, is based on the
priority, match fields and actions of flow entry to control the timer where two
flow entries can be synchronized for proper measurement.

We propose and design an automatic test framework based on controller-agent
architecture. The framework is capable of controlling remote agents to generate
and analyze networking traffic. The agents provide the analysis results to the
controller for test cases.
%We adopt the white-box test methodology to evaluate
%and ensure the accuracy of our black-box test cases. In white-box test, we
%modify and design specific codes in Open vSwitch. The goal is to measure and
%compare the test results with all test matrices based on the same test steps
%stated in the black-box test cases. Finally, these test cases are applied to
%the OpenFlow switches.

The remainder of this work is organized as follows. We introduce the backgrounds
of OpenFlow with emphasis on the performance related issues and related works
for testing in Chapter 2. The problem statement is provided in Chapter 3. In
Chapter 4, we introduce the proposed methodologies for the
mirror-in-processing and the masked entry. We further discuss our implementation
detail and experiment results in Chapter 5. Conclusions and future works are
finally presented in Chapter 6.
