A SOFT Way for OpenFlow
Interoperability Testing
Maciej Kuźniar, Peter Perešini, Marco Canini†,
EPFL
Daniele Venzano, Dejan Kostić‡
†TU Berlin/T-Labs
‡IMDEA Networks
1
Software-Defined Networking (SDN)
Third-party
Controller
control
program
2
Interoperability at Deployment Time
Release
OpenFlow
One OpenFlow API
specification…
program
Are OF switches interoperable?
OpenFlow
messages
Interop is critical for the success of SDN
3
Inconsistency - Example
FlowMod message
1. Modify VLAN ID to 4097
2. Forward packet
Network in 3 different states
Which state is assumed by the controller?
Switch IWhere are packets
Switch II forwarded?Switch III
1. Responds with an
error message
1. Trims VLAN
value to 12 bits
2. Installs the rule
1. Silently drops
packets
4
Interop: How Hard Can It Be?
OpenFlow
messages
Inputs
OpenFlow
interface
OF Switch
OpenFlow Agent
Likely
source of
OpenFlow
interop
issues
OS
Flow Table
Packets
Hardware
Hardware
ASIC Abstraction
switch chipLayer
This work: Finding differences correctness
is formally
between
OpenFlow
Agent implementations
“Forwarding”
interface
verified
5
OpenFlow Software Agent
Switch software is not provably correct 
Specifications
• Rapid flux (3 revisions in ~ 1 year)
• Ambiguities
Specifications  Implementation
• Implementation freedom
• Vendors may not follow the specs
Testing, testing and testing…
6
·
·
·
·
·
Topology discovery (LLDP method)
Layer 2 Ethernet/VLAN path (circuit) provisioning (primary and backup)
Layer 3 (IP) learning (shortest path primary and backup path)
Layer 3 (IP) load balancing
Enabling multi-controller connectivity using FlowVisor to slice the network
Interoperability Event
Each one of these applications requires the switches to support the OpenFlow v1.0 protocol.
• Gather various vendors
• Hook up switches and controllers
• Create and run test cases
• See what breaks and fix it
[ONF Interop
March ’12]
TestingWG,
at the Interoperability
Event
• Very high manual effort
• Test cases are not exhaustive
• It is not a one time thing
8
7
Automating Interop Testing
Insight:
systematically crosscheck OF implementations
8
The 10,000 foot view
Test inputs
OF Agent 1
OF Agent 2
Input-driven
execution
Problem I: What inputs should we use?
Observable
behaviors
Inconsistency!
9
Symbolic message
65535
...
...
25
1
port
24
Symbolic Execution
port = ∗
port < 25
yes
port < 25?
no
Problem II: Path explosion
port ≥ 25
∧
port = CTRL
Forward
port == CTRL?
yes
Send to CTRL
no
port ≥ 25
∧
port ≠ CTRL
Send ERROR
10
Challenges
• Manage test inputs and coverage efficiently
• Capture behaviors
• Avoid simultaneous access to all code
11
SOFT
(Systematic OpenFlow Testing)
OF Agent 1
OF Agent 2
• Automated solution to interop testing
• Systematic code coverage
• No simultaneous access to all agents
?
12
FLOW
MOD
* ** ** **
1.0
STAT
REQ
LEN2
1.0
LEN1
Structuring Inputs
*C3 *C1* * C2
*
Further reductions
• Some messages are independent
• Many inputs are entirely concrete
• Small number of messages
• Concrete values at cost of coverage
13
Benefit of Concretizing
Coverage
50%
50%
40%
40%
Concrete
30% Actions
Concrete
Match
28h
20%
Fully
Symbolic 30%
20%
10%
10%
0%
0%
0
1
2
Time [h]
3
4
14
Capturing Behaviors
Externally observable outputs
• OpenFlow reply messages
• Data plane packets
• Normalize harmless nondeterminism (e.g., Buffer IDs)
Internal state changes affect successive inputs
• Use concrete probe packets
15
Example
Agent 1
Agent 2
16
Finding Inconsistencies
Agent 1
Agent 2
17
65535
...
...
25
24
port
1
Finding Inconsistencies
Agent 1:
Agent 2:
Is there an input that
No
false
positives
causes two distinct behaviors?
18
Limitations
• Short sequences of inputs
• Unable to find problems with a complex state
• Is an inconsistency harmless?
• Can it affect the controller?
• How to test all initial configurations?
• Agent’s behavior depends on initial config
19
Prototype & Evaluation
• SOFT prototype built on top of Cloud9/Klee
• Compared
• OpenFlow 1.0 Reference Switch (55k LoC)
• Open VSwitch 1.0.0 (80k LoC)
• Input Sequences containing 1 - 4 messages
20
Does SOFT Work?
Found 7 classes of inconsistencies:
Packets dropped when action is invalid
Different ports considered invalid
Lack of error messages
Different order of message validation
Silently ignored statistics request
Missing features
Switch terminates with an error
Mostly related
to message
validation
Result of underspecification
• No expected behavior in the specification
• Inconsistent interpretation of the specification
21
Summary
SOFT automates interoperability
testing of OpenFlow Agents
Also useful for:
• regression testing
• specification improvements
22