Software-defined Measurement
Minlan Yu
University of Southern California
Joint work with Lavanya Jose, Rui Miao, Masoud Moshref,
Ramesh Govindan, Amin Vahdat
1
Management = Measurement + Control
• Accounting
– Count resource usage for tenants
• Traffic engineering
– Identify large traffic aggregates, traffic changes
– Understand flow characteristics (flow size, etc.)
• Performance diagnosis
– Why my application has high delay,
low throughput?
2
Yet, measurement is underexplored
• Measurement is an afterthought in network device
– Control functions are optimized w/ many resources
– Limited, fixed measurement support with NetFlow/sFlow
• Traffic analysis is incomplete and indirect
– Incomplete: May not catch all the events from samples
– Indirect: Offline analysis based on pre-collected logs
• Network-wide view of traffic is especially difficult
– Data are collected at different times/places
3
Software-defined Measurement
• SDN offers unique opportunities for measurement
– Simple, reusable primitives at switches
– Diverse and dynamic analysis at controller
– Network-wide view
Controller
Heavy Hitter
detection
1 1 (Re)Configure
Configure resources
resources
Change
detection
2 Fetch statistics
4
Challenges
• Diverse measurement tasks
– Generic measurement primitives for diverse tasks
– Measurement library for easy programming
• Limited resources at switches
– New data structures to reduce memory usage
– Multiplexing across many tasks
5
Software-defined Measurement
Data plane
Primitives
OpenSketch
(NSDI’13)
DREAM
(SIGCOMM’14)
Sketch-based
commodity switch
components
Flow-based
OpenFlow TCAM
Resource alloc Optimization w/ Provable
resource-accuracy bounds
across tasks
Dynamic Allocation
w/ Accuracy
estimator
OpenSource
NetFPGA + Sketch library
networks of 6
hardware switches
and Open vSwitch
Prototype
Software-defined Measurement
with Sketches
(NSDI’13)
7
Software Defined Networking
Controller
Configure devices and
collect measurements
API to the data plane (OpenFlow)
Fields
action counters
Src=1.2.3.4drop, #packets, #bytes
Rethink the abstractions for measurement
Switches
Forward/measure packets
8
Tradeoff of Generality and Efficiency
• Generality
– Supporting a wide variety of measurement tasks
– Who’s sending a lot to 23.43.0.0/16?
– Is someone being DDoS-ed?
– How many people downloaded files from 10.0.2.1?
• Efficiency
– Enabling high link speed (40 Gbps or larger)
– Ensuring low cost (Cheap switches with small memory)
– Easy to implement with commodity switch components
9
NetFlow: General, Not Efficient
• Cisco NetFlow/sFlow
– Log sampled packets, or flow-level counters
• General
– Ok for many measurement tasks
– Not ideal for any single task
• Not efficient
– It’s hard to determine the right sampling rate
– Measurement accuracy depends on traffic distribution
– Turned off or not even available in datacenters
10
Streaming Algo: Efficient, Not General
• Streaming algorithms
– Summarize packet information with Sketches
– E.g. Count-Min Sketch, Who’s sending a lot to host A?
Data plane
# bytes from
23.43.12.1
Hash1
Hash2
Hash3
Control plane
3
0
5
1
9
0
1
9
3
0
5
1
2
0
3
4
Pick min: 3
Query: 23.43.12.1
3
4
• Not general:Each algorithm solves just one question
– Require customized hardware or network processors
– Hard to implement every solution in practice
11
Where is the Sweet Spot?
General
Efficient
NetFlow/sFlow
(too expensive)
Streaming Algo
(Not practical)
OpenSketch
• General, and efficient data plane based on sketches
• Modularized control plane with automatic configuration
12
Flexible Measurement Data Plane
• Picking the packets to measure
– Hashes to represent a compact set of flows
• A set of blacklisting IPs
– Classify flows with different resources/accuracy
• Filter out traffic for 23.43.0.0/16
• Storing and exporting the data
– A table with flexible indexing
– Complex indexing using hashes and classification
– Diverse mappings between counters and flows
13
A three-stage pipeline
– Hashing: A few hash functions on packet source
– Classification: based on hash value or packets
– Counting: Update a few counters with simple calc.
Data Plane
pkt.
Hashing
# bytes from
23.43.12.1
Classification
Hash1
Hash2
Hash3
Counting
3
0
5
1
9
0
1
9
3
0
1
2
0
3
4
Build on Existing Switch Components
• A few simple hash functions
– 4-8 three-wise or five-wise independent hash functions
– Leverage traffic diversity to approx. truly random func.
• A few TCAM entries for classification
– Match on both packets and hash values
– Avoid matching on individual micro-flow entries
• Flexible counters in SRAM
– Many logical tables for different sketches
– Different numbers and sizes of counters
– Access counters by addresses
15
Modularized Measurement Libarary
• A measurement library of sketches
– Bitmap, Bloom filter, Count-Min Sketch, etc.
– Easy to implement with the data plane pipeline
– Support diverse measurement tasks
• Implement Heavy Hitters with OpenSketch
– Who’s sending a lot to 23.43.0.0/16?
– count-min sketch to count volume of flows
– reversible sketch to identify flows with heavy counts in
the count-min sketch
16
Support Many Measurement Tasks
Measurement
Programs
Building blocks
Line of Code
Heavy hitters
Count-min sketch;
Reversible sketch
Count-min sketch; Bitmap;
Reversible sketch
Count-min sketch;
Reversible sketch
Config:10
Query: 20
Config:10
Query:: 14
Config:10
Query: 30
Traffic entropy on Multi-resolution classifier;
port field
Count-min sketch
Config:10
Query: 60
Flow size
distribution
Config:10
Query: 109
Superspreaders
Traffic change
detection
multi-resolution classifier;
hash table
17
Resource management
• Automatic configuration within a task
– Pick the right sketches for measurement tasks
– Allocating resources across sketches
– Based on provable resource-accuracy curves
• Resource allocation across tasks
– Operators simply specify relative importance of tasks
– Minimizing weighted error using convex optimization
– Decompose to optimization problem of individual tasks
18
OpenSketch Architecture
Control Plane
measurement program
Heavy Hitters/SuperSpreaders/Flow Size Dist.
...
measurement library
CountMin
Sketch
Reversible
Sketch
Bloom
filter
SuperLogLog
Sketch
query
configure
report
Data Plane
pkt.
Hashing
Classification
...
Counting
Evaluation
• Prototype on NetFPGA
– No effect on data plane throughput
– Line speed measurement performance
• Trace Driven Simulators
– OpenSketch, NetFlow, and streaming algorithm
– One-hour CAIDA packet traces on a backbone link
• Tradeoff between generality and efficiency
– How efficient is OpenSketch compared to NetFlow?
– How accurate is OpenSketch compared to specific
streaming algorithms?
20
Heavy Hitters: false positives/negatives
• Identify flows taking > 0.5% bandwidth
OpenSketch requires less
memory with higher accuracy
21
Tradeoff Efficiency for Generality
In theory,
OpenSketch requires 6 times memory
than complex streaming algorithm
22
OpenSketch Conclusion
• OpenSketch:
– Bridging the gap between theory and practice
• Leveraging good properties of sketches
– Provable accuracy-memory tradeoff
• Making sketches easy to implement and use
– Generic support for different measurement tasks
– Easy to implement with commodity switch hardware
– Modularized library for easy programming
23
Dynamic Resource Allocation
For TCAM-based Measurement
SIGCOMM’14
24
SDM Challenges
Many Management
tasks
Controller
Heavy
Hitter
detection
Change detection
Heavy
Hitter
detection
Heavy Hitter detection
H
Dynamic Resource Allocator
1 1 (Re)Configure
Configure resources
resources
2 Fetch statistics
Limited resources (TCAM)
25
Dynamic Resource Allocator
• Diminishing return of resources
Recall
Recall=
detected true HH/all
– More resources make smaller accuracy gain
– More resources find less significant outputs
– Operators can accept an accuracy bound <100%
1
0.8
0.6
0.4
0.2
0
256 512
1024
Resources
2048
26
Dynamic Resource Allocator
• Temporal and spatial resource multiplexing
Recall=
detected true HH/all
– Traffic varies over time and switches
– Resource for an accuracy bound depends on Traffic
27
Challenges
• No ground truth of resource-accuracy
– Hard to do traditional convex optimization
– New ways to estimate accuracy on the fly
– Adaptively increase/decrease resources accordingly
• Spatial & temporal changes
– Task and traffic dynamics
– Coordinate multiple switches to keep a task accurate
– Spatial and temporal resource adaptation
28
Dynamic Resource Allocator
Controller
Heavy
Hitter
detection
Heavy
Hitter
Heavy Hitterdetection
detection
Estimated accuracy
Change detection
H
Estimated accuracy
Allocated resource
Allocated resource
Dynamic Resource Allocator
• Decompose the resource allocator to each switch
– Each switch separately increase/decrease resources
– When and how to change resources?
29
Per-switch Resource Allocator: When?
• When a task on a switch needs more resources?
Controller
Heavy Hitter detection
Detected HH:5 out of 20
Local accuracy=25%
A
B
Detected HH: 14 out of 30
Global accuracy=47%
Detected HH:9 out of 10
Local accuracy=90%
– Based on A’s accuracy (25%) is not enough
• if bound is 40%, no need to increase A’s resources
– Based on the global accuracy (47%) is not enough
• if bound is 80%, increasing B’s resources is not helpful
– Conclusion: when max(local, global) < accuracy bound
30
Per-Switch Resource Allocator: How?
• How to adapt resources?
– Take from rich tasks, give to poor tasks
• How much resource to take/give?
– Adaptive change step for fast convergence
– Small steps close to bound, large steps otherwise
Resource
Resource
1500
1500
1000
1000
Goal
Goal
MM
MM
AM
AM
AA
AA
MA
MA
500
500
00
00
100
100
200
300
200
300
Time(s)
400
400
500
500
31
Task Implementation
Controller
Heavy
Hitter
detection
Heavy
Hitter
Heavy Hitterdetection
detection
Estimated accuracy
Change detection
H
Estimated accuracy
Allocated resource
Allocated resource
Dynamic Resource Allocator
1 1 (Re)Configure
Configure resources
resources
2 Fetch statistics
32
Flow-based algorithms using TCAM
Current
36
***
26 0**
12 00*
01* 14
001
5
000
5
011
7
12
010
New
1** 10
10*
11*
111
101
2
0
100
5
5
2
110
3
• Goal: Maximize accuracy given limited resources
• A general resource-aware algorithm
– Different tasks: e.g., HH, HHH, Change detection
– Multiple switches: e.g., HHs from different switches
• Assume: Each flow is seen at one switch (e.g., at sources)
33
Divide & Merge at Multiple Switches
• Divide: Monitor children to increase accuracy
– Requires more resources on a set of switches
• Example: Needs an additional entry on switch B
26 0**
{A,B}
12 00*
{A,B,C}
Current: A:0**, B:0**, C:0**
{B,C}
New: A:00*, B:00*,01*, C:01*
01* 14
• Merge: Monitor parent to free resources
– Each node keeps the switch set it frees after merge
– Finding the least important prefixes to merge is the
minimum set cover problem
34
Accuracy Estimation: Heavy Hitter Detection
• Any monitored leaf with volume > threshold is a
true HH
• Recall:
– Estimate missing HHs using volume and level of counter
Threshold=10
76
With size 26
missed <=2 HHs
***
26 0**
12 00*
000
1** 50
01* 14
001
5
At level 2
missed <=2 HH
15 10*
011
7
12
010
11* 35
111
101
2
0
100
15
20
110
15
35
DREAM Overview
6) Estimate
accuracy
DREAM
SDN Controller
7) Allocate /
Drop
4) Fetch
counters
Task object n
Resource Allocator
Task object 1
• Task type (Heavy hitter, Hierarchical
heavy hitter, Change detection)
• Task specific parameters (HH threshold)
Prototype Implementation with DREAM
• Packet header field (source IP)
algorithms on Floodlight and Open vSwitches
• Filter (src IP=10/24, dst IP=10.2/16)
• Accuracy bound (80%)
1) Instantiate task
2) Accept/Reject
5) Report
3) Configure
counters
36
Evaluation
• Evaluation Goals
– How accurate are tasks in DREAM?
• Satisfaction: Task lifetime fraction above given accuracy
– How many more accurate tasks can DREAM support?
• % of rejected/dropped tasks
– How fast is the DREAM control loop?
• Compare to
– Equal: divide resources equally at each switch, no reject
– Fixed: 1/n resources to each task, reject extra tasks
37
Prototype Results
DREAM: High satisfaction for avg & 5th % of tasks
with low rejection
Mean
100
5th %
% of tasks
80
60
DREAM-reject
Fixed-reject
DREAM-drop
40
20
0
512
1024
2048
Switch capacity
4096
Equal: only keeps small tasks satisfied
Fixed: High rejection as over-provisions for small tasks
256 tasks (various task types) on 8 switches
38
Prototype Results
DREAM: High satisfaction for avg & 5th % of tasks
at the expense of more rejection
100
DREAM-reject
Fixed-reject
DREAM-drop
% of tasks
80
60
40
20
0
512
1024
2048
Switch capacity
4096
Equal & Fixed: only keeps small tasks satisfied
39
Control Loop Delay
Allocation delay is
negligible vs. other delays
Incremental saving lets reduce save delay
40
DREAM Conclusion
• Challenges with software-defined measurement
– Diverse and dynamic measurement tasks
– Limited resources at switches
• Dynamic resource allocation across tasks
– Accuracy estimators for TCAM-based algorithms
– Spatial and temporal resource multiplexing
41
Summary
• Software-defined measurement
– Measurement is important, yet underexplored
– SDN brings new opportunities to measurement
– Time to rebuild the entire measurement stack
• Our work
– OpenSketch:Generic, efficient measurement on sketches
– DREAM: Dynamic resource allocation for many tasks
42
Thanks!
43