ITU-T Workshop on
IP Traffic Flow Measurement
(Geneva, Switzerland, 24 March 2011)
Cisco experiences of IP traffic flow
measurement and billing with NetFlow
Benoit Claise,
Distinguished Engineer, Cisco
Geneva, 24 March 2011
What is NetFlow?
NetFlow
Records
export
Cache
Collector
Over UDP
or SCTP
Traffic
What is NetFlow?
NetFlow is used for traffic monitoring, security
analysis, capacity planning and billing
Billing is just a few % of our customers, mainly for charge
back within enterprise network (not between service
providers)
NetFlow = a exporting protocol: NetFlow v5, 7, 8, 9
(RFC3954), and IPFIX (RFC5101/RFC5102)
NetFlow v9 and IPFIX work with a template based mechanism
Advantage: extensibility, just need to add new Information Element
NetFlow = a metering process: Flexible NetFlow
Advantages: cache and export content flexibility
User selection of flow keys
User definition of the records
Flexible NetFlow:
Potential Key Fields
Flow
IPv4
Sampler ID
IP (Source or
Destination)
Payload Size
IP (Source or
Destination)
Payload Size
Prefix (Source or
Destination)
Packet Section
(Header)
Prefix (Source or
Destination)
Packet Section
(Header)
Mask (Source or
Destination)
Packet Section
(Payload)
Mask (Source or
Destination)
Packet Section
(Payload)
Minimum-Mask
(Source or
Destination)
TTL
Minimum-Mask
(Source or
Destination)
DSCP
Dest VLAN
Protocol
Options
bitmap
Protocol
Extension Headers
Dot1q VLAN
Fragmentation
Flags
Version
Traffic Class
Hop-Limit
Dot1q priority
Fragmentation
Offset
Precedence
Flow Label
Length
Option Header
Next-header
Identification
DSCP
Header Length
TOS
Header Length
Version
Direction
Interface
Input
Output
Layer 2
Source VLAN
Source MAC
address
Destination
MAC address
Total Length
IPv6
Payload Length
Flexible NetFlow:
Potential Key Fields
Routing
Transport
src or dest AS
Destination Port
TCP Flag: ACK
Peer AS
Source Port
TCP Flag: CWR
Traffic Index
ICMP Code
TCP Flag: ECE
Forwarding
Status
ICMP Type
TCP Flag: FIN
IGMP Type*
TCP Flag: PSH
TCP ACK Number
TCP Flag: RST
TCP Header Length
TCP Flag: SYN
TCP Sequence Number
TCP Flag: URG
IGP Next Hop
BGP Next Hop
Input VRF
Name
Application
Application ID*
Multicast
Replication
Factor*
RPF Check
Drop*
Is-Multicast
TCP Window-Size
UDP Message Length
TCP Source Port
UDP Source Port
TCP Destination Port
UDP Destination Port
TCP Urgent Pointer
*: IPv4 Flow
only
Flexible NetFlow:
Potential Non-Key Fields
Counters
Timestamp
IPv4
IPv4 and IPv6
Bytes
sysUpTime First
Packet
Total Length
Minimum (*)
Total Length
Minimum (**)
Bytes Long
sysUpTime First
Packet
Total Length
Maximum (*)
Total Length
Maximum (**)
Bytes Square Sum
TTL Minimum
Bytes Square Sum Long
TTL Maximum
Packets
Packets Long
Plus any of the potential “key” fields: will be the value from
the first packet in the flow
(*) IPV4_TOTAL_LEN_MIN,
IPV4_TOTAL_LEN_MAX
(**)IP_LENGTH_TOTAL_MIN,
IP_LENGTH_TOTAL_MAX
Performance
Limited Resources in Router
Don’t enable all flow keys
The routers still have to route packets
NetFlow for Billing: Experience
Issue: Can we use Sampled NetFlow for
billing?
Estimation Accuracy
(PLT_NZIX1, S24D00, Cisco, f=5%
#Packets Nf
Huge amount of data,
must sometimes deal
with sampled NetFlow,
i.e. 1 out of N packets,
depending on the
platform
Packet Sampling for Flow
Accounting: Challenges
and Limitations,
Tanja Zseby,
Thomas Hirsch,
Benoit Claise, PAM 2008
Mean Packet Size µf
Issue: Can we use Sampled NetFlow
for billing?
Square sum of bytes available in Flexible NetFlow
Not used in practice, not even by the collectors!
Customers afraid of legal issues with sampling
along with a billing service
Destination Sensitive Billing Proposal
(many years ago)
1. BGP routing updates
2. Go through a table-map statement
3. table-map calls a route-map
Customer
AS 192
I-BGP
Forwarding Information Base
Prefix
AS=196
prefix one
prefix two
Traffic-index
E-BGP
traffic
trafficindex
index==11
traffic index = 2
ISP 1
$5.00 per 100 MB
4. route-map’s criteria:
if criteria 1
-> traffic-index = 1
if criteria 2
-> traffic-index = 2
E-BGP
Accounting
AS=193
ISP 2
$7.00 per 100 MB
BGP Policy Accounting Principles
Allows to classify packets based on
IP access lists,
BGP community list
to characterize the exit points, where each exit point
would set an specific community
BGP AS paths
Issue: What about the Returning
Packets?
The Customer
FTP Request
The ISP
ISP 1
$5.00 per 100 MB
Who should pay for the 100
MB back?
Destination Sensitive Billing
requires also source lookup
(Source Sensitive Billing)
100 MB back
ISP 2
$7.00 per 100 MB
Issue: What about the Returning
Packets?
The Customer
FTP Request
The ISP
ISP 1
$5.00 per 100 MB
Lookup:
• On the outgoing packets
(on the packets coming
back)
• On the source
• Same selection criteria
100 MB back
ISP 2
$7.00 per 100 MB
Issue: BGP Asymmetry Problem
The Customer
in Europe
FTP Request
Will charge the 10 Meg
as if they were directly
coming from the US!!!
The ISP
ISP 1 in Asia
ISP 2 in US
100 MB back
Issue: BGP Asymmetry Problem
The source lookup is based on the
route the router would take to reach
the source!
Too Many Issues
Destination Sensitive Billing requires Source
Sensitive Billing
BGP asymmetry problem
Only the traffic following the BGP routes will
be accounted
What if local policies outside of BGP?
Limited amount of buckets in the Destination
Sensitive Billing
Doesn’t scale: too many entries
Performance issues
Entire NMS solution to be put in place
Destination Sensitive Billing
Conclusion/feedback from customers:
too many issues
not realistically deployable -> back to some sort
of flat rate
Benoit’s concern:
If we bill per AS-PATH and each AS get a piece of
the pie, people will create new AS and try to
attract traffic
Bad for the internet performance
ITU-T Workshop on
IP Traffic Flow Measurement
(Geneva, Switzerland, 24 March 2011)
Cisco experiences of IP traffic flow
measurement and billing with NetFlow
Benoit Claise,
Distinguished Engineer, Cisco
Geneva, 24 March 2011