Internet2 QoS:
Overview and Early Experiences
Russ Hobby <rdhobby@internet2.edu>
December 4, 2000
Campus Focused Workshop on Advanced Networks
How We Got Here (short version)
• Began chanting:
“enable advanced applications,…”
• Assessed requirements
• Recommended DiffServ
• Selected “Premium” service to meet
demands of loss/jitter sensitive apps
• Charted QBone initiative
• Specified QBone architecture
• Now proceeding to implement it and
tweak the architecture
architecture
deployment
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
2
Internet2 QoS Milestones1/2
Jun 1997
Jan 1998
Bay Workshop
QoS WG DiffServ
Recommendation
Early Internet2 QoS
requirements gathering
Jun 1998
Jan 1999
QBone CFP
QIG opened
RTP QIG
Evanston QIG
QBone initiative launched
Jun 1999
Jan 2000
BBop
QCon
QBone architecture draft (v1.0)
QUALIT
Houston I2/DOE
QBone Workshop
QBone architecture “last call”
Pittsburgh
QBone
BOF
QPS demonstrated
(nM demo)
Las Cruces QIG
APS Test
Program
launched
QBone Forum
launched
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
3
Internet2 QoS Milestones2/2
Jun 2000
Jan 2001
Toronto
QoS BOF
WG re-chartering
You are here!
SC2K QBone demo
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
4
IntServ/RSVP vs DiffServ
IntServ/RSVP
•Per-flow service state at every hop
•Scalability problems
•Focus on multipoint multicast
BB
BB
DiffServ
•Abstract/manage each cloud’s resources (BBs)
•Packets colored to indicate forwarding “behavior”
•Focus on aggregates not individual flows
•Policing
at edge to get services
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
5
DiffServ Overview
Applications contract for specific QoS profiles
• Policing at network periphery
• “Color” packets with a few simple, differentiated per-hop
forwarding behaviors (PHBs)
– Indicated in packet header
– Applied to PHB traffic aggregates
• PHBs + policing rules = range of services
DS domains contract with each other for
aggregate QoS traffic profiles
• Policing at cloud-cloud boundary
• Supports simple, bilateral business agreements
Exploits edge/core distinction for scalability
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
6
Example Service #1: Premium
Assurance: like a leased line
PHB: Expedited Forwarding (RFC 2598)
• EF in separate queue configured with minimum
departure rate
• Example mechanisms: strict priority, MDRR, WFQ
Policing: police to a specified peak rate
and drop out-of-profile packets; effectively
a leaky bucket with depth 1 MTU
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
7
Example Service #2:
Controlled Load
Assurance: network looks “lightly-loaded”
for conforming traffic
PHB: Assured Forwarding (RFC 2597)
• 4 independent AF classes
• 3 drop preference levels within each class
• Example mechanisms: WRED, WFQ
Policing: police to specified rate and burst
profile, remarking out-of-profile packets to
have higher drop probability
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
8
Example Service #3: CoS
Assurance: “better than Joe”
PHB: “drop the lower classes first ” (AF or
class selector PHBs)
Policing: could be based on anything
(e.g. higher priority for the CEO)
A.K.A.“Olympic” classes of BE service
(e.g. Gold, Silver, Bronze)
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
9
QBone Architecture
A Service: QBone Premium Service
• Built on Expedited Forwarding (EF) (RFC 2598)
• Assurance: near-zero loss & low, bounded jitter for
marked traffic conforming to a specified peak rate
– a.k.a. “virtual leased line”, “virtual wire”
Reservation Setup Protocol
• Now: long-lived, manual setup
• Proposed: SIBBS protocol between QBone domains;
RSVP end-to-end between hosts
QBone Measurement Architecture
• Uniform collection of QoS metrics
• Uniform dissemination interface
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
10
QBone E2E Picture
Campus
A
GigaPoP
A
Campus
B
Backbone
Campus
C
Key
X Kbps of QPS from
hither to………..yon
GigaPoP
B
Campus
D
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
11
Abilene Premium Service (APS)
Goal:
• Make APS a reference implementation of the QBone
architecture
Why?
• Catalyst: Enable wide-area QoS experimentation and
propagate Abilene’s experiences to larger community
• Safety Belt: Congestion is only one success
catastrophe away!
The Team:
• Cisco
• UCAID
• Qwest
• Indiana University
• I-TECs (NCNI, OARnet) • Nortel
• NLANR/NCNE
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
12
APS Participation
Participation Criteria
• Abilene connector status necessary but not sufficient
• QBone participation
• Capability of access router / Abilene edge card
Current Participants
• MAGPI (U. Penn)
• iCAIR
• PSC (Penn State)
• OARNet (Ohio State)
• ANL
• UIUC
• DOE Science GRID (peering
transit network)
Others in the wings
• NASA EOS
• Various international
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
13
Initial Engineering Plan (obsolete)
Sweetwater Midland
Odessa
Pecos
Measurement (Surveyor + SNMP + HTTP)
Edge Policing (“Firehose” CAR)
Manual Setup (Whiteboard + CLI)
EF Core Forwarding (MDRR)
EF Edge Forwarding (MDRR)
Automated Setup (BB)
Shaping (GTS)
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
14
APS Phase 1: “Sweetwater”
First, crawl…
• Ingress policing
• Measurement feedback
• Limited technical support
• No priority queuing yet, but still get benefit of Abilene’s
over-provisioning!
Goals
• Begin to establish operational practices for responding
to admissions requests
• Provide participants with the experience of sitting behind
a policer
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
15
APS Measurements
Goal: implement QBone measurement
architecture (collection + dissemination) for a
single QBone domain
Hoped-for Side Effects:
• Tools to ease implementation for others
• Adjustments to QMA based on implementation
experiences
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
16
Committed Access Rate (CAR)
Classifies traffic based on certain matching
criteria and meters it to a leaky bucket traffic
profile
Depending on metering result, different
actions applied (drop, transmit, set DSCP,…)
Syntax:
rate-limit {input | output} [access-group [rate-limit]
acl-index] bps burst-normal burst-max conform-action
action exceed-action action
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
17
CAR Experience
For the most part, CAR is exactly what the
DiffServ doctor ordered
However, there are some limitations…
• Performance
• Token bucket depth
• Classification of DiffServ aggregates
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
18
CAR Limitation 1: Performance
On E0 edge cards, no ASIC support for CAR
 there is a performance hit
pps @ 256 bytes
250000
200000
150000
no CAR
100000
CAR
50000
0
OC-12 POS
OC-3 POS
OC-12 ATM
qOC-3 ATM
* Preliminary E0 CAR performance figures under retest by ITEC-NC
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
19
Addressing CAR Performance
Out of our hands
The Plan:
• Load on access interfaces is still light, so performance
not really an issue for now
• Wait for E3 edge cards, which will have CAR in
hardware
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
20
Virtual Trunk vs.“Firehose”
Classification
What we want...
What we have now...
The good news: CAR can also classify by:
• qos-group (Cisco proprietary concept)
• With packets assigned to QoS groups through QoS
Policy Propagation via BGP (QPPB)
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
21
Abilene Architecture Limitation:
“Porous” Edge Problem
QoS theft problem with current architecture
MDRR (EF forwarding) on interior interfaces
easily subverted by unpoliced connectors
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
22
Current Engineering Plan
Sweetwater Midland
Odessa
Pecos
Measurement (Surveyor + SNMP + HTTP + WHOIS + traffic matrices)
Edge Policing (CAR + QPPB + PIRC filtering?)
Manual Setup (Whiteboard + CLI)
EF Core Forwarding (MDRR)
EF Edge Forwarding (MDRR)
Automated Setup (DBP-TE? + BB)
Shaping (GTS)
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
23
SC2000 Interdomain QoS Demo
November 6-9, 2000
Premium service over two wide-area paths
• LBNL-ESnet-Abilene-SCinet-Internet2 booth
• Stanford-CalREN2-Abilene-SCinet-Internet2 booth
Congestion induced at multiple points
CD-quality interactive audio application
shown with/without QoS
ESnet and Abilene QoS capabilities formed
nucleus of QBone
SC2000 Network Challenge Winner: "Most
Captivating and Best Tuned Award"
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
24
Interdomain Quality of Service Demo
Extreme
Summit1i
ALS
GE
Device
control Extreme
Summit1i
LBL-TB1
ANL-TB1
IPLS-QOS
Cisco 7505
Cisco 7505
Cisco 7505
OC-12
GE
10 Mbps
ESNet/DOE Science Grid
stan.calren2
OC-48 POS
Abilene Core
qsv.calren2
OC-12
LBNL
OC-3
Cisco 12008
CalREN2
OC-12
Cisco
12008
Abilene
OC-48
OC-12
I2-GW
Cisco 12008
CCRMA
OC-3
Audio
SND-GW
CCRMA
Audio
Access
Edge
Cisco 7505
Cisco 7505
Q-GW
OC-3
core-conf
Cisco
12008
SCinet
SmartBits
NetCom
Cisco 7505
Cisco 7505
Traffic Generators
X-Port
Crystallography
SmartBits 200
Network
Monitoring
GARA
Internet2 SC2000 Booth
Stanford University
= congestion + QoS control
* All links 100Mbps fast Ethernet (FE) unless otherwise noted
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
25
Deployment
Good News:
• DiffServ functionality in most modern routers
• Many hosts support QoS signaling
• Lots of isolated testbed trials
• Some partial backbone implementations
Bad News:
• Weak demand
• Turnkey solutions are a long way off
• Vendor implementations don’t always live up to hype
• HR scarcity
Bottom line:
• Progress can be made, but it takes a lot of people, time,
and effort
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
26
Going Forward
Need increased focus on low-hanging fruit
• Less than Best-Efforts (LBE)
• Ad-hoc CoS at congestion points
Basic LBE idea
• Low-priority traffic class
• Agree on an Internet2 LBE code point
• Benefits
– Traffic management
– License to hog (some users are self-policing)
– New advanced applications (e.g. large scale distributed
computation)
– It’s easy!
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
27
Starvation: Making LBE Stronger
Define “starvation class selector” (SCS)
• An SCS packet MUST never consume a network
resource desired by a non-SCS packet
• Corollary 1: there must not be a configured minimum
departure rate for SCS traffic
• Corollary 2: all queued SCS packets must be evicted
before a forwarding element can drop a non-SCS
packet (probably difficult to implement)
Result: it’s the dual of EF!
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
28
Any Questions?
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
29
For more information...
Internet2 Home:
• http://www.internet2.edu/
Internet2 QoS Working Group Home:
• http://www.internet2.edu/wg/qos/
QBone Home:
• http://qbone.internet2.edu/
Abilene Premium Service Home:
• http://www.internet2.edu/abilene/qos/
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
30
Internet2 QoS: Overview and Experiences—Campus Focused Workshop on Advanced Networks
31