IEEE P802.17
Resilient Packet Ring
Stein Stein Gjessing
Simula Forskningslaboratorium
e-mail: steing@ifi.uio.no,
Stein Gjessing, Simula Forskningslab
http://www.ifi.uio.no/~steing
1
Agenda
‹ Historic
background
‹ An IEEE standardization project
‹ Introduction to the technology
‹ Present technology and standardization
status
‹ My simulation work
Stein Gjessing, Simula Forskningslab
2
SDH/Sonet rings
‹ Sonet/SDH
rings used for several years
‹ Cisco introduces DPT (Dynamic Packet
Transport) and SRP (Spatial Reuse
Protocol) around 1998
‹ Other vendors introduced similar consepts
‹ Protection is important
Stein Gjessing, Simula Forskningslab
3
SDH/Sonet connected dual ring
e.g. Cisco DPT
4
5
The SDH-lines are
terminated at each
station.
The SDH frames are
unpacked and repacked at each station.
0
Can this be simplified?
3
2
1
Stein Gjessing, Simula Forskningslab
Protection !
4
Stanardization
‹ Discussed
in IETF (Link layer ...)
‹ Made an IEEE Study Group (SG) in spring 2000
‹ Made a full IEEE Working Group (WG) project
with first official meeting in March 2001
‹ Under the IEEE 802 LAN/MAN umbrella
‹ Called IEEE P802.17 Resilient Packet Ring
‹ Around 7 meetings in SG and WG so far
attended by ”all” vendors and major service
providers. 120 people at each meeting
Stein Gjessing, Simula Forskningslab
5
IEEE 802 (LAN/MAN) is
‹ 802.3
– Ethernet
‹ 802.5 – Token Ring
‹ 802.11 – Wireless LAN
‹ 802.17 – Resilient Packet Ring
Other:
‹ 1394.1 Serial bus (”FireWire”, iLink, ...)
‹ 1596
Scalable Coherent Interface
‹ 754
Floating point arithmetic
‹...
Stein Gjessing, Simula Forskningslab
6
PAR
‹ All
IEEE standardization projects has a
”Project Authorization Request” (PAR)
‹ P802.17 PAR: . . .
9. Scope of Project: Define a Resiliant Packet
Ring Access Protocol for use in Local,
Metropolitan and Wide Area Networks [ ] for
transfer of data packets at rates scalable to
multiple gigabits per second. . . .
Stein Gjessing, Simula Forskningslab
7
IEEE 802.17 – Resilient Packet Ring
4
5
3
Packet sizes from 80?
to 16K? bytes.
MAC frames (?)
Station numbers are
MAC addresses.
Utelize Ethernet
transeivers.
0
2
1
Stein Gjessing, Simula Forskningslab
Send shortes path
around ring.
Destination stripping
No token but
”Insertion buffer”
(not slotted)
8
From the ring
RPR station
Into the ring
output
selector
Bypass buffers
Stein Gjessing, Simula Forskningslab
stripper
stripper
Bypass buffers
output
selector
Choice of
ring to travel
to the
destination
(?)
9
RPR MAC layer station architecture
From the ring
stripper
Bypass buffers
output
selector
Bypass buffers
Stein Gjessing, Simula Forskningslab
Into the ring
output
selector
Into the ring
stripper
From the ring
10
MAC interface ?
From the ring
stripper
Bypass buffers
output
selector
Bypass buffers
Stein Gjessing, Simula Forskningslab
Into the ring
output
selector
Into the ring
stripper
From the ring
11
Single direction interface
From the ring
stripper
Bypass buffers
output
selector
Into the ring
1. Transit traffic flows through the node
2. If this station is transmitting when a new packet arrives from
previous station (from left) then the new packet is temporarily
buffered in the bypass buffer
Stein Gjessing, Simula Forskningslab
12
Single direction interface (cont´d.)
From the ring
stripper
Bypass buffers
output
selector
Into the ring
1. This station is only allowed to send when bypass buffer
is empty
Stein Gjessing, Simula Forskningslab
13
Spatial Reuse
From the ring
stripper
Bypass buffers
output
selector
Into the ring
1. Incoming packet (from left) is stripped off (and removed) from
the ring if destined for this station
Stein Gjessing, Simula Forskningslab
14
Spatial Reuse
4
5
3
E.g.
4 sends to 1
at the same time as
2 sends to 3
(on the same ring)
0
2
1
Stein Gjessing, Simula Forskningslab
15
Flow control / back presure - starvation
4
5
3
Scenario:
1 sends a continuous
stream to 4.
5 wants to send,
but can not,
because the bypass
buffer is filled with
data from station 1
0
2
1
Stein Gjessing, Simula Forskningslab
16
Starvation
Data waiting to
get into the ring
stripper
Bypass buffers
output
selector
From the ring
This station can not send because one or more upstream
node are sending all the time.
Stein Gjessing, Simula Forskningslab
17
Starvation (soon to end)
Data waiting to
get into the ring
stripper
Bypass buffers
output
selector
From the ring
When upstream nodes have stopped sending,
(or is sending to this station),
the bypass buffer will first be emptied
Stein Gjessing, Simula Forskningslab
18
Starvation (has ended)
Data is sent
into the ring
stripper
Bypass buffers
output
selector
From the ring
When the bypass buffer is (nearly) empty,
the station can send its own data.
Stein Gjessing, Simula Forskningslab
19
Bypass queue filling up
Data is sent
into the ring
stripper
Bypass buffers
output
selector
From the ring
When new packets arrive from upsteam nodes while
this staion is sending, the bypass buffer will
again be filled up
Stein Gjessing, Simula Forskningslab
20
How to force upstream node to slow down
Into the ring
From the ring
Into the ring
Bypass buffers
output
selector
Bypass buffers
Stein Gjessing, Simula Forskningslab
output
selector
stripper
Send small ”slow
down” packet
stripper
From the ring
21
Flow control / back presure (cont´d)
Sending
”slow down”
packet in
opposite
direction
4
5
3
0
2
Can go
up to half
way around
the ring
1
Stein Gjessing, Simula Forskningslab
22
QoS / ToS
‹ High
level problem statement:
Mapping of IP QoS / ToS onto
MAC layer (level 2) ”ToS”
Stein Gjessing, Simula Forskningslab
23
QoS / ToS
‹ How
many RPR traffic classes?
‹ How should IP ToS field map into these RPRclasses?
- PHB (Per Hop Behaviour)
‹ How many classes is it naturally to have within an
RPR ring?
‹ What should the different behaviours be?
‹ How much real estate is neccesarry to suport this
behaviour?
‹ How complex would the algorithms be?
Stein Gjessing, Simula Forskningslab
24
Station designed for two traffic classes
High priority
synchronous application logic (provisioned)
priority sink
fast insertion
buffer
normal transit path
normal sink
normal insertion
buffer
1
3
4
outputSelector( )
stripper( )
fast transit path
2
asynchronous application logic
Low priority
Stein Gjessing, Simula Forskningslab
25
QoS / ToS
‹ One
set of buffers for each traffic class
‹ Some traffic classes provisioned, no flow
control needed
‹ Absolute priority
or
(probably)
(probably not)
‹ Weighted fair queueing
‹ QoS
/ ToS only in Fast Ethernet bridges
before
Stein Gjessing, Simula Forskningslab
26
Packet discard – preemption ?
‹ Should
packets be allowed to be discarded
within the ring? - NO!
‹ Should a high priority packet be allowed to
(non fatally) preempt a lower proiority
packet?
MAYBE!
– Can we handle packets within packets
Stein Gjessing, Simula Forskningslab
27
Nested packets = preemption
4
5
First and second
part of low priority
preempted packet
3
0
High priority packet
2
1
Stein Gjessing, Simula Forskningslab
28
Cut-through vs. Store&Forward
‹ In
a Store&Forward system the complete packet is
stored within a switch before it is sent on
‹ In a (virtual) Cut-through system the head of the
packet can be sent on before the complete packet
has entered the node.
One requirement is that the output link is not
faster than the input link.
‹ (Virtual) Cut-through routing spreads the packet
over potentially many switches and links
Stein Gjessing, Simula Forskningslab
29
Cut-through vs. Worme-hole routing
‹ (Virtual)
Cut-through and Worm-hole routing is
almost the same
‹ In a Cut-through system the buffers in switches
must be able to store a complete packet
‹ In a Worm-hole system the packets (and the
buffers) are divided into ”flits”, and a the buffer in
one switch might be able to store only parts of the
buffers (an integer number of flits). Flow control
stops the rest of the packet from leaving a switch
when the buffers of the next switch are full.
Stein Gjessing, Simula Forskningslab
30
Protection - Resilient
4
5
50 ms protection
Wrap-around
3
0
Gradually let all
stations know so
they can send packets
in more optimal
direction
2
1
Stein Gjessing, Simula Forskningslab
31
RPR performance modeling
‹A
simple RPR model written in the
programming language Java:
–
–
–
–
–
Class Node
Class DualNode
Class Buffer
Class Link
Class Packet
// single direction node
// several needed in each node
// one (out) for each single Node
// new one for each packet sent
– Class Application
– Class Kernel
Stein Gjessing, Simula Forskningslab
// generating system load, etc
Class Unit // simulation environment
32
Simulation Topology
‹ 16
nodes (numbered 0 – 15), dual rings
‹ 250 microsec. cable between each node
– includes one node bypass latency
– ( ~ 50 km between each node )
‹ 1Gbyte/sec
bandwidth (= 10Gbit/sec)
Stein Gjessing, Simula Forskningslab
33
Two basic Scenarios
‹ Scenario
A – Random receiver
– Overloaded system – 10Gbit/sec/link
– Three background packet sizes:
1600 , 16K and 520 bytes
‹ Scenario
B – Hot receiver
– Partly highly loaded system – 10Gbit/sec/link
– Three background packet sizes:
1600, 16K and 520 bytes
Stein Gjessing, Simula Forskningslab
34
Simulation Scenario - A
Random receiver
12
11
13
10
Only traffic
streamed from
7 to 15 are logged
14
9
15
8
0
7
1
6
5
1
Streaming
from 7 to 15
Stein Gjessing, Simula Forskningslab
3
2
Random traffic
between all nodes.
30% bandwidth
provisoned
small high prio.
70% bandwidth
overloaded,
large low prio.
35
Simulation Scenario - B
12
11
13
10
9
Hot receiver
14
All traffic from
all nodes are sent
to node 15
8
Only traffic
streamed from
7 to 15 are logged
15
0
7
1
6
5
1
Streaming
from 7 to 15
Stein Gjessing, Simula Forskningslab
3
2
Links into 15
have close to
100% utilization
15 is hot receiver
30% bandwidth
provisoned
small high prio.
70% bandwidth
large, low prio.
Last link to 15
almost fully used
36
Measured traffic
‹ Latency
/ jitter
‹ Streaming small high priority packets
(80 bytes including header)
from node 7 to node 15
– 8 hops, ~ 400 km distance = 2ms min. latency
‹2
us. between packets
‹ 125 us. between packets (TDM frame interval)
Stein Gjessing, Simula Forskningslab
37
Background traffic
‹ Load distribution
– 30 % bandwidth high priority small packets
– 70% bandwidth low priority
» - ”IP-packets” (1600 bytes)
- Jumbo-packets (16K)
– - Jumbo-packets with preemption (1/2 K)
W A.
Random receiver
– network overloaded
W B.
Hot receiver (node 15)
– Almost full utilization of last link into hot receiver
(i.e. lighter loaded system than A, but not easy to get
comparable load in all cases)
Stein Gjessing, Simula Forskningslab
38
Overall Conclusion – 10Gbit/sec system
‹ In
a ”high load” system 1% of the packets observe
half of the theoretical max latency
‹ In an overloaded system 1% of the packets
observe close to the theoretical max latency
‹ Hence with Jumbo packets (16K) and no
preemption it is possible to get 100 us. added
latency with 8 nodes (128 us. theoretical max).
This is close to the 125 us. synchronous stream
interval (TDM frame interval)
‹ Jitter almost as large as total latency variation
Stein Gjessing, Simula Forskningslab
39
RPR main architecture (dual ring)
From the ring
stripper
Bypass buffers
output
selector
Bypass buffers
Stein Gjessing, Simula Forskningslab
Into the ring
output
selector
Into the ring
stripper
From the ring
40