WPP Study Group Tutorial
• Group created at Vancouver meeting to
create test definitions and methods for
802.11
• Presentations today will cover WPP from
several points of view
– Large user
– IC vendor
– Testing community
Agenda
•
•
•
•
•
•
•
Bob Mandeville: Iometrix
Don Berry: Microsoft
Mike Wilhoyte: Texas Instruments
Kevin Karcz: UNH-IOL
Jason A. Trachewsky: Broadcom
Fanny Mlinarsky: Azimuth Systems
Round Table Discussion and Q&A
802.11 Requirements for
Testing Standards
Bob Mandeville
bob@iometrix.com
WPP
•
•
•
•
What is the need for 802.11 metrics
What problems will they help solve?
Who will the primary users be?
How do we go about creating new metrics
for wireless?
Two Approaches to Creating
Testing Standards
• IETF (BMWG)
– Based on two-step
approach to
definitions:
• Terminology document
(all relevant functional
characteristics are
defined)
• Methodology document
• This method is most
appropriate for
performance testing
• ATM Forum, Metro
Ethernet Forum
– Based on ratified
standards documents
– Each test definition is
referenced to
standards source text
• This method is most
appropriate for
conformance testing
IETF BMWG Test Standards
Templates
• Terminology Definition Template:
– Term to be defined. (e.g., Latency)
– Definition: The specific definition for the term. Discussion: A brief
discussion about the term, it's application and any restrictions on
measurement procedures.
– Measurement units: The units used to report measurements of this
term, if applicable.
– Issues: List of issues or conditions that effect this term.
– See Also: List of other terms that are relevant to the discussion of this
term.
• Methodology Definition Template:
–
–
–
–
–
Objectives
Setup parameters
Procedures
Measurements
Reporting formats
Conformance Oriented
Test Methods Template 1/2
Test Name
Test Definition ID
Name derived from reference document
A punctuated alphanumeric string assigned to each defined requirement and test procedure
couple using the following convention: ‘one to three letter abbreviated source document
name’ . ‘section number’ - ‘paragraph number in the section from which requirement is
derived’. This number always figures as the last number of an ID. Ethernet Services
Model=M; Ethernet Services Definitions=S; Traffic and Performance Parameters for
SLSs=T. Example: M.6.1-4
Reference
Document
Source
Reference document and section (and paragraph when useful for clarity)
Test Type
Functional, Conformance, Interoperability or Performance
Test Status
Requirement
Description
Description of
DUT/SUT
Normative, optional, additional
Brief description of the service requirement that the device must or should satisfy
Type of Ethernet frame forwarding Device Under Test (DUT). Common designations used
in this document are: CE Device (Customer Equipment Device); UNI Device (User
Network Interface Device); MEN Device (Metro Ethernet Network Device). A UNI device
may be considered as a kind of MEN ‘border’ device.
Conformance Oriented
Test Methods Template 2/2
Test Object
Test Bed
Configuration
Test Procedure
Units
Variables
Results
Remarks
Succinct description of test purpose
Succinct description of test bed configuration
Succinct description of the test procedure with mention of the test stimulus and
expected output
Units can be time units, rates and counts in integers such as milliseconds, frames
per second and numbers of frames transmitted or received. For the most part
units used are defined in RFCs 2285, 2544, 2889
Variables such as frame length, test duration, numbers of interfaces under test
must be described
Description of the textual, numerical and/or graphical format in which to display
test results
Description of any particular observations that might effect the test result
Sample List of 802.11 Terms
to be Defined by Category:
Requirements for
Testing Standards
•
Roaming Test
– A practical example which shows poor performance
related to a lack of test standards definitions
•
To roam a 802.11 a/b/g device will:
1.
2.
3.
4.
•
disassociate from one AP
search for a stronger RF signal from another AP
then associate and authenticate with that A
resume normal data transmission
Roaming can fail due to:
– transient RF conditions
– the time that APs and devices take to complete the four step
roaming process
Test Configuration
Azimuth W800 Chassis
Channel 1
Monitor STM
(Airopeek)
STM with roaming client
(DUT)
Channel 2
Monitor STM
(Airopeek)
C
o
m
b
i
n
e
r
RFPM-1
RFPM-2
RF Shielded Enclosure
RF Shielded Enclosure
R
R
Access Point 1
Access Point 2
Fiber Optic
Repeater
R
Azimuth Director®
Hub Test Network
R
Test Procedure
• Roaming Test recorded:
Total Roaming Time = Decision Time + Switch Over Time
– The Decision Time is the time it took the NIC to stop attempting to
transmit packets to AP 1 after the attenuation of the RF signal
– The Switch Over Time is the time it took the NIC to complete its
association with AP2 after it stopped attempting to transmit packets to
AP 1
– During the Decision Time cards recognized the signal attenuation and
invoked proprietary algorithms to adapt their rates to slower speeds.
– Switch Over Time ends when the NIC receives the AP’s
acknowledgement to its association request.
– This time should only be recorded as valid if data traffic from the NIC to
the AP successfully resumes transmission after association.
Test Results
802.11a/b/g Roaming Times
35
30
Seconds
25
20
Total Roaming Time
Switch Over Time
15
10
5
0
802.11a 802.11a 802.11a 802.11g 802.11g 802.11g 802.11g 802.11b 802.11b 802.11b
APa-NICa APa-NICb APa-NICe APb-NICb APb-NICc APc-NICb APc-NICc APd-NICa APd-NICc APd-NICe
Best to Worst Roaming Times
35
30
Seconds
25
20
15
10
5
0
Best Case Single
Vendor (802.11a)
Best Case Vendor
Mix (802.11b)
Average (802.11a)
Average (802.11b)
Average (802.11g)
Overall Average
(802.11a/b/g)
Worst case
(802.11g)
Test Conclusions
• Need to break out Decision Time and Switch over Time
• Switch Over Times are as low as 60 milliseconds and averages a
little over 700 milliseconds over all the combinations excepting two
outliers which took over 8 seconds.
• In the majority Decision Time is the largest contributor to the overall
Roaming Times.
• Packet traces show that most implementations of the rate adaptation
algorithms maintain transmission at the lowest 1 Mbps rate for
several seconds after loss of RF signal has been detected.
• These algorithms will need to be revisited to deliver quality roaming.
• Test standards for measuring roaming times can make a significant
contribution by aligning all vendors and users on a common set of
definitions
• This applies to roaming but also to a large number of other
undefined terms
Challenges of Operating an
Enterprise WLAN
Don Berry
Senior Wireless Network Engineer
Microsoft Information Technology
Microsoft’s Current
WLAN Network
– 4495 Access Points
– 1 AP per 3500 sq/ft
– ~15 Million sq/ft covered in 79 countries
– 70,000 users
– 500,000+ 802.1x authentications per day –
EAP-TLS
– Supports 5.5 and 11Mbps only
Wireless Service Challenge
•
•
•
•
What is “Wireless Service”?
How is it measured?
What factors impact Wireless Service?
How do you improve Wireless Service?
Wireless Service and Support
• Service Goals
– Make Wireless service equivalent to wired
– Offer unique mobile services
• Operational Goals
– Reduce operational costs
– Minimize support personal
How can WPP Help?
• Produce criteria that reflect the client
experience
• Offer information that can compare
different environments – Enterprise,
SOHO, home
Desired Outcome of WPP:
A Perspective From a Si Provider
Mike Wilhoyte
Texas Instruments
Key Issues We Face Relevant to WPP
• Supporting Customers w/ Custom Test Fixtures
– Are often uncontrolled and therefore repeatability is questionable
– May introduce unintentional impairments and therefore don’t
effectively isolate the feature under test
– May unintentionally push beyond the boundary of the
specification
– May stress the system beyond what any user or evaluator will do
– May overlook other critical areas of system performance
• The complexity of the specification has grown since the
days of 11b and more than ever, performance is setup
dependent
– Are tests really apples-to-apples?
These Issues Result in:
• Confusion over unexpected test results
• Resource drain
A Real Example:
Customer ACI Test Fixture
RF Shield Room – non Anechoic
AP
Over 30 Active APs
SMIQ
DUT
TCP/IP
AP
AP
AP
People Observing the Results
Test:
Plot TCP/IP throughput with increasing
levels of interference from the SMIQ
AP
Issues With This ACI Test Fixture
Can you imagine trying to repeat any test result from this fixture in YOUR lab?
• Metal walls in shield room producing multipath making
the test results depend even more on the position of
the laptop (in a fade?)
• People (2.4 GHz RF absorbers) in the room
• Over 30 AP’s active which may couple into the RF
front-end (even though it’s cabled) of the test AP
• SMIQ produces a non-realistic signal since the carrier
is always present even though it may be triggered
externally
– There are ways around this
• The test AP is not isolated from the interference and
its behavior will affect the test result of the DUT
– Rx performance in the same interference
– Deferral behavior in the Tx (CCA) is affected
– Rate adjustment behavior
A “Better” ACI Test Fixture
Testing the STA
Channel 6
AP
Atten
out
in
DUT
clp
Anechoic Chamber 3
PA
PA isolates interfering network and is
not affected by traffic in chambers 3,4
30 dB
Anechoic Chamber 4
Attenuator
clp
in
Interfering
network swept
on channels
1-11
20 dB
pad
AP
Anechoic Chamber 1
out
30 dB
pad
STA
Anechoic Chamber 2
A “Better” ACI Test Fixture
Testing a Network (AP+STA)
Channel 6
AP
DUT
30 dB
PA
PA isolates interfering network and is
not affected by traffic in chamber 3
Anechoic Chamber 3
Attenuator
clp
in
Interfering
network swept
on channels
1-11
20 dB
AP
Anechoic Chamber 1
out
30 dB
STA
Anechoic Chamber 2
Desirable Outcomes of WPP
• Develop a minimal set of test metrics that are
relevant to key performance parameters such
as:
– Robustness
– Throughput/Capacity
– Range
• Develop a Set of Test Best Practices that:
– Produce repeatable results
– Achieve the right balance between complexity and
cost
– The industry will use
UNH-IOL perspective on
WLAN Performance testing
Kevin Karcz
March 15, 2004
A Quick Background
• UNH-IOL Wireless Consortium primarily has focused on
interoperability and conformance tests for 802.11, not
performance testing
• Have generated traffic loads to observe a DUT’s
behavior under stress, but not specifically to measure
throughput or related parameters.
• However, QoS is inherently about providing performance
while sharing limited resources
– Optimization of: Throughput , Range, Delay & Jitter
– Constrained by:
• User resources: CPU load, DC power
• Community resources: Available spectrum, aggregate users
–
Examples of performance tests
• PHY layer
– Multi-path fading channel emulation using a
Spirent SR5500 fader.
• What fading models should be used?
• MAC layer
– Throughput varies with Traffic Generator used
– CPU load differs significantly for between
different vendors. Much greater than CPU
load for a typical Ethernet device.
Clear methods of testing are
needed…
• As we start measuring more performance
metrics
• Can each layer of the network be measured
independently?
• Which metrics need to look at the interaction of
multiple layers?
• Hassle of real world scenario testing vs. a PHY
test mode?
– Black box testing requires DUT to authenticate and
associate with test equipment and interact at the MAC
layer, not just the PHY layer.
Some gray areas of testing
•
Throughput
– Is throughput measured as a MAC layer payload? At IP layer? TCP or UDP
layer?
– One DUT may have better PER measurements at the PHY layer than a 2nd DUT,
but may get worse throughput if it’s rate selection algorithm is poor.
– Difficult to maintain consistency in an open (uncontrolled) environment
• Can throughput be measured in a cabled environment without an antenna?
• What if the DUT has a phased array antenna?
• What if the device is mini-PCI and inherently has no antenna?
•
Range test
– What if a higher TX level causes higher adjacent channel interference and brings
the aggregate throughput down for a neighboring BSS?
•
Power consumption
– Is this just the DC power drain at the cardbus card interface?
– Should CPU load be included if the DUT implements much of it’s MAC
functionality on a host PC?
•
Roaming
– Quickest time: 1 STA, 2 APs on same channel
– More realistic: AP reboots, Multiple STAs roam to new AP on new Channel
Why WPP role is important to UNHIOL?
• Open standards are desired for the basis
of test suite development
• Defining test parameters and
standardization of Test Scenarios makes
comparison of ‘Apples to Apples’ easier
• IEEE focuses on the technical content
• Our interest is the testing, not determining
how results are utilized
Why WPP should define the tests
• UNH-IOL follows IEEE PICS for test cases
• More detailed info for test results
– Cases: PDA/laptop/AP weight test results
differently
Example criteria weighting
Laptop
Through Delay &
put
Jitter
++
+
DC
power
++
Roaming
time
+
AP
++
++
N/A
N/A
VoIP
phone
0
+++
++
++
Comments on Wireless LAN
Performance Testing And
Prediction
Jason A. Trachewsky
Broadcom Corporation
jat@broadcom.com
Topics
• Test Categories for WPP
• Some Test Configurations
Test Categories for WPP
• Deciding what parameters are to be
considered is the challenge.
• How do we transform user perception of
performance into a set of repeatablymeasurable quantities?
–Throughput and Range (what environments?)
–Frame latency
–Visibility of APs
Test Categories for WPP
• How do we transform user perception of
performance into a set of repeatablymeasurable quantities?
–Delays in association/authentication
–Host CPU utilization
–Ability to roam without loss of connections
–Etc.
Test Categories for WPP
• Basic PHY/RF Measurements
–Transmitter Parameter Measurements
• TX EVM or Frame Error Rate (FER) with
Golden/Calibrated Receiver
• Carrier suppression
• Carrier frequency settling
Test Categories for WPP
–Receiver Parameter Measurements
• RX FER vs. input power
–Flat channel (controlled through cabled/shielded
environment)
–Controlled frequency-selective channels (known multipath
power-delay profile)
–Antenna measurements
• cable/feed losses (S11 and S21)
• gain vs. azimuth and elevation angle
• One can easily take a great receiver design and
blow away all gains with a bad antenna or lossy
feed!
Test Categories for WPP
• MAC Layer Measurements
–rate adjustment behavior
• specific parameters? test conditions?
–association and roaming behavior
• specific parameters? test conditions?
–frame latency
–layer-2 throughput with encryption
–host CPU cycles consumed?
Test Categories for WPP
• Layer-4 Measurements
–UDP frame loss rate and latency vs. received
power
• flat or frequency-selective channels?
–TCP throughput vs. received power
• flat or frequency-selective channels?
Test Categories for WPP
• Open-air Measurements
–Open-air measurements are always subject to
imperfectly-known time-varying multipath
power-delay profiles.
• There is substantial variation at 2.4 and 5.5 GHz
over 10s of msec.
Test Categories for WPP
–We have established that frequency-selectivity
due to multipath can result in higher-loss
channels having higher capacity than lowerloss channels.
• The capacity of the channel can vary rapidly.
• This is a more significant problem for systems like
802.11a and 802.11g which include a large number
of possible rates to better approach the practical
capacity.
• (The problem won’t get easier for future WLAN
standards.)
Test Categories for WPP
• Open-air Measurements
–What can we do?
• Try to perform as many measurements as possible
with cable networks.
• Perform open-air measurements in an area in
which the distance between transmitter and
receiver is small compared with the distance
between either transmitter or receiver and any other
object. I.e., strong LOS.
–Helpful but not sufficient, as even small reflections affect
channel capacity.
Test Categories for WPP
• Open-air Measurements
–What can we do?
• Give up and perform a large ensemble of
measurements and gather statistics.
UPS
10 Mhz clk
Channel Measurement Block
Diagram
ARBtx0
Filt
ARBtx1
Filt
ARBtx2
Filt
ethernet
DC
PWR
linux
controller
10/100 hub
• Scope provides 10Mhz reference clk for all systems
• 3 long interconnect cables connect tx and rx side
• Filt module includes LNA
10 Mhz clk
10/100 hub
Filt
trigger
ethernet
ARBtxctl
Scoperx
gpib
I/Q
ARBtx3
UPS
Time- and Frequency-Selective
Fading
time evolution of TX_ANT0 to RX_ANT0 channel
-58
-60
> 10-dB change in
received signal
power in some bins
over 60 msec.
-62
gain (dB)
-64
-66
-68
-70
0 msec.
10 msec.
20 msec.
30 msec.
40 msec.
50 msec.
60 msec.
-72
-74
-76
-25
-20
-15
-10
-5
0
5
subcarrier index
10
15
20
25
Topics
• Test Categories for WPP
• Some Test Configurations
System Block Diagram
–Main boardtest system
–Transmit test system
boardtest
system
Transmit
Test Filter
Mux
controller
(attenmach)
RF cable
GPIB or DIO line
GPIB
ethernet
Microwave
Spectrum Analyzer
Boardtest System
controller
(attenmach)
DUT
mux-2
coupler
atten
coupler
mux-3
REF
SA
5V,12V,GND
mux-6
RF cable
GPIB or DIO line
GPIB
Ethernet
power
meter
RF “Loop” Block Diagram
Does WPP specify RF test fixtures?
ATTEN
Split /
Golden
Node
ATTEN
Split /
Combine
Combine
ATTEN
ATTEN
6-dB
Loss
6-dB
Loss
ATTEN
Does WPP specify fading channel emulators (no!) or
a set of fading channel profiles (maybe!).
DUT
Multipath Channel 2
channel 2 s
( t = 133 nsec.)
-30
Example fixed
multipath channel
power-delay profile.
-40
-50
path gain (dB)
-60
-70
-80
-90
-100
-110
-120
-130
0
50
100
150
200
250
300
delay (nsec)
350
400
450
500
Comments on Wireless LAN
Performance Testing And
Prediction
Fanny Mlinarsky
Azimuth Systems
fanny_mlinarsky@azimuth.net
Ethernet vs. WiFi
Wired test metrics
RFC 2285,2544,2889
Wireless test metrics
Test Metrics
Packet forwarding
Behavioral
Security
Roaming
QoS
Rate adaptation
Encryption
WEP
TKIP
AES
Authentication
Offered
Load
Packet
Size
# of
Clients
% of
Power Save
Clients
% Load
Of Assoc/
De-assoc/
Re-assoc
RTS/CTS
Fragmentation
EAP-TLS
EAP-TTLS
PEAP
LEAP
Forwarding Rate (Frames/Sec)
2500
Forwarding Rate (Frames/Sec)
Forwarding Rate Measurement
2500
Open air
2250
2000
1750
1500
Controlled RF
2250
2000
1750
1500
0
120
240
360
Time (Seconds)
480
600
Controlled Test Environment
• If measurements are
not repeatable then
the test is invalid
• Open air creates
unpredictable test
conditions due to
interference and
multipath
• Shielded and cabled
test environment may
be necessary for
some measurements
Summary
• The IT manager’s question: “How
well do mobile computing solutions
perform in the enterprise?”
• The dilemma: Standard ‘wired’
benchmarking techniques won’t give
the right answer
• Verifying function and performance in
the inherently unstable wireless space
calls for a new methods & metrics
• The answer: New methods to test and
measure every aspect of wireless
protocols
• Wireless metrics outnumber traditional
Ethernet metrics 5:1