Brussels 05 June 2012
Measuring quality of user
experience in communication
networks
Pravir Chawdhry
European Commission - Joint Research Centre
ITU Regional Conference on Broadband Mapping
Warsaw, Poland – 11-12 April 2016
Brussels 05 June 2012
Quality of User experience
One of the most important aspects that affects the Internet
consumer experience is the quality of network performance.
Traffic growth is dominated by on-line video and audio-streaming
so low latency and high speed add great value to user
experience.
Brussels 05 June 2012
Quality of User experience
One of the most important aspects that affects the Internet
consumer experience is the quality of network performance.
Traffic growth is dominated by on-line video and audio-streaming
so low latency and high speed add great value to user
experience.
Speed test applications have become widely used tools for
customers.
Brussels 05 June 2012
Quality of User experience
One of the most important aspects that affects the Internet
consumer experience is the quality of network performance.
Traffic growth is dominated by on-line video and audio-streaming
so low latency and high speed add great value to user
experience.
Speed test applications have become widely used tools for
customers.
But how reliable are these tools?
Brussels 05 June 2012
Quality of User experience
One of the most important aspects that affects the Internet
consumer experience is the quality of network performance.
Traffic growth is dominated by on-line video and audio-streaming
so low latency and high speed add great value to user
experience.
Speed test applications have become widely used tools for
customers.
Are these tools reliable?
What do these tools really measure?
Measuring network performance
Brussels 05 June 2012
ICT Technicians can use advance tools and
use them with their domain competences.
Measuring network performance
Brussels 05 June 2012
ICT Technicians can use advance tools and
use them with their domain competences.
Know-how to check
network activity.
Measuring network performance
Brussels 05 June 2012
ICT Technicians can use advance tools and
use them with their domain competences.
Know-how to
inspect network
characteristics
with command
like tracert and
nmap.
Know-how to check
network activity.
And run specialized
applications like
speedtest-cli, curl, etc..
Measuring network performance
Brussels 05 June 2012
Nontechnical citizens search for a Speed test
tool over Internet.
Measuring network performance
Brussels 05 June 2012
Nontechnical citizens search for a Speed test
tool over Internet.
Measuring network performance
Brussels 05 June 2012
Nontechnical citizens search for a Speed test
tool over Internet.
Measuring network performance
Brussels 05 June 2012
But results can be very confusing.
Measuring network performance
Brussels 05 June 2012
How can we know if these tools are reliable?
Measuring network performance
Brussels 05 June 2012
How can we know if these tools are reliable?
Here is an example of speed test measures over the
Internet from the same Workstation using 4
different tools
App
Download
Ping
Speed
ms
Mb/s
Upload
Speed
Mb/s
SpeedTest
14
204,49
83,36
mLab
22
9,75
13,19
netBravo
71
15,74
59,63
RTR-NetTest
18
200,00
42,00
Measuring network performance
Brussels 05 June 2012
How reliable are these tools?
And what are they measuring?
Measuring network performance
Brussels 05 June 2012
How reliable are these tools?
And what are they measuring?
UDP
Measuring network performance
Brussels 05 June 2012
How reliable are these tools?
And what are they measuring?
UDP
TCP
Measuring network performance
Brussels 05 June 2012
How reliable are these tools?
And what are they measuring?
UDP
TCP
HTTP
Measuring network performance
Brussels 05 June 2012
The discrete behaviour of the NETWORK is very complex and fairly
unpredictable
Compression
Server
Switch
Router
Firewall
Internet
Every transition can have effect on the final resulting speed due to:
• High bandwidth usage on any route segment
• Router, firewall or switch misconfigurations
• CPU overhead on any used device
• Client network activity
• Compressible or uncompressible content
• ….
What about benchmarking?
Brussels 05 June 2012
To reduce the complexity of the network and test the reliability of
Speed Test tools we can create a reference framework to test and
benchmark the applications using a simulated simplified network in
which we can operate on various parameters:
•
•
•
•
•
Up and Down stream Bandwidth
Latency
Packet loss
Round time trip
….
Following the directive of IPPM (IETF)
Benchmarking: an example
Brussels 05 June 2012
Previous speed test tools tested on a simulated network that cut
download and upload bandwidth to 1, 2, 5, and 10 Mb/s
WebApp
NetBravo
M-Lab
Ookla
Z-Test
NetBravo
M-Lab
Ookla
Z-Test
NetBravo
M-Lab
Ookla
Z-Test
NetBravo
M-Lab
Ookla
Z-Test
Latency
1Mbs
23 ms
410 ms
30 ms
72 ms
2Mbs
23 ms
230 ms
62 ms
52 ms
5Mbs
53 ms
102 ms
30 ms
48 ms
10Mbs
18 ms
76 ms
31 ms
51 ms
Download
Upload
0,90
0,88
0,97
0,94
Mb/s
Mb/s
Mb/s
Mb/s
1,02
0,91
0,98
0,94
Mb/s
Mb/s
Mb/s
Mb/s
1,74
1,84
1,92
1,90
Mb/s
Mb/s
Mb/s
Mb/s
2,08
1,88
1,93
1,90
Mb/s
Mb/s
Mb/s
Mb/s
4,38
4,63
4,80
4,80
Mb/s
Mb/s
Mb/s
Mb/s
4,96
3,84
4,81
4,70
Mb/s
Mb/s
Mb/s
Mb/s
8,59
9,22
9,61
9,70
Mb/s
Mb/s
Mb/s
Mb/s
9,33
7,91
9,31
4,60
Mb/s
Mb/s
Mb/s
Mb/s
Brussels 05 June 2012
JRC experience: netBravo
JRC developed a mobile crowd sourcing application that measures
quality of user experience:
 Speed test: Download and upload data transfer rates.
 Latency, ping and DNS lookup test
 Local GSM / UMTS / LTE signal strength
 Local WiFi 2.4 and 5 GHz access point signal strength, sid and
characteristics (used channel, security protocols, sid, etc..)
 Network neutrality test (voip, nat, p2p, etc..)
Brussels 05 June 2012
iOS Application screen-shots
Brussels 05 June 2012
iOS Application screen-shots
Brussels 05 June 2012
Android Application screen-shots
Brussels 05 June 2012
Android Application screen-shots
netBravo Mapping: netbravo.org
Brussels 05 June 2012
Brussels 05 June 2012
netBravo Application optimizations
The application is:
 Optimized for battery usage
 Optimized for Bandwidth consumption
 Takes measurements only when a location is available
 Measurements are made based on battery state and location
availability
 Uploads measurements to the server when the preferred connection is
available
 Application user data is fully anonymized.
Brussels 05 June 2012
Thank you
Further information:
www.netbravo.eu
Pravir.Chawdhry@ec.europa.eu
+39 0332 78 58 23