Results of benchmarking traffic services provided
by TomTom and Garmin
22.01.2015
© NavCert
1
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
2
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
3
Executive Summary
On behalf of TomTom a benchmark of the traffic services offered by Garmin and TomTom was
conducted in and around the region Berlin. The following Personal Navigation Devices (PNDs) were
used:
•
Garmin 2699 LMT-D with Garmin Live Traffic via Smartphone Link on a Bluetooth connected
Samsung S5 using T-Mobile
•
Garmin 3598 LMT-D with Garmin HD Digital Traffic® based on DAB+
•
TomTom Go 500 with TomTom Traffic via a Bluetooth connected Samsung S5 using T-Mobile
The objective was the independent evaluation of the performance of the traffic services by Garmin and
TomTom for typical commuter routes in Berlin. The tests have been conducted during the typical
commuter times with respective traffic jams:
•
Morning session: 6:30 to 9:30
•
Afternoon session: 16:00 to 19:00
To provide similar conditions the DUTs (Devices under Test) were installed in identical or comparable
vehicles.
Trips with the same origin – destination pairs (“OD pairs”) and the same start times were driven with
each DUT in parallel as shown in Annex 1. Each device was used in average 107 trips to get statistical
significant data.
22.01.2015
© NavCert
4
Executive Summary ctd.
The below table provides the results for the measured delay per trip per DUT. The provided values
in table1 refer to the percentage of delay in relation to shortest duration for that trip.
Minimum
Maximum
Median
Mean
Std. Deviation of
Sample
Std. Deviation of
Mean
Lower 95% CI of
mean perc. delay
Upper 95% CI of
mean perc. delay
TomTom
GO 500
0.0%
37.5 %
0.0%
3.0%
Garmin
2699
0.0%
42.5%
7.1%
9.5%
Garmin
3598
0.0%
55.4%
8.1%
11.0%
6.0 %
10.9%
12.5%
0.6%
1.1%
1.3%
16%
14%
12%
Mean
Confidence Interval
10%
8%
6%
4%
2%
1.8%
7.4%
8.4%
0%
TomTom GO 500
4.2%
11.6%
13.4%
Garmin 2699
Garmin 3598
Figure 1
Table 1
TomTom outperforms Garmin: The average delay of TomTom
with 3.0% is significantly better than the mean of the two
Garmin devices with 7.1% and 8.1% as visualized in figure 1.
22.01.2015
© NavCert
5
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
6
Signals conform to
Galileo specification
Test laboratory conform
to ISO 17025
Receiver integrity
testing (RAIM) for SoL
applications
Who is NavCert?
Functionality
checked
www.navcert.de/
PPP80020
NavCert is a technical service provider
offering testing, verification, validation and certification in the fields of positioning,
navigation and communication worldwide. The enthusiastic NavCert team offers a
dependable and high quality service behind a strong logo. Its services are complemented
with detailed knowledge of navigation and localization applications and experience in the
analysis of both commodity and safety critical systems and services. NavCert provides
certification for a variety of GNSS based products, systems and services.
NavCert is an accredited laboratory by the German national accreditation body DAkkS in
the scope of GNSS, eCall and EETS.
NavCert has agreed to cooperate with TÜV SÜD product services, which allows NavCert
to market certifications in an area not regulated by law and to mark them with the TÜV
SÜD quality logo.
22.01.2015
© NavCert
7
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
8
Methodology
The test methodology is divided in three parts:
• Preparation
– Devices and supporting equipment
– Definition of start and end points per trip
– Test team requirements
• Execution
– Per day
– Per trip
– Data logged
• Results analysis & documentation
– Preparation of data
– Processing of data
– results
22.01.2015
© NavCert
9
Preparation: Devices and supporting equipment
The following devices were used to validate the performance of the traffic service
provider:
•
Garmin 2699 LMT-D with Garmin Live Traffic connected via Bluetooth to a
Samsung S5 using T-Mobile
•
Garmin 3598 LMT-D with Garmin HD Digital Traffic® based on DAB+
•
TomTom Go 500 with TomTom Traffic connected via Bluetooth to a Samsung S5
using T-Mobile
As mobile service provider T-Mobile was selected with a data only service for the
Samsung S5.
The PNDs were bought a few days prior to the execution of the tests and updated to the
latest map and software one working day prior to the execution of the tests. The
smartphones were updated to the latest version both for the operating system Android
and the used apps (Garmin Smartphone Link, MyDrive mobile app). PNDs were set to
“fastest route” and “automatically reroute,” or equivalent.
To ensure equal test conditions for all traffic service providers, passenger cars with
similar dimensions and engine specifications were used as detailed in Annex 2.
22.01.2015
© NavCert
10
Preparation: Definition of trips
The total selection of addresses was representative for the geographical area covered by
Berlin. In order to represent typical commutes, origin/destination pairs representing
centre-to-periphery, periphery-to-centre, within-centre, and within-periphery routes were
identified.
The definition of trips was planned to cover typical commutes (including delay times), with
various times ranging from 30 minutes to 1 hour, with the average trip times being around
40 minutes. During the planning an average delay of 7.5 minutes was assumed.
In order to allow a combined start of all vehicles for the next trip, gathering points were
defined as new starting points. Three trips were grouped into one session and in total 34
sessions were defined. The average, minimum and maximum length of the trips in km
together with the same values for the duration in minutes are depicted below:
average
length
km
32
time
minutes
48
minimum
11
43
maximum
100
53
The distribution of the start and end points is provided in Annex 1.
22.01.2015
© NavCert
11
Preparation: Test team
The test team consisted of a project manager, team leaders and drivers with following
profiles:
•
project manager: senior expert and technical certifier in the scope of GNSS
•
team leader: technical expert experienced in laboratory work and team leading
•
Driver: driving license > 3 years, driving regularly and using PNDs
A half day training session consisted of one theoretical part, the introduction to the
benchmark drive test with the defined processes and a hands-on session, in which all
drivers got used to interact with the DUTs. A hand out was provided describing all
processes and the driver interaction with the DUTs.
A trip logging template was prepared, to log all relevant data during a trip and descriptions
of unexpected events (e.g. engine failure)
After installation into the vehicles, the team leader checked the proper installation and
followed the procedure defined for the start point. The team leader joined the drivers of his
team for the first trip. He validated proper driving and monitored the processes on arrival,
proper documenting and departing from the destinations of the first session.
22.01.2015
© NavCert
12
Execution: Per day
Trips were executed during typically commuter times from Monday till Friday in the
morning from 06.30 to 09.30 and in the afternoon from 16.00 to 19.00 hours. In every
session three trips were driven. All vehicles were parked in the same location, named the
base, from where they started to their first team specific starting point. Drivers received
instructions which sessions they had to drive on that day. Within a team the driver rotated
the used DUTs from one to the other before each session. If necessary additional trip logs
were provided. All vehicles were equipped with a video camera logging date, time and GPS
data.
The team leader validated the proper installation and configuration of all DUTs and required
supporting equipment and provided a new SD card for the video camera. Smartphones and
tethered devices were mounted in close proximity to each other.
In a team leader log the number plate of the vehicle, driver name and used DUT were
documented together with the inventory id of the SD card and the session id. Then the
team leader started his team to go to the first starting point.
All teams drove back after each session to the same parking space (the base). The team
leader interviewed the driver, collected the SD-card of the video camera, the trip logs, and
checked the proper storage of the equipment.
22.01.2015
© NavCert
13
Execution: Per trip
Before the start of each trip, every driver verified where possible that traffic information
services are available. Route planning was finished on each device, and all cars were ready
to go facing the same direction before departure. Cars within a team departed in as short
as feasible intervals from each other. The duration from the departure of the first to the last
car was as short as possible and the drivers strived to keep the duration less than two
minutes. The route was followed without deviation; in case automatic rerouting was not
available, any reroute prompts from the device was accepted and followed. Drivers obeyed
the driving regulations at all times, never exceed speed limits, or break traffic regulations /
take forbidden turns etc. Drivers matched the traffic flow and they didn’t take any kind of
priority lanes like bus or taxi lanes.
22.01.2015
© NavCert
14
Execution: data logged
The following data was recorded per trip, using the trip log (on paper):
• Trip number
• Start and arrival times for each DUT, marked at the exact times in HH:MM format
• Estimated time of arrival (ETA, e.g. 17:45 with a start time of 17:00) at the start of the
trip for each DUT
• Estimated kilometres and actual kilometres driven
• Periods during which a device loses GPS reception, from start to end time, in
HH:MM:SS, however not in locations where loss of GPS connectivity is expected like in
tunnels
• Periods during which a device loses network connectivity, from start to end time, in
HH:MM:SS
• Periods during which a device loses connectivity to the traffic service, from start to end
time, in HH:MM:SS
In parallel a video camera documented the behaviour of the DUT complemented by GPS
location and date and time.
All times documented do not include seconds to avoid wrong impression of high precision
which cannot be measured. This does not influence overall results as this is the same and
consistent methodology applied to all devices
22.01.2015
© NavCert
15
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
16
Result analysis: preparation of data
In the first step an elimination of invalid trips took place. A trip was regarded as invalid if
due to system extrinsic factors a deviation from the route given by the DUT took place or an
external event influenced the driving time:
• Driver error
• Devices crashed or stopped working
• Devices loose Bluetooth tethering, necessary for traffic information
• Map error (guidance in conflict with applicable laws)
• Car break down
• Accident
• Police control
• etc.
However if a route was temporarily blocked, the respective trip was not excluded from
processing as this information was in part communicated by the traffic service providers.
22.01.2015
© NavCert
17
Result analysis: delay per trip
The delay per trip is determined as delay in respect to the minimum duration for this trip.
𝑲𝑷𝑰𝟏𝒂 𝑫𝑼𝑻𝒊 ∶= duration of trip DUT𝒊 − duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
In order to reflect the duration of the single trips in relation to the respective delay, the
percentage of the delay is evaluated.
𝑲𝑷𝑰𝟏𝒃 𝑫𝑼𝑻𝒊 ∶=
22.01.2015
duration of trip DUT𝒊 − duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
© NavCert
18
Result analysis: processing of data
As a first step in the analysis the outlier detection according to GRUBBS with alpha of 0.01
was performed. Then the statistic values were calculated. In the end the statistical
significance was determined with the Anova in the first step for all DUTs. Then the student
test was used to compare TomTom against the other two DUTs.
Below table depicts the performance of the DUTs with respect to the selection of fastest
route based on KPI 1b, the percentage of delay in relation to shortest duration for that trip.
Minimum
Maximum
Median
Mean
Std. Deviation of Sample
Std. Deviation of Mean
Lower 95% CI of mean
Upper 95% CI of mean
22.01.2015
TomTom GO 500
0.0%
37.5 %
0.0%
3.0%
6.0 %
0.6%
1.8%
4.2%
© NavCert
Garmin 2699
0.0%
42.5%
7.1%
9.5%
10.9%
1.1%
7.4%
11.6%
Garmin 3598
0.0%
55.4%
8.1%
11.0%
12.5%
1.3%
8.4%
13.4%
19
Result analysis: statistical evaluation
Trip data was analyzed in an unpaired t test comparing TomTom against the two Garmin
devices. Then a paired comparison between TomTom and one DUT was done. Here only
those trips were analyzed which both TomTom and the other DUT provided evaluable
results. The results are depicted below, where the first three lines are the results from the
unpaired and the lower 6 representing the paired tests. The P value doesn’t differ
between the two approaches. Therefore the significance in the performance between
TomTom and the two Garmin devices is confirmed.
22.01.2015
Unpaired
evaluation of all
results
Dif. btw. means
95% conf. Int.
P value
Paired evaluation
only for paired
evaluable for the
paired results
P value
Number of pairs
Mean of dif.
SD of dif.
SEM of dif.
95% conf. int.
TomTom Garmin 2699
-9.1% ± 1.4%
-9.4% to -8.8%
TomTom - Garmin 3598
-9.2% ± 1.5%
-9.5% to -8.9%
< 0.0001
< 0.0001
< 0.0001
93
-7.4%
14%
1.4%
-10% to -4.5%
< 0.0001
84
-8.3%
16%
1.7%
-12% to -4.9%
© NavCert
20
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
21
Conclusions
An independent benchmark of the performance of the traffic services provided by Garmin
and TomTom was done in Berlin for typical commuter trips in beginning of December 2014.
The test was set up to eliminate all external influences from vehicle, driver, etc.
The delay per trip has been measured and statistically evaluated.
The mean value of the percentaged delay is the smallest with 3% for the TomTom GO 500,
compared to the 6.1% and 7.1% for the two Garmin devices.
TomTom GO 500 outperforms the Garmin 2699 and the Garmin 3598 significantly with
respect to the percentaged delay per trip.
22.01.2015
© NavCert
22
Content
•
•
•
•
•
•
Executive Summary
Who is NavCert
Used Methodology
Data Analysis
Conclusions
Annex
22.01.2015
© NavCert
23
Annex 1: Start / End points of trips
Annex II: Used Vehicles
Vehicle type
VW Touran
105hp, diesel, manual
gearbox
VW Golf Sportsvan
125hp, petrol, manual
gearbox
A6, Mercedes E class,
BMW 5
Automatic gearbox
22.01.2015
Quantity of
vehicles
Number of trips
per vehicle
3
54
12
15
3
3
© NavCert
25
NavCert GmbH
● NavCert GmbH
Hermann-Blenk-Straße 22
D-38108 Braunschweig
Germany
Registered
Amtsgericht Braunschweig
HRB 200 364
VAT ID: DE 249 45 61 89
● Phone: +49 531 35479-0
Fax: +49 531 35479-491
Mobil: 0151 11315420
● Managing Director
Martin Grzebellus
● info@navcert.com
www.navcert.de
22.01.2015
© NavCert
26