Proposed
Performance Metrics
Block Time & Predictability
Marc Rose, MCR/SETA-II
Gabriela Rohlck, MCR/SETA-II
15 March, 2006
Presented at
NEXTOR - Asilomar
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
1
Objective
¾ An evaluation of two specific metrics: Scheduled
Block Time and Predictability
¾ A general discussion of types of metrics
¾ This work has been sponsored by two ATO
Planning offices:
1) The Office of Strategy
2) The Office of Performance Analysis
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
2
Three Primary Types of Metrics
¾ System Activity or Workload
– Number of flights per month, overall average block time
– Provides an estimate of workload, generally NOT performance
– Impacted by non-performance based changes
• Avg Flight length - more longer flights, not performance related
¾ Performance
– Generally a system activity metric in constrained times
– Normalized block time, arrival delay (sometimes)
¾ Diagnostic
– Needed to understand changes in performance
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
3
Three Primary Types of Metrics
ALL metrics have flaws/limitations:
Identification of these limitations is vital to understanding
and proper usage
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
4
Data and Definition
¾
Data Sources
– Airline Service Quality Performance (ASQP)
• Data available from 1/1995 to 12/2005
– Aviation System Performance Metrics (ASPM)
• Data available from 1/2000 to 12/2005
– Official Airline Guide (OAG)
• Data available from 1/1982 to 9/2005
¾
Definition:
– Block Time: Airborne+Taxi-Out+Taxi-In (Gate-Out to Gate-in)
– Total Flight Time: Block Time + Departure Delay
• Or: Actual arrival (in) - Scheduled Departure
– Scheduled: Airline published departure/arrival times
• Based on OAG or ASPM data (pre departure)
– Actual: departure/arrival times as flown (post departure)
• Based on ASQP or ASPM data
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
5
Scheduled Block Time
¾
¾
A Performance Metric (a lagging indicator)
Metric based on OEP to OEP flights
–
¾
¾
Considered the “NAS Level” (All* flights) and the OEP-OEP
level (only flights to/from an OEP 35 airport)
Used ASPM data, limited to those flights that have OAG data
–
–
¾
A current FAA Executive Council decision
OAG=“Y” flag
Does not include cancelled flights
Two metrics proposed
1) “Normalized” average block time
–
Normalization consists of identical set of O-D pairs weighted using
the average number of flights/month across all datasets
2) Delta metric - A relative change from the overall average
*Not truly ALL flights – as will be seen in the development
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
6
Normalization of Block Time
Mathematical Details
Two intermediate values are calculated
1) the mean time µ
2) the average number of flights n̂ij
µijk =
∑x
Where x is the block time for flight l
for O-D pair ij within month k and
n is the number of flights for O-D pair
Within all (majority*) of data sets
ijkl
l
nijk
N
nˆij =
∑n
k =1
ijk
Where N is the number of months of data
N
Combining into a normalized NAS level “Performance” measure:
∑ µ nˆ
=
∑ nˆ
ijk
µk
i, j
ij
ij
Where ij is an O-D pair,
and k is month/year
i, j
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
7
nAp 00
r-0
Ju 0
l-0
O 0
ct
J a 00
nAp 01
r-0
Ju 1
l-0
O 1
ct
J a 01
n0
Ap 2
r-0
Ju 2
l-0
O 2
ct
J a 02
nAp 03
r-0
Ju 3
l-0
O 3
ct
J a 03
nAp 04
r-0
Ju 4
l-0
O 4
ct
J a 04
n0
Ap 5
r-0
Ju 5
l-0
O 5
ct
-0
5
Ja
Normalized Scheduled
Block Time (OEP to OEP)
150
149
148
147
146
145
FEDERAL
Moving average (ttm)
144
143
142
141
140
Source:
ASPM
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
8
80
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
Jan-05
Jan-04
Jan-03
Jan-02
Jan-01
Jan-00
Jan-99
Jan-98
Jan-97
Jan-96
Jan-95
Jan-94
Jan-93
Jan-92
Jan-91
Jan-90
Jan-89
Jan-88
Jan-87
Jan-86
Jan-85
Jan-84
Jan-83
Jan-82
Normalized Average Block Time (min)
Normalized Scheduled
Block Time (NAS Level)
105
100
95
Moving average (ttm)
90
Easy to see missing data
85
Source: OAG
ORGANIZATION
9
Scheduled Block Times A
Lagging Metric
1.05
160
Relative
Changing
Flight
Times (min)Blocktime
1.04
155
Scheduled
Actual
Total Flight Time (ttm)
1.03
150
1.02
1.01
Scheduled Block Time (ttm)
145
1
140
Actual Block Time (ttm)
0.99
135
0.98
0.97
D
ec
-9
5
Ap
r9
Au 6
g96
D
ec
-9
Ap 6
r9
Au 7
g97
D
ec
-9
Ap 7
r9
Au 8
g9
D 8
ec
-9
Ap 8
r9
Au 9
g9
D 9
ec
-9
Ap 9
r0
Au 0
g00
D
ec
-0
Ap 0
r0
Au 1
g01
D
ec
-0
Ap 1
r0
Au 2
g02
D
ec
-0
Ap 2
r0
Au 3
g03
D
ec
-0
Ap 3
r0
Au 4
g04
D
ec
-0
Ap 4
r05
130
2000
2001
2002
2003
2004
2005
Source: ASPM, OAG and ASQP
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
10
Delta Metric – Another Approach
nijk
µijk =
∑x
l =1
Where x is the block time for flight l
and O-D pair ij within month k and
n is the number of flights
ijkl
nijk
N
µˆ ij
∑µ n
=
∑n
ijk ijk
k =1
A multi-month average block time
ijk
k
∑ (µ µˆ
=
∑n
ijk −
δk
ij
ij
ijk
)nijk
Which is a relative performance
metric for the block time that includes
ALL flights
ij
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
11
Scheduled Block Time
Delta Metric (OEP-OEP)
2.5
2
1.5
Minutes
1
0.5
0
-0.5
-1
-1.5
-2
5
l-0
Ju
Ja
n05
4
l-0
Ju
Ja
n04
3
l-0
Ju
n03
Ja
2
l-0
Ju
n02
Ja
1
l-0
Ju
n01
Ja
0
l-0
Ju
Ja
n00
-2.5
Data Source: ASPM
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
12
Ja
n0
A 0
pr
-0
Ju 0
l-0
O 0
ct
-0
Ja 0
n0
A 1
pr
-0
Ju 1
l-0
O 1
ct
-0
Ja 1
n0
A 2
pr
-0
Ju 2
l-0
O 2
ct
Ja 02
n0
A 3
pr
-0
Ju 3
l-0
O 3
ct
-0
Ja 3
n0
A 4
pr
-0
Ju 4
l-0
O 4
ct
-0
Ja 4
n0
A 5
pr
-0
Ju 5
l-0
O 5
ct
-0
5
Minutes
Scheduled Block Time
Delta Metric (NAS Level)
2
1.5
1
0.5
0
-0.5
-1
-1.5
-2
Data Source: ASPM
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
13
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
Jan-05
Jan-04
Jan-03
Jan-02
Jan-01
Jan-00
Jan-99
Jan-98
Jan-97
Jan-96
Jan-95
Jan-94
Jan-93
Jan-92
Jan-91
-2
Jan-90
Jan-89
Jan-88
Jan-87
Jan-86
Jan-85
Jan-84
Jan-83
Jan-82
Block Time Changes (min)
Historical Scheduled Block Time
Delta Metric (NAS Level)
6
4
2
0
Moving average (ttm)
-4
-6
-8
Data Source: OAG
ORGANIZATION
14
-4
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
Jan-05
Jan-04
Jan-03
Jan-02
Jan-01
Jan-00
Jan-99
Jan-98
Jan-97
Jan-96
Jan-95
Jan-94
Jan-93
Jan-92
Jan-91
Jan-90
Jan-89
Jan-88
Jan-87
Jan-86
Jan-85
Jan-84
Jan-83
Jan-82
Scheduled Block Time Changes
Accounting for Aircraft Type
3
2
1
0
-1
-2
-3
Data Source: OAG
ORGANIZATION
15
Total Flight Time Variability
a Predictability Metric?
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
16
Percentiles as a
Predictability Metric
•
Start at origin-destination level:
– How does measured block time of airport-pair (OD) differ from average time
where:
M k = xij − µij
xij is measured time for O-D pair (i, j)
µij is mean time for O-D pair (i, j)
• Flight-level Mk is generated for all flights and the
Selected percentiles are output as a NAS level metric
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
17
NAS Predictability
Performance Metric
(Total Flight Time – includes gate delay)
60
Predictability Metric (min)
50
40
30
20
10
0
-10
-20
-30
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Source: ASQP
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
18
Predictability Metric
OEP Arrival Airports – Sep/2005
50
40
30
20
10
0
-10
-20
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
TPA
STL
SLC
SFO
SEA
SAN
PIT
PHX
PHL
PDX
ORD
MSP
MIA
MEM
MDW
MCO
LGA
LAX
LAS
JFK
IAH
IAD
HNL
FLL
EWR
DTW
DFW
DEN
DCA
CVG
CLT
CLE
BWI
BOS
-40
ATL
-30
ORGANIZATION
19
Observations
•
Using 10th, 90th percentiles
–Eliminate significant outliers
–Analysis shows distribution right-skewed.
•Average closer to lower bound than upper bound.
–Metric defined as difference between lower, upper
percentiles.
•Can be calculated with one month of data.
–Does not require historical data.
•
Seasonality
–This Metric partially accounts for known seasonality
(e.g., winter headwinds) by using current months
averages rather than historical
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
20
Backup
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
21
Metric with Normalized
Average Flight Time
100
90
140
80
120
Metric (min)
70
100
60
50
80
40
60
30
40
20
Average Total Flight Time (min)
160
20
10
0
0
Jul-05
TRAFFIC
Jan-05
Jul-04
Jan-04
AIR
Jul-03
Jan-03
Jul-02
•
Jan-02
Jul-01
ADMINISTRATION
Jan-01
Jul-00
Jan-00
Jul-99
Jan-99
Jul-98
Jan-98
AVIATION
Jul-97
Jan-97
Jul-96
Jan-96
Jul-95
Jan-95
FEDERAL
ORGANIZATION
22
Actual Block Time Variability:
Percentiles and Weighted Average
(10th and 90th percentiles)
160
Block time (min)
150
140
130
120
110
TRAFFIC
2005
2004
AIR
2003
•
2002
1999
1998
1997
ADMINISTRATION
2001
AVIATION
2000
FEDERAL
1996
1995
100
ORGANIZATION
23
Predictability Metric
Calculation - Example
Origin
A
A
A
A
C
C
C
C
C
C
E
E
E
E
E
E
E
Destination Flight Time Avg Flt tm
B
90
98.0
B
110
98.0
B
92
98.0
B
100
98.0
D
97
81.8
D
67
81.8
D
72
81.8
D
101
81.8
D
80
81.8
D
74
81.8
F
200
174.9
F
202
174.9
F
160
174.9
F
145
174.9
F
170
174.9
F
180
174.9
F
167
174.9
Delta
-8.0
12.0
-6.0
2.0
15.2
-14.8
-9.8
19.2
-1.8
-7.8
25.1
27.1
-14.9
-29.9
-4.9
5.1
-7.9
Sorted Delta
-29.9
-14.9
-14.8
-9.8
-8.0
-7.9
-7.8
-6.0
-4.9
-1.8
2.0
5.1
12.0
15.2
19.2
25.1
27.1
Metric = -14.8 to +21.5
FEDERAL
AVIATION
ADMINISTRATION
•
AIR
TRAFFIC
ORGANIZATION
24