FAA-NEXTOR
NAS/ATM Performance Indexes
Dr. Alexander (Sasha) Klein
CENTER FOR AIR
TRANSPORTATION SYSTEMS
Acknowledgments
This research was funded by the FAA-NEXTOR-GMU contract
#DTFAWA-04-D-00013
Many thanks to our FAA sponsors:
• Steve Bradford, Rich Jehlen, Diana Liang – FAA
Also to:
• Stephane Mondoloni – CSSI
• Glenn Roberts - MITRE CAASD
• George Donohue, Lance Sherry – GMU
• Barry Davis, Carlton Wine – FAA; Doug Williamson - Crown
2
CATS
R
Project Objectives
CATS
R
Develop a framework for assessing NAS/ATM performance on a
recurring / daily basis
• Come up with a simple yet informative index of weather-related ATM
performance on a given day (“one number”)
• Produce charts for each season
• Compare different seasons: “Did we do
better this year than last year?”
Account for major external factors:
• Weather
Weather
Traffic
Demand
(Schedules)
Delays
NAS
Procedures
Facilities
• Traffic Demand
Costs
ATM
Enhance existing methods for NAS performance analysis
• Refine computation of the effects of both en-route and
terminal weather
• Consider additional metrics alongside Delay
3
Operational Response Index (ORI), 2004
Components, Excluding Highest-Cost Days
CATS
R
ORI Component Analysis, 2004 (excluding a few highest-cost snowstorm outliers)
ORI:
5-Period Moving Average of $$ Cost
$14,000,000
Direct airline
operating costs
per flight for
an “averaged”
narrowbody
fleet
ORI = Total daily OPSNet cost of
• Excess block time vs. Schedule,
• Excess distance vs. Flight-planned,
• Cancellations, and
• Diversions
per flight
$12,000,000
$10,000,000
Cancellations (light blue)
$8,000,000
All OPSNet
flights daily
$6,000,000
A cost-derived
metric
$4,000,000
Computation
details here
Excess Block Time (pink)
Diversions (yellow)
$2,000,000
Excess Distance (navy blue)
$0
1
21
41
61
81 101 121 141 161 181 201 221 241 261 281 301 321 341 361
Day, from lowest to highest cost
4
1/
1
/2
00
4
5/
20
0
1/
29 4
/2
00
2/
12 4
/2
00
2/
26 4
/2
00
3/
11 4
/2
00
3/
25 4
/2
00
4
4/
8/
20
0
4/
22 4
/2
00
4
5/
6/
20
0
5/
20 4
/2
00
4
6/
3/
20
0
6/
17 4
/2
00
4
7/
1/
20
0
7/
15 4
/2
00
7/
29 4
/2
00
8/
12 4
/2
00
8/
26 4
/2
00
4
9/
9/
20
0
9/
23 4
/2
00
10
4
/7
/2
00
10
4
/2
1/
20
04
11
/4
/2
00
11
4
/1
8/
20
04
12
/2
/2
00
12
4
/1
6/
20
12
04
/3
0/
20
04
1/
1
ORI for 366 Days
01/01 – 12/31/2004, sorted by Date
900
Snowstorms
600
100
5
Volatile Wx in
early spring
Quiet period
CATS
R
ORI, 2004, All days
1,000
Hurricanes and
convective Wx
800
Snowstorms
700
Convective season
500
400
300
200
Somewhat quieter
period
0
Weather-Impacted Traffic Index (WITI)
Combined En-Route and Terminal Wx
En-Route WITI:
• Find intersections of each flow (GC
track) with hex cells where
convective Wx was reported
• Multiply by each hex cell’s total
NCWD count (reflects Wx duration)
and by # of daily flights on this flow
• Add up all flows: En-Route WITI
Terminal WITI:
• Hourly surface Wx observations at major airports
• Capacity degradation % for each Wx type * hourly movement rate
• Add up all airports: Terminal WITI
Combined WITI (CWITI):
• Weighted sum of En-route and Terminal WITI
• Reflects “front-end” impact of Wx on intended flights
6
CATS
R
Combined WITI and NAS Performance Metrics
Example: ORI & Delays, June-Oct 2004 Including Outliers
CATS
R
ASPM Arr Delays and Operational Response Index (ORI) vs. Combined WITI
90000000
Hurricane Frances
9/5
9/4
80000000
70000000
Hurricane Jeanne
9/26
ASPM Arrival Delay
60000000
Hurricane Ivan
9/16
9/3
50000000
Even in ideal
weather, there
are significant
“residual”
delays and
costs
40000000
30000000
ORI
20000000
10000000
0
0
10000000
20000000
30000000
40000000
Combined WITI
7
50000000
60000000
70000000
ORI and Delays vs. Combined WITI
Example 1: Convective Weather
CATS
R
ASPM Arr Delay and ORI vs. Combined WITI
Outliers (Exceedingly high costs on or around hurricane days) are removed
Jun-Nov 2004
90,000,000
July 14, 2004
80,000,000
Very high ORI
($485/flight)
70,000,000
ASPM Arrival Delay
Very high delays
60,000,000
Medium-high WITI
50,000,000
Checking…
40,000,000
30,000,000
20,000,000
10,000,000
ORI
0
0
10,000,000
20,000,000
30,000,000
Combined WITI
8
40,000,000
50,000,000
Zooming In on July 14, 2004…
ORI, En-Route and Terminal WITI
CATS
R
100,000,000
En-route WITI is
high (July…)
90,000,000
Terminal WITI is
low
80,000,000
Combined WITI is
medium-high
E-WITI
70,000,000
60,000,000
WITI A
Terminal WITI
50,000,000
ORI
Combined WITI
40,000,000
Comb-WITI
ORI
30,000,000
20,000,000
High “operational
response cost”
(ORI) was caused
by en-route
thunderstorms
leading to delays
and cancellations
NAS performance
was worse than
usual on this day
10,000,000
T-WITI
71804
71704
71604
71504
71404
71304
71204
71104
71004
70904
70804
0
9
Very high % of
cancellations:
3x the usual
ORI and Delays vs. Combined WITI
Example 2: Non-Convective Weather
CATS
R
ASPM Arr Delay and ORI vs. Combined WITI
Outliers (Exceedingly high costs on or around hurricane days) are removed
Jun-Nov 2004
90,000,000
October 20, 2004
80,000,000
ORI is high but in
line with average
trend
70,000,000
ASPM Arrival Delay
Relatively low
delays
60,000,000
50,000,000
Very high WITI
40,000,000
Checking…
30,000,000
20,000,000
10,000,000
ORI
0
0
10,000,000
20,000,000
30,000,000
Combined WITI
10
40,000,000
50,000,000
Zooming In on October 20, 2004…
ORI, En-Route and Terminal WITI
CATS
R
70,000,000
En-route WITI is
low (late October)
But Terminal WITI
is very high (rain,
low ceilings etc)
60,000,000
T-WITI
50,000,000
So the Combined
WITI is high
40,000,000
Comb WITI
WITI A
Terminal WITI
ORI
30,000,000
Combined WITI
ORI
20,000,000
NAS performance
was actually good
for this “IMC day”
10,000,000
(Better in terms of
delays than costs)
E-WITI
102504
102404
102304
102204
102104
102004
101904
101804
101704
101604
101504
0
11
That is, high ORI
($/flight) and
delays were
caused mostly by
terminal Wx
ORI and Delays vs. Combined WITI
Example 3: Two Metrics Yield Different Results
CATS
R
ASPM Arr Delay and ORI vs. Combined WITI
Outliers (Exceedingly high costs on or around hurricane days) are removed
Jun-Nov 2004
November 23,
2004
90,000,000
80,000,000
High CWITI
70,000,000
High delays but
just average ORI
ASPM Arrival Delay
60,000,000
NAS performance
could be judged
as “poor” if only
delays were
considered
50,000,000
40,000,000
30,000,000
But considering
costs (ORI), it
was about
average given the
weather and the
demand
20,000,000
10,000,000
ORI
0
0
10,000,000
20,000,000
30,000,000
Combined WITI
12
40,000,000
50,000,000
Comparing Delays for 2004 and 2005
May-September
CATS
R
2005 delays were on average about the same as in 2004 (1% diff.)
ASPM Arrival Delays (avg delay for all flights, ASPM 55 airports, vs. schedule)
40
35
2004
2005
30
25
20
15
10
5
But, weather (CWITI) was on average better in May-Sep 2005
(If 2004 average = 100, then 2005 average = 83)
13
8
9/
2
8
9/
1
9/
8
9
8/
2
9
8/
1
8/
9
0
7/
3
0
7/
2
0
7/
1
0
6/
3
0
6/
2
0
6/
1
1
5/
3
1
5/
2
1
5/
1
5/
1
0
Normalized Delay-to-Wx Ratio Comparison
Delay-Based NAS/ATM Performance Index, ‘05 vs.’04
CATS
R
Normalized ASPM Arr Delay vs Weather, May-Sep 2004 and 2005
Against 2004 Trendline (all days, including hurricane-impacted)
350
300
2004 average Delay vs. 2004 WITI = 100
2005 average Delay vs. 2005 WITI vs. 2004 WITI = 120
250
2004
2005
200
150
2005 average
2004 benchmark
100
50
14
4
00
4
25
/2
00
9/
/2
18
9/
/2
00
4
04
11
9/
20
4
4/
9/
/2
00
4
28
8/
/2
21
8/
/2
00
00
4
04
14
8/
20
4
7/
00
/2
31
7/
8/
4
00
4
/2
24
7/
17
/2
00
00
7/
/2
20
4
04
10
7/
4
3/
00
/2
26
7/
4
00
6/
19
/2
00
6/
/2
12
20
4
04
6/
5/
/2
00
4
6/
4
29
5/
22
/2
00
4
00
5/
/2
15
20
04
5/
8/
5/
5/
1/
20
04
0
Delays and Traffic Demand
Taking Exponential Delay-vs.-Demand Factor into Account
CATS
R
Monthly total delays vs operations, Jan 1990 - Aug 2005
3000000
• 2005: a 10% traffic increase
2500000
15
500000
Monthly instrument ops, NAS OPSNet
4.
6E
+0
6
4.
4E
+0
6
4.
3E
+0
6
4.
2E
+0
6
4.
1E
+0
6
4.
0E
+0
6
0
3.
9E
+0
6
• Adjusted chart is shown on
next slide
1000000
3.
7E
+0
6
• The trend doesn’t depend on
weather
1500000
3.
5E
+0
6
• This factor ought to be
taken into account when we
talk about NAS / ATM
performance
2000000
1.
2E
+0
6
• 10% increase in traffic (from
4.3M to 4.7M ops) can lead to a
45% increase in delays (from
1.25M to 1.8M minutes)
Monthly delay total, minutes
• Looking at 1990-2005 historical
monthly averages…
ATM Performance Index, 2005 vs. 2004
Adjusted by Exponential Delay-vs-Demand Factor
CATS
R
2004 benchmark = 100
2005 average adjusted for Weather only = 120
Adjustment factor: divide by 145% (exponential delay increase rate), multiply by 110%
(traffic increase rate; need to pro-rate 2005 back to 2004)
2005 adjusted-for-Weather-and-Demand average = 120 / (1.45 / 1.1) = 91
Normalized ASPM Arr Delay vs Weather, May-Sep 2004 and 2005
Against 2004 Trendline, Adjusted by Delay-vs-Demand Factor (all days, including hurricane-impacted)
350
2005 adjusted average = 91
300
250
2004
2005
200
150
2004 benchmark
100
2005 adjusted average
50
9/25/2004
9/18/2004
9/11/2004
9/4/2004
8/28/2004
8/21/2004
8/14/2004
8/7/2004
7/31/2004
7/24/2004
7/17/2004
7/10/2004
7/3/2004
6/26/2004
6/19/2004
6/12/2004
6/5/2004
5/29/2004
5/22/2004
5/15/2004
5/8/2004
16
5/1/2004
0
Discussion
Delays
CATS
R
Did the NAS/ATM do 9% better in 2005 than in 2004?
•
NAS delays were similar; delays vs. weather were worse in 2005…
•
But, relative to weather and traffic demand, the ATM component of
the NAS did do better in 2005 than in 2004
•
D-RVSM and other measures may have helped
3000000
Even so,
•
17
We are on the ascending slope of the
exponential delay curve
Peak delays in bad weather (July 2005)
were highest ever
•
Delay variance is significant
•
The exact proportion (45% delay
increase due to 10% traffic demand
growth) needs to be fine tuned
We are here
Monthly delay total, minutes
•
2500000
2000000
1500000
1000000
500000
0
Monthly instrument ops, NAS OPSNet
ORI: Cost-to-Wx Ratio
The Ruler for ORI (100) is an Averaged Day
CATS
R
3.00
300
Normalized Indices, Jun-Sep 2004
"Cost", "Weather", and "NAS Cost-to-Wx Index", normalized (Current-day / Average)
250
2.50
Outliers (hurricane-impacted days) removed
Standard Deviation: Cost 0.22, Weather 0.4, NAS Cost-to-Wx Index 0.6
200
2.00
Normalized Cost (ORI)
Normalized Wx
150
1.50
100
1.00
50
0.50
300
000
0.00
60
60104
61104
62104
70104
71104
72104
-50
250
50
-0.50
-100
200
00
-1.00
Normalized-Cost-to-Normalized-Wx Ratio
(is roughly inversely proportional to ORI)
-150
150
50
-1.50
-200
100
00
-2.00
-250
50
50
-2.50
18
-300
000
-3.00
6
Values below 100 are good
High peaks are bad
73104
81004
82104
83104
91404
9300
Conclusions
CATS
R
Delay, cost (ORI) and weather (WITI) metrics computed for
2004 and 2005
• Delay metric can be normalized vs. seasonal-average (e.g. 2004’s)
• Normalized cost (ORI) is a useful additional metric
WITI calculation refined for both en-route and terminal parts
Delay/Cost metrics should account for traffic demand, not just
weather, if used as NAS/ATM performance indicators
• 10-15% traffic demand increase can cause 45-60% increase in delays
• Slightly better NAS/ATM performance in 2005 if both weather and
traffic demand are taken into account
These metrics can advance our understanding of NAS
response to external impacts
19
NAS Response to External Impacts
Traffic Demand
NAS
Weather
Impact
Excess Demand
vs. Capacity
Unavoidable
Unavoidable
Avoidable
NAS Response
(Delays, Costs)
Inefficiencies
1)
2)
20
What portion of
delays/costs is due to
system inefficiencies as
opposed to unavoidable
weather and traffic demand
outside ATM’s control?
Can we quantify positive
impact of NAS/ATM
efficiency improvements?
ATM
NAS/ATM Efficiency
Improvements
Excess D vs. C
Unavoidable
Inefficiencies
CATS
R
CATS
R
Back-up Slides
21
Operational Response Index (ORI)
Components
CATS
R
Using direct carrier costs only
•
Passenger impact (value of pax time, ‘ill will’, re-issuing tickets etc) excluded
Flights per day: OPSNet daily totals (varies between 37,000 and 50,000)
•
Simplifying assumption: all aircraft are narrowbodies
Cost of 1 Minute of Delay
•
Used $22/min (based on total non-fuel operating costs averaged for a narrowbody jet)
Cost of 1 Extra Mile Flown (expressed in $/min)
•
Equivalent to $18/min (based on 2004 fuel cost average for a narrowbody in cruise at
$1.25 / gallon)
Cost of a [Narrowbody] Cancellation
•
US carrier-reported average cost was $4,500 in ’94 which equates to $6,000 per
cancellation in 2004
Cost of a [Narrowbody] Diversion
•
Assuming 4 hrs extra block time and a $2,500 hourly operating cost for a narrowbody,
we get $10,000 per diversion
Sources: OIG; BTS; MITRE CAASD; FAA APO; FAA OPSNet database
22
Operational Response Index (ORI)
Calculation
ORI =
CATS
R
(Num_fl * Avg ∆dist * Avg_fuelburn * Fuel_cost +
Num_fl * Avg ∆time * Avg_nonfuel_oper_cost
Num_diversions * Avg_cost_of_diversion +
Num_Cancellations * Avg_cost_of_cancellation) / Num_fl
where:
∆dist = average excess distance per flight (actual vs. flight-planned)
∆time = average excess block time per flight (actual vs. scheduled)
Avg_fuelburn = fuelburn for a generic narrowbody jet in cruise at FL330
Num_fl = daily number of OPSNet flights
Sources of data: FAA APO Lab; FAA ASPM
23
Operational Response Index – Reality Check
Comparison with $$ Quoted in Literature
CATS
R
For OPSNet flights:
• Total “excess airline cost” for 2004 (all flights, all days) is $4.6B
• For a baseline “ideal” day (ORI = $150/flight):
if all days in 2004 were like it, total cost would have been $2.7B
Difference = $1.9B in direct operating costs
References show comparable excess-cost estimates:
24
•
“Some figures have indicated that the total average direct annual costs of the irregular
operations of ten U.S. major airlines for the period 1996-1999 have been about $1.9B”
(M.Janic – TRB Report, 2003)
•
“…the Air Transport Association estimated that delays cost the air carriers approximately
$2.0B in direct operating costs in 1999”: OIG Report, 2000
•
“The Air Transport Association's amount increases to nearly $5B when indirect costs and
the value of passengers' lost time are included”: OIG Report, 2000 (extrapolation of our
calculations to include indirect costs produces comparable numbers – AK)
•
Total operating costs of delays: $1.8-2.4B in 1987-94: FAA APO-130, “Total Cost for Air
Carrier Delay Report”, 1996
•
1999 total cost of disruptions estimated at $1.8B (Z.Shavell – Effects of Schedule
Disruptions on the Economics of Airline Operations. In: Air Transportation Systems
Engineering, 2001, Chapter 8).
2005 Delay/Wx: Linear vs. Exponential Trend
A Sign of a Worsening Delay Situation?
CATS
R
Exponential trendline: a better fit for 2005 Delay-vs-Weather Plot?
Linear trendline is a better fit for 2004 data
ASPM Arr Delay vs CWITI, 2004-2005 May-Sep
Excluding hurricane-impacted days 0.0078x
ASPM Arr Delay vs CWITI, 2004-2005 May-Sep
Excluding hurricane-impacted days
y = 47.823e
250
y = 0.832x + 30.734
A rr D e la y p e r F lig h t, N o rm a lize d to 2 0 0 4
a v e ra g e
A rr D e lay p e r F lig h t, N o rm a lize d to 20 0 4
a ve rag e
250
200
150
100
50
Linear trendline
200
150
2005
Linear (2005)
100
0
2005
Expon. (2005)
50
Exponential trendline
0
0
50
100
150
CWITI, Normalized to 2004 average
25
200
250
0
50
100
150
CWITI, Normalized to 2004 average
200
250
“Quiet Period” Monthly ASPM Delays vs. Ops
1995-2005
CATS
R
10-Year Trend: NAS Delays vs Ops, "Quiet months"
(Mar,Apr,Oct,Nov)
3200000
3000000
2800000
Total monthly delay minutes vs. Schedule
2600000
2400000
2200000
2000000
1800000
1600000
1400000
1200000
1000000
800000
600000
400000
200000
38
87
0
39 61
19
3
39 73
49
7
40 38
14
1
40 68
43
1
40 93
76
2
40 75
97
8
41 86
17
9
41 45
30
8
41 77
66
9
41 76
97
4
42 72
31
0
42 55
54
2
43 36
06
8
43 68
31
3
43 62
58
7
43 76
90
8
44 65
51
5
44 86
99
0
45 93
71
3
46 69
62
78
9
0
Monthly ops
26
Terminal Capacity Degradation
Weather factor
Available airport capacity, % nominal
•
THUNDERSTORM
10
•
HEAVY_SNOW
30
•
HIGH_WIND (>30 kt)*
30
•
HEAVY_RAIN
40
•
LOW_VISIBILITY
70
•
LOW_CEILING
70
•
SNOW
70
•
RAIN
70
•
WIND (20-30 kt)
70
•
NO_WEATHER
100
*sustained wind above 30 kt, higher gusts
27
CATS
R
“Flows” and Actual Tracks
Similarity
28
CATS
R