Estimating Sources of Temporal
Deviations from Flight Plans
Ms. Natasha Yakovchuk (yakovn2@rpi.edu)
Prof. Thomas R. Willemain (willet@rpi.edu)
Department of Decision Sciences and Engineering Systems
Rensselaer Polytechnic Institute
Sponsored by Free Flight Program, Federal Aviation Administration
Contract # DTFA01-98-00072
Agenda
§ FAA interests: system predictability, assessing interventions
§ Problem introduction
§ Proposed methodology
§ Evaluation of alternative estimation methods
§ System implementation in SAS for annual and next-day reporting
§ Ongoing research
− Operational validation
− Statistical analysis of results
2
Motivation and FAA sponsorship
§ Client: Free Flight Program of the Federal Aviation
Administration
§ Need: improve system predictability and decrease unexpected
flight delays
§ More specifically: trace flight delays to their sources, and
quantify them
§ Intended use: next-day and annual reporting, special studies
§ Potential use: evaluating impact of FAA initiatives
3
Problem introduction
Average daily airtime for flights from
Atlanta to Boston for year 2001
150
airtime
140
ç Average airtime fluctuates
(due to winds aloft, weather
and congestion in airports,
etc.)
130
ç Flight plans anticipate
“normal problems”
120
110
100
date
ê Shift attention to
Deviation = Actual Airtime – Flight Planned Time
2 types of Deviations: “ETE” and “G2G”
4
Problem introduction: Deviations
Deviations from flight plans for flights
from Atlanta to Boston for year 2001
deviation from flight plan
20
ç Deviations from flight plans
measure unanticipated
problems, or “surprises”
ç Common factors for different
flights are considered as
systemic sources of deviations
10
0
-10
-20
date
ê Decompose deviations into four
sources:
System + Origin airspace + Destination airspace + En route airspace
5
FAA data as a two-way table
Destination airport
§ 31 major US airports
Origin airport
§ Each table represents one day of
operations
§ Each cell contains an average
deviation from flight plan
§ One observation per cell,
averaging over multiple flights
§ Data available for January ’01March ’03
§ Presence of structural ‘holes’
structural ‘holes’
6
Fragment of the table
Origin airport
Destination airport
7
Row + Column Analysis
en route effects
(residuals)
destination
origin
1
2
…
1
y11 y12
…
y1m
2
y21 y22
…
y2m
…
…
n
yn1 yn2
1
m
…
linear additive
model
yij = µ + αi + β j + εij
1
2
2
…
ε11 ε12
ε21 ε22
n
destination
effects
m
…
…
ε1m
ε2m
…
…
ynm
origin
effects
εn1 εn2
β1
β2
α2
…
…
…
εnm
αn
βm
µ
µ = system effect (e.g., September 11th)
αi = origin effect (e.g., restricted departure gates)
β j = destination effect (e.g., fog)
εij = en route effect (e.g., convective weather, MIT, circular holding)
8
α1
system
effect
Which estimation method to use?
Ø Methods:
§ Ordinary Least Squares
§ Least Absolute Deviations (LAD)
§ Median Polish
Ø Full factorial design:
§ Factors (at 3 levels each):
− table size
− percentage of holes
− percentage of outliers
§ Responses (comparison criteria):
− accuracy of estimates (RMSE and MAE for effects)
− outlier detection capability (sensitivity and specificity)
9
Modeling FAA data
Normal probability plot for BWI:IAD effect
§
Can use estimates from LAD
§
Generate origin, destination and
en route effects independently
ç Most effects can be modeled by
N(µ, σ2) after removing outliers
Scatterplot of µ against σ for en route effects
ç µ and σ of effects can be modeled
independently
10
§
µ is modeled by Normal
§
σ is modeled by Lognormal
Major findings
Root mean square error for en route effects,
10% outliers
rmse_term
rmse_enroute
1.32
1.28
1.26
0.94
1.0
0.92
0.9
0.90
0.88
1.24
0.86
1.22
0.84
LAD
least
squares
sensitivity
1.34
1.30
Sensitivity for enroute effects,
10% outliers
Root mean square error for terminal effects,
10% outliers
median
polish
0.8
0.7
0.6
0.5
Benchmark
LAD
least
squares
median
polish
LAD
least
squares
§
All error measures are on the order of only one minute for all three methods !
§
Since FAA data have up to 10% outliers, we choose resistant methods (better
in accuracy and outlier detection capability)
11
median
polish
§
LAD is slightly better in estimating terminal effects than median polish
§
Choose LAD for estimation
System implementation for the FAA
A turnkey system implemented in SAS that produces:
§ Next-day estimates of effects
§ Map-based displays
§ Statistical graphics
§ Datasets for use in one-off statistical studies
12
Timeplot of system effects, 2001
Timeplot of system effect for year 2001
Computations based on ETE
13
Boxplots of destination effects, 2001
PHL
SFO
LGA
14
LAS
Map of destination effects
15
Map of en route effect outliers
16
Ongoing research #1: Operational validation
Ø Validate the results against other databases:
§ Aviation System Performance Metrics (ASPM)
§ Operations Network (OPSNET)
§ Post Operations Evaluation Tool (POET)
§ Strategic Plans of Operation (SPO)
§ National Oceanic and Atmospheric Administration (NOAA)
Ø Conduct at two levels:
§ Macroscopic validation (compare statistics for a certain time period)
§ Microscopic validation (detailed validation for selected days)
17
Macroscopic validation: ASPM
percentage of late arrivals
airport: ORD
80
60
40
20
0
-20
-10
0
10
20
30
20
30
destination effect
percentage of late arrivals
airport: SFO
80
60
40
20
0
-20
-10
0
10
destination effect
18
ç Strong correlation between
destination effects (calculated from
G2G data) and ASPM percentage of
late arrivals (Jan’2001-March’2003)
Microscopic validation: Weather
February 15, 2001
19
Ongoing research #2: Statistical analysis of effects
Objective: Use the estimated effects to study the NAS
ê Origin versus destination
effects for LAX (G2G):
negative correlation
ê Origin versus destination
effects for MEM (G2G):
no correlation
airport: MEM
20
15
10
10
dest.effect
dest.effect
airport: LAX
0
-10
0
-20
-5
-20
-10
0
orig.effect
20
5
10
-5
0
5
10
orig.effect
15
20
Statistical analysis of effects
ê Heteroscedasticity
(ETE)
Destination effects for LGA (year 2001)
ê Positively correlated origin effects
(G2G)
las
15
10
5
lax
0
-5
san
-10
-15
-20
21
phx
Acknowledgements
Ø FAA/Free Flight Program
§ Mr. Dave Knorr
§ Mr. Ed Meyer
Ø CNA Corporation
§ Mr.Joe Post
§ Mr.James Bonn
Ø Metron Aviation
§ Dr. Bob Hoffman
22