MIT
International
MIT
ICAT
Center
for
Air
Transportation
Standard Flow Abstractions as Mechanisms
for Reducing ATC Complexity
Jonathan Histon
May 11, 2004
MIT
ICAT
Introduction
y
Research goal: Improve our understanding of
complexity in the ATC domain.
y
Complexity represents a limiting factor in ATC
operations:
† Limit sector and system capacity to prevent controller
“overload.”
y
ATC environment is extremely structured:
† Standardized procedures
† Division of airspace into sectors
† ATC preferred routes
y
Structure is believed to be an important
influence on cognitive complexity.
† Not considered in current metrics.
y
Research Question:
† What is the relationship between this structure and
cognitive complexity?
Not quite right - I need to iterate on this more....
Picture From
FlightExplorer
Software
2
MIT
ICAT
Previous Work:
Structure-Based Abstractions
y
Standard Flows
† Aircraft classified into
standard and nonstandard classes based
on relationship to
established flow
patterns.
y
Groupings
† Common, shared
property, property can
define non-interacting
groups of aircraft
o E.g. non-interacting
flight levels
y
Critical Points
† Sector “Hot Spots”
† Reduce problem from
4D to 1D “time-ofarrival”.
Standard
flow
Non-standard
aircraft
Grouping
Critical
point
Standard
flow
Standard
aircraft
Sector
boundary
3
MIT
ICAT
Example Basis for
Standard Flow Abstraction
> 0.5
Density Map, Utica Sector (ZBW), October 19, 2001 4
MIT
ICAT
Mechanisms of Structure
y Hypothesis: structure-based abstractions reduce cognitive /
situation complexity through reducing “order” of problem space
y Where “order” is a measure of the dimensionality of the problem
y Example:
1 D Problem Space
(T)
2 D Problem Space
( X, T )
“Point” Scenario
“Line” Scenario
3 D Problem Space
(X, Y, T )
“Area” Scenario
5
MIT
ICAT
Experiment Task
y Observe ~ 4
minutes of
traffic flow
through
“sector”
y Monitor for
potential
conflicts
y When
suspect
conflict,
pause
simulation
and identify
aircraft
involved
6
MIT
ICAT
Experiment Design
y Independent Variable
† 3 Levels of “problem dimensionality”
o “Area”
o “Line”
o “Point”
y Dependent Variables
† Time-to-Conflict when detected
† Detection accuracy
† Subjective questionnaires
y Within Subjects design
Conflict: Point
Line
Area
C1
C2
C3
C4
C5
C6
† 6 conflicts (trials) per level of independent variable
† Scenario for each level of independent variable
o All conflicts for each level occurred within the scenario
† Order of scenarios counterbalanced
7
MIT
ICAT
Equivalency of Levels of
Independent Variable
y In order to evaluate hypothesis, scenarios should be as similar
as possible
y Scenario design established general similarity:
† Same aircraft rate (~ 6.5 aircraft / minute / flow)
† Same range of # of aircraft on screen (6-12 aircraft)
† Similar range of # of aircraft on screen when conflict occurred
o Point: 9 +/- 1
o Area: 9 +/- 2
o Line: 9 +/- 2
8
MIT
ICAT
19 Participants
y Predominantly students
† 2 Air Traffic Control Trainees from France
y Predominantly male (80%)
y Age ranged from 23 – 42
y Few participants regularly play computer games (27%)
† Most never played ATC simulations (71%)
9
MIT
ICAT
Both
Aircraft
Visible
Primary Dependent Variable:
Time-to-Conflict
User
Identifies
Conflict
Time-toConflict
Conflict
Occurs
Time
10
MIT
ICAT
y
y
Computed
average
Time-toconflict per
scenario for
each subject
ANOVA is
significant at
p < 0.00002
Follow-up
two-tailed ttests
indicate all
differences
statistically
significant at
p < 0.002
10.0
T im e -to -C o n flic t (s e c )
y
Conflicts are Identified Earlier in “Point”
and “Line” Scenarios
8.0
6.0
4.0
2.0
0.0
Point
Line
Area
11
MIT
ICAT
Time-to-Conflict Distributions
y Peak in “Line” condition clearly earlier than for “Area”
y “Point” condition much flatter
† Sharp drop indicative of attention capture?
25%
Point
Line
Area
15%
10%
5%
% of Conflicts
20%
0%
10.0
8.0
6.0
4.0
Time-to-Conflict (sec)
2.0
Missed
12
More Errors Occurred
in “Area” Scenario
MIT
ICAT
Missed detections occurred
primarily in the “Area” Scenario
y
Incorrect identifications
occurred primarily in the “Area”
Scenario
15%
2.50
Incorrect Conflicts
(per Scenario)
% of Conflicts Missed
y
10%
5%
0%
Point
Line
Area
2.00
1.50
1.00
0.50
0.00
Point
Line
Area
13
MIT
ICAT
Subjects are Least Comfortable
Identifying Conflicts in “Area” Scenario
Did you feel you were able to
comfortably identify all conflicts in
the scenario?
Average Comfort Level
“Very Comfortable”
“Not Very Comfortable”
5.0
4.0
3.0
2.0
1.0
Point
Line
Area
14
MIT
ICAT
Most Subjects Identified Point
Scenario as Easiest
Which scenario did you find it easiest
to identify conflicts in?
% of Subjects
67%
33%
0%
Point
Line
Area
All Same
15
MIT
ICAT
Subject Comments
y “Think aloud” protocol
† Pair-wise comparisons
† Grouping / Standard flow indicators
o “gap”, “between them”, “through here”
y What made the hardest scenario difficult?
† “Lack of predetermined routes … Lack of intersection points between
possible routes”
† “Multiple horizontal streams - gives multiple intersection venues. Hard to
memorize them and monitor them continuously”
y What made the easiest scenario easier?
† “The intersecting stream structure made it simpler to do.
† …Simultaneous near collisions were not possible, so I could pay more
attention to the aircraft with near-term possible conflicts.”
16
MIT
ICAT
Two Issues Probed Further
y Possible Learning Effect Due to Design of Training
y Characteristics of Individual Conflicts
17
MIT
ICAT
Training Issue
y Previous results encompass entire population of subjects
y Initial group of 6 showed some possible learning effects:
80%
60%
40%
20%
0%
First
Middle
Last
All
Same
Position of Easiest Scenario
Average Comfort
Level
% of Responses
† Easiest scenario usually identified as “last” scenario
† Average comfort level slightly higher in last scenario
† User comments strongly suggesting easiest scenario was easier
because of experience
5.0
4.0
3.0
2.0
1.0
First
Middle
Last
Scenario Position
18
MIT
ICAT
Modifications to Training
y Created new training scenarios:
† Subjects trained on 14 conflicts (increase from 4)
† Subjects completed 2 complete practice scenarios (increase from 0)
† Exposed to subjects to all conditions (vs. only point condition)
% of Responses
80%
Training 1
Training 2
60%
40%
20%
0%
First
Middle
Last
All Same
Position of Easiest Scenario
Average Comfort
Level
y New training appears to have changed perceived training
effect:
5.0
Training 1
Training 2
4.0
3.0
2.0
1.0
First
Middle
Last
Scenario Position
19
MIT
ICAT
Little change on Time-toConflict performance:
10.0
Training 1
y
Training 2
8.0
6.0
4.0
2.0
0.0
First
M iddle
Last
Time-to-Conflict (sec)
Time-to-Conflict (sec)
y
Effect on Performance
Exposure to Line and Area in
training appears to have
decreased performance
10.0
Training 1
Training 2
8.0
6.0
4.0
2.0
0.0
Point
Line
Area
20
MIT
ICAT
Characteristics of Conflicts:
Conflict Exposure Time
Time-to-Conflict
Time
Both Aircraft Visible
User Identifies Conflict
Conflict Occurs
“Conflict Exposure”
21
MIT
ICAT
Conflict Exposure Times
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
POINT
LINE
AREA
0.0
10.0
20.0
30.0
Conflict Exposure Time (sec)
40.0
22
MIT
ICAT
Comparison of “Quick”
Conflicts ( < 7 sec)
Time-to-Conflict (sec)
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Line
Area
23
MIT
ICAT
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
Differences Between “Quick” Line
and “Area” Reflected in Error Data
5%
0%
5%
POINT
5%
0%
0%
0%
0%
LINE
5%
5%
11%
0%
11%
21%
AREA
5%
26%
0%
0%
5%
10%
20%
% of Conflicts Missed
30%
40%
24
MIT
ICAT
Selected only those conflicts
with Conflict Exposure Times of
20 +/- 5 sec
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
C1
C2
C3
C4
C5
C6
POINT
LINE
AREA
0.0
10.0
20.0
30.0
Conflict Exposure Time (sec)
40.0
y
Time-to-Conflict (sec)
y
Variance of Conflict Exposure Time
Does Not Change Fundamental Result
ANOVA still significant at
p < 0.005
10.0
8.0
6.0
4.0
2.0
0.0
Point
Line
Area
25
MIT
ICAT
Challenges and
Insights
y Display design issues:
† Overlapping data tags
† Effect of choice of
separation standard
y Experiment design
issues:
† Importance of pilot testing
through statistical analysis
† Scenario design is
difficult!
y Establishing
“equivalency” of
scenarios provides
insight into
characterizing
complexity
† Categorizing aircraft
based on point of closest
approach
26
MIT
ICAT
Summary
y Results support hypothesis that problem spaces of fewer
dimensions reduce complexity
† Performance
† Subjective assessments
† User comments
y Identified and addressed potential learning effect
27
MIT
International
MIT
ICAT
Center
for
Air
Transportation
Backup Slides
MIT
ICAT
15 Participants
% of Subjects
% of Subjects
100%
80%
60%
40%
20%
100%
75%
50%
25%
0%
Yes
0%
Male
Have You Ever Played any
ATC Simulation Games?
Female
% of Subjects
% of Subjects
Gender
45%
30%
15%
0%
<25
25 29
Age
30 34
35 39
40 44
No
80%
60%
40%
20%
0%
Never
From M onthly At Least Several
Once a Times a
Time-toWeek
Week
Time
Daily
How Often Do You Play
Computer Games?
29
# of Subjects
MIT
ICAT
ATC Experience?
10
8
6
4
2
0
None
Slight
Fairly
Very
Familiar
Controller
How Familiar with ATC Concepts and
Typical Operating Procedures Are You?
30
Differences Clearer in
Cumulative Distributions
MIT
ICAT
y How many conflicts were identified by “at least” this much time
prior to the conflict?
75%
50%
25%
% of Conflicts
100%
Point
Line
Area
0%
10.0
8.0
6.0
4.0
2.0 Missed
Time-to-Conflict (sec)
31
In Line, Quick Conflict is
Unremarkable
MIT
ICAT
100%
90%
80%
Quick
Conflict
60%
50%
40%
30%
% of Conflicts
70%
Line
Line
Line
Line
Line
Line
- C1
- C2
- C3
- C4
- C5
- C6
20%
Shorter
Conflict
10%
0%
10.0
8.0
6.0
4.0
Tim e-to-Conflict (sec)
2.0
M issed
32
“Point” Conflicts Very
Consistent
MIT
ICAT
(No “Quick” / “Long” Possible)
100%
90%
70%
60%
50%
40%
30%
% of Conflicts
80%
Point - C1
Point - C2
Point - C3
Point - C4
Point - C5
Point - C6
20%
10%
0%
9.0
6.0
3.0
T im e-to-Conflict (sec)
Missed
33
MIT
ICAT
In “Area”, Both Quick and Long Conflicts Were
Among Worst Performance
100%
90%
70%
60%
Long
Conflict
Quick
Conflict
10.0
8.0
6.0
4.0
T im e-to-Conflict (sec)
2.0
50%
40%
30%
% of Conflicts
80%
Area
Area
Area
Area
Area
Area
- C1
- C2
- C3
- C4
- C5
- C6
20%
10%
0%
Missed
34
Total Time Spent Paused (sec)
MIT
ICAT
Total Time “Paused” Indicates Less Confidence
in Selections in “Area” Scenario
Not Statistically Significant at p < 0.10
1:30
1:15
1:00
0:45
0:30
0:15
0:00
Point
Line
Area
35
Time-to-Conflict Data was
Inconclusive
Time-to-Conflict
(sec)
MIT
ICAT
10.0
8.0
6.0
4.0
2.0
0.0
First
Middle
Last
Scenario Position
36