DEVELOPING A TRAINING PROGRAM FOR
THE TRAFFIC ALERT AND COLLISION
AVOIDANCE SYSTEM IN CONTEXT
Elizabeth “Scottie-Beth” Fleming
Committee Members:
Dr. Amy Pritchett, Chair
Dr. Karen Feigh
Dr. Ute Fischer
Sponsored by the FAA, Tom McCloy as Technical Monitor
Overview
Develop a training program intended to improve pilots’
understanding of TCAS use for collision avoidance in a
range of traffic situations
+
+
+
+
+
Introduction to TCAS and Background Information
Design of Training Program
Evaluation of the Training Program
Results
Conclusions
2
Traffic alert and Collision Avoidance System (TCAS)
+ Pilot always shown the Traffic Situation Display (TSD)
+ TCAS delivers a two stage advisory and vertical avoidance maneuver
 Traffic Advisory (TA) - ‘Traffic Traffic’
 Resolution Advisory (RA) - ‘Climb Climb’
+ Pilot is to follow an RA, even if it conflicts with ATC instructions, unless
the pilot believes that safe flight would be jeopardized
Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.
3
Collision Avoidance in a Broader Context
Pilots don’t only interact with TCAS, they also receive
information from ATC and the environment
Notification and
Awareness
TCAS Traffic Advisory
ATC Traffic Callout
Visual acquisition of
other aircraft
Awareness of other
aircraft via TSD
Advised Maneuvers
for Collision
Avoidance
Pilot’s Response
Compliance to TCAS
TCAS Resolution
Advisory
ATC Maneuver
Compliance to ATC
Personal Assessment
and Maneuver
Decision
Party-line Information
4
Current TCAS Training Requirements: Ground Training
+ Classroom or computer based
+ >60 training requirements
+ Measures performance through
quizzes and/or activities
To comply to the RA,
you should
Pull the stick back
Push the stick forward
Federal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C.
5
Current TCAS Training Requirements: Flight Training
6
Observed TCAS Use and Operation
o The TCAS Operational Performance Assessment (TOPA)
monitored the terminal area of 8 major airports and
examined pilot compliance to climb and descend advisories
TOPA observed compliance rates of
Non-Compliance
41% to Climb RA’s
Misunderstanding/
Confusion
59% to Descend RA’s
Aggressiveness
Olson, W. and J. Olszta (2010). TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference.
Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the
Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany.
7
Observed TCAS Use and Operation
Pilot reports flying into the red region on the VSI
“[My FO and I] incorrectly interpreted the red 'above' target on the VSI and
responded improperly. We further reviewed the procedures, agreeing that a
person should fly 'away' from the red VSI indication, if instructed via RA.”
ACN:785761, 2008
Non-Compliance
Misunderstanding/
Confusion
Pilot disagrees with advised descend RA
“Descending into an airplane that is clearly descending? TCAS software
clearly did not give appropriate guidance, nor did it self-correct when the
initial guidance was so clearly wrong”
ACN: 854982, 2009
Aggressiveness
NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/
8
Observed TCAS Use and Operation
Far Eastern Air B757 Response to Descend RA
(TCAS advised a descent rate of 1500 FPM)
Non-Compliance
Misunderstanding/
Confusion
Aggressiveness
Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1
9
Research Statement and Objectives
Develop a training program intended to improve
pilots’ understanding of TCAS use for collision
avoidance in a range of traffic situations
(1) Train pilots to understand TCAS use for collision avoidance in
the actual traffic and operational traffic environment
(2) Provide pilots with a well-rounded knowledge of different
traffic situations that may result in TCAS advisories
10
Approach to Training Design
Complement of Two Methods
Demonstration Based Training
Event Based Training
11
Demonstration Based Training (DBT)
+ Computer-Based Training
 ~25 minutes
 Provides conceptual understanding of TCAS
 Outlines rules for compliance
+ 6 segments






Introduction to TCAS
Traffic Situation Display
TCAS Advising Logic
Traffic Advisories
Resolution Advisories
Example Timeline of RA Evolution
+ Mid-Training Quizzes
12
DBT: Demonstrations of RA’s
13
Event Based Training (EBT)
+ Presents traffic events that create the requirement to act
+ Builds context and complexity into each scenario as the flight
progresses
+ Uses a more realistic training environment
Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221.
14
Simulator Study in Integrated Flightdeck – ATC Environment
Audio Communications (Aviation Intercom)
SideStick
ATC->TSD
VGA
Simulation Architecture
Touch
screen
First Officer
PFD
PartyLine
B747-400 Simulator (RFS)
ND
Charts &
Checklists
TSD
PFD
Air Traffic Simulator
(TGF)
Air Traffic
Transcripts
Captain
Eyetracker
ATC
ND
TCAS Logic
Experimenter/Instructor
TCAS Alerts
Coded Log of
Flights
Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference
on Application and Theory of Automation in Command and Control Systems (ATACCS), London.
15
Structure of Events: Descend RA Example
Training
Objective
Context of Event

Instrument
Meteorological
Conditions (daytime, 
clouds, no winds)
ATC provides no traffic 
information
Accurate
interpretation
o
Conflict caused by IFR
of and response
traffic enroute (most
to TCAS
likely on departure)
Descend RA.

RA maneuvering
should not violate ATC
o
instructions
No conflicting ATC or
party-line information 
Performance Measures
Feedback
Session
Pilot disengages autopilot
and flight directors
Pilot responds to advisory
with appropriate vertical
If the pilot did
speed
Pilot ensures vertical speed is not meet any
particular
not excessive
performance
Pilot notifies ATC of response
measure,
to TCAS advisory as the
review the
maneuver is performed
correct
Pilot reengages autopilot and response in
flight directors
regards to that
measure
Pilot notifies ATC of TCAS
advisory and response after
clear of conflict
Pilot returns to original
clearance (if needed)
16
Evaluating the training program’s impact
Overview of Study
+
+
+
+
Baseline Study
Comparison to identify the impact of training
Training Study
6 scenarios with 2 traffic events per scenario
Traffic events defined by
 RA type
 ATC information
 Traffic density
18
Overview of Study
Pre-Training Data
Collection
(50 minutes)
Pre-Experiment Questionnaire
Pre-Experiment Quiz
Introduction to TCAS
TCAS Training
Program
(50 minutes)
Demonstration Based Training
Short Section Quizzes
Event Based Training
Evaluating TCAS
Training Program
(120 minutes)
Debrief
(15 minutes)
Flight Scenarios
Post Scenario Questionnaires
Post- Experiment Questionnaire
19
Does the training program improve pilot
performance in response to TCAS advisories
and increase pilot understanding of TCAS?
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
20
Does the training program improve pilot
performance in response to TCAS advisories
and increase pilot understanding of TCAS?
 Decrease time to achieve compliance
 Decrease time to disconnect autopilot
 Reduce aggressive response features
 Increase percentage of RA duration in compliance
 Increase appropriate response post-Clear of Conflict
 Increase understanding of TCAS
 Increase trust in TCAS
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
21
Assessing the Pilot’s Response: Skill-Based Behaviors
Pilot’s Response
Vertical Speed
TCAS RA Maneuver
5 seconds
RA
Climb
X
Time Pilot
TCAS TCAS assumed Autopilot
assumed constant vertical Disconnect First Achieves
Compliance
RA rate
Time
¼ g pull-up
Skill-Based
Rule-Based
Behavior
Behavior
2 ½ sec
TCAS weakens
required
vertical rate
KnowledgeBased Behavior
time
Clear of
Conflict
22
Mean Time to Comply: During Training
pMM = Significance for the mixed model
ps2 = Significance of the variance
pm = Significance of the means
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
23
Autopilot Disconnect: Post Training
+ During training, no significant differences
+ Post training
 Autopilot disconnect time decreased
 Pilot response was more consistent for one event
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
24
Impact of Training Program
 Decrease time to achieve compliance
 Decrease time to disconnect autopilot
 Reduce aggressive response features
 Increase percentage of RA duration in compliance
 Increase appropriate response post-Clear of Conflict
 Increase understanding of TCAS
 Increase trust in TCAS
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
25
Assessing the Pilot’s Response: Aggressiveness
Pilot’s Response
TCAS RA Maneuver
Maximum
vertical rate
Vertical Speed
*Altitude Deviation
Maximum
vertical rate
difference
5 seconds
RA
Climb
TCAS
TCAS assumed
assumed
constant vertical
¼ g pull-up
RA rate
Skill-Based
Behavior
Vertical rate
difference
2 ½ sec
TCAS weakens
required
vertical rate
Rule-Based
Behavior
KnowledgeBased Behavior
time
Clear of
Conflict
26
Assessing the Pilot’s Response: Compliance
Pilot’s Response
TCAS RA Maneuver
Not in
Compliance
Vertical Speed
In Compliance
5 seconds
RA
Climb
TCAS
TCAS assumed
assumed
constant vertical
¼ g pull-up
RA rate
Skill-Based
Behavior
2 ½ sec
TCAS weakens
required
vertical rate
Rule-Based
Behavior
KnowledgeBased Behavior
time
Clear of
Conflict
27
Assessing the Pilot’s Response: Return to Clearance
Pilot holds new altitude
Contacts ATC and request new clearance
or ask for further instructions
Altitude
Pilot’s Response
TCAS RA Maneuver
Pilot begins descent
back to originally
cleared altitude
May inform ATC of
response to RA
Cleared altitude
at time of RA
time
5 seconds
RA
Climb
TCAS TCAS assumed
assumed constant vertical
¼ g pull-up
RA rate
Skill-Based
Behavior
TCAS weakens
required vertical rate
Rule-Based
Behavior
Clear of
Conflict
KnowledgeBased Behavior
28
Before Training
+ 89% of the pilots did not know that an RA should cause less than 500 feet
of altitude deviation
+ 56% of the pilots responded that they would hold current altitude
achieved after responding to an RA, as opposed to returning to their
clearance
+ When asked about airline procedures for following TCAS, all 18 pilots
noted the need for complying with an RA
 BUT 28% of the pilots commented compliance wasn’t necessary is there was a
TCAS “malfunction” or if the RA would cause an “unsafe situation”
“[Pilots] must always comply with a TCAS RA unless
[aircraft] performance is hindered (i.e. operating single
engine) or [there is] an obvious TCAS malfunction (ie you
can see traffic and it is not a threat)”
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
29
Aggressiveness
+ During training, aggressive response features decreased
+ Post training, same decreasing trend observed
Altitude Deviation
Average Vertical Rate Difference
Maximum Vertical Rate Difference
Maximum Vertical Rate Difference
Skill-Based
Behavior
Rule-Based
Behavior
All decreased
KnowledgeBased Behavior
30
Percentage Compliance
+ During training, percentage compliance decreased in training event with
“Climb RA” (93.1% compared to 99.6%)
+ Post training
 No significant differences in means observed
 But, trained pilots had a more consistent response
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
31
Return to Clearance
Pilot returns to original clearance
Yes
No
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
32
Impact of Training Program
 Decrease time to achieve compliance
 Decrease time to disconnect autopilot
 Reduce aggressive response features

Increase percentage of RA duration in compliance
 Increase appropriate response post-Clear of Conflict
 Increase understanding of TCAS
 Increase trust in TCAS
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
33
Before Training
+ 94% pilots agreed with statement “I understand TCAS maneuvers when
they are issued”, but…
 Only 50% of the pilots correctly identified the assumptions made by TCAS
advisory logic
 When asked to interpret TSD symbols, only 50% of the pilots got all parts of
the associated questions correct
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
34
Post Training Understanding of TCAS
“My understanding of TCAS has increased”
61%
33%
0
0
6%
Strongly
Disagreed
Disagreed
Neutral
Agreed
Strongly
Agreed
+ 50% pilots reported an increase in understanding TCAS logic
+ 27% pilots claimed to have learned about different types of TCAS RA’s
(notably, “Crossing RA’s”)
“Types of RA's were not previously taught.
We were taught simply to comply”
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
35
Post Training Trust in TCAS
“I am more likely to trust TCAS after completing today’s training”
39%
39%
11%
11%
0
Strongly
Disagreed
Disagreed
Neutral
Agreed
Strongly
Agreed
“My trust in TCAS was already at a maximum
so I wouldn’t be ‘more’ likely to trust it”
Skill-Based
Behavior
Rule-Based
Behavior
KnowledgeBased Behavior
36
Post-Training
 Decrease time to achieve compliance
 Decrease time to disconnect autopilot
 Reduce aggressive response features

Increase percentage of RA duration in compliance
 Increase appropriate response post-Clear of Conflict
 Increase understanding of TCAS
 Increase trust in TCAS
37
Conclusions and Future Work
Conclusions
+ Pilots may not need more training, but instead need better training
 Integrating DBT and EBT methods permits ground-based and flight training
material to be more cohesive
 EBT structure allows for the design of purposeful training events
+ Current FAA mandated TCAS training objectives may not fully reflect all
training areas
 Language needed to address the reduction of excessive responses to RA’s
Future Work
+ What facilities and technologies would be required to implement this type
of training program?
+ What implications arise when considering training design versus system
design?
 Would incorporating human factors considerations in initial design stages
decrease the amount of required training?
38
39
Conclusions and Future Work
Conclusions
+ Pilots may not need more training, but instead need better training
 Integrating DBT and EBT methods permits ground-based and flight training
material to be more cohesive
 EBT structure allows for the design of purposeful training events
+ Current FAA mandated TCAS training objectives may not fully reflect all
training areas
 Language needed to address the reduction of excessive responses to RA’s
Future Work
+ What facilities and technologies would be required to implement this type
of training program?
+ What implications arise when considering training design versus system
design?
 Would incorporating human factors considerations in initial design stages
decrease the amount of required training?
40
Acknowledgements
+
+
+
+
+
+
+
+
+
+
Work sponsored by the FAA, Tom McCloy as Technical Monitor
34 pilot participants
Dr. Amy Pritchett
Dr. Karen Feigh
Dr. Ute Fischer
Dr. Wesley Olson, MIT Lincoln Labs
Wayne Gallo, FAA
Roger Sultan, FAA
Kylie Garey
TCAS Team: William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan,
Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar,
Jonathan Zoetrum, Jelle Wissink
+ CEC Lab Members
41
42