Developing a Computational Model of the
Pilot’s Best Possible Expectation of Aircraft
State Given Vestibular and Visual Cues
Can Onur
Master’s Thesis Defence
Committee Members
Dr. Amy Pritchett (Chair)
Dr. Eric Johnson
Dr. Santosh Mathan
This work is funded by the NASA
Aviation Safety Program
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
Spatial Disorientation in Aviation
Spatial Disorientation (SD): occurs when a pilot fails to properly
sense the aircraft’s motion, position or attitude
“The chance of a pilot experiencing SD during their career is in the
order of 90 to 100 per cent.”
Australian Transport
Safety Bureau, 2007
Spatial
Disorientation (SD)
leads
Loss of Control (LOC)
SKYbrary
Flight Safety Foundation, 1992
3
Loss of Control Accidents
Fatal Accidents – Worldwide Commercial Jet Fleet – 2003 Through 2012
(
)
Boeing, 2013
Bateman, et al., 2011
4
Aircraft State Perception and Susceptibility to SD
Contributor #1: The Vestibular System
Vestibular
System
Semi-Circular
Canals
Otolith
Human vestibular system evolved in a 1-g environment (walking, running, sitting)
Limitations:
+ Threshold Values (No sensation in case of a sub-t maneuver)
+ Sensor Dynamics (Signals exponentially decay during a sustained stimulus)
5
Vestibular Illusions in Aviation
Limitations are causing illusions (especially when visual cues are lacking)
Somatogyral Illusions (mostly due to SCC)
(e.g., dead-man’s spiral, leans)
6
Somatogravic Illusions (mostly due to otolith)
(e.g., false sensation of pitch)
Aircraft State Perception and Susceptibility to SD
Contributor #2: The Visual System
Flight desk displays are the most reliable
source of information (If scanned)
Contributor #3: Pilot Knowledge of the Aircraft Dynamics
• Pilot expertise through training and experience
• Ability to generate internal expectations of the aircraft state
based on sensory cues
7
Aircraft State Perception and Susceptibility to SD
Contributor #2: The Visual System
Flight desk displays are the most reliable
source of information (If scanned)
Problem 1: How does a pilot incorporate these
sensory inputs and the expertise into their
Contributorexpectation
#3: The Knowledge
of theorientation?
Aircraft Dynamics
of spatial
• Pilot Expertise through training and experience
• Ability to generate internal expectations of the aircraft state
based on sensory cues
8
Countermeasures to SD
+ Training
 Simulators
 “Believe your flight instruments”
trainings
+ Alerting Systems
 Auditory
 Tactile
 Visual
+ Flight Deck Display Designs
 NextGen Flight Deck Displays
 Software/Hardware Enhancements
9
Countermeasures to SD
+ Training
 Simulators
 “Believe your flight instruments”
trainings
+ AlertingProblem
Systems 2: How to identify the pilot’s
information
 Auditory
 Tactile
 Visual
requirements?
Problem 3: How to help analyze potential flight
deck technology interventions?
+ Flight Deck Display Designs
 NextGen Flight Deck Displays
 Software/Hardware Enhancements
10
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
Objectives
1.
Develop a computational model (Model-Based Observer) to predict the
pilot’s best possible expectation of the aircraft state given vestibular and
visual cues.
1.
Parameterize and verify & validate the model using:
 Preliminary scenarios (brief turns, banking maneuvers etc.)
 Empirical data from the literature
12
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
The Model-Based Observer
C
1
Aircraft Dynamics
2
Measurements of
Aircraft State
3
Pilot’s “Internal
Simulation” of the
Aircraft
+
+
2
y
Best Possible Pilot
Expectations
4
+
residuals
v
-
⌃
⌃
⌃
14
4
Discrepancy
between
estimated and
actual
measurements
Discrete Visual Scanning Model
x = [ h u v w p q r q{q, f, j }]T
Discrete
Measurements
q{q, f, j }
f, q
u, v, w
h
u, v, w
+ Measurement error (v) for visual scan
…
 Sensor noise
 Error due to display design (thick needle -> elevated error)
 Pilot perception error
+ MBO stable for a range of error values
15
timeline
Vestibular Model
Continuous measurements of the aircraft states and state derivatives
The SCC Dynamics
(based on Merfeld’s model)
ySCC =
p
* q
r
The Otolith Dynamics
(based on Grant & Best’s model)
yOTO =
*
+ Measurement error (v) for vestibular model
+ Error values given in the previous work
Merfeld 1990, Grant & Best 1986
16
MBO Structure: Hybrid Kalman Filter
+ Continuous-time non-linear system dynamics (aircraft dynamics)
 P -> the error covariance matrix
(a measure of the estimated accuracy of the state estimate)
P+ = (I - Kdk Cdk )P- (I - Kdk Cdk )T + Kdk Rdk K Tdk
(Discrete RE)
·
P = AP + PAT + GQGT - PCcT Rc-1Cc P
(Continuous RE)
Kdk = P-CdTk (Cdk P-CdTk + Rdk ) -1
(Discrete Kalman Gain )
Kc = P-CcT (Rc ) -1
(Continuous Kalman Gain)
Kd & Kc are the optimal gains (Kalman Gains) to generate the best possible estimate
17
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
Verification of the Model Components
i.
ii.
iii.
iv.
19
The Hybrid Kalman Filter
The Semi-Circular Canal Model
The Otolith Model
The Dryden Model
The MBO components
Implemented in
aircraft simulation to
emulate turbulence
i) Kalman Filter Verification
Gaussian Error is used in the
Kalman Filter
Estimation error is expected to
be Gaussian
Kalman Entries
filter estimate=
Diagonal
SCC afferences
Predicted
estimation
actual
error variances of
each state
q - value (rad/s)
0.1
0.05
0
-0.05
-0.1
0
5
10
15
20
time (s)
25
30
35
q - error value
0.1
+2sigma
-2sigma
Estimate error
0.05
0
-0.05
-0.1
20
40
0
5
10
15
20
time (s)
25
30
35
40
i) Kalman Filter Verification
Gaussian Error is used in the
Kalman Filter
Estimation error is expected to
be Gaussian
q (roll rate) - estimation error distribution
450
Kalman Entries
filter estimate=
Diagonal
SCC afferences
Predicted
estimation
actual
error variances of
each state
0.1
350
q - value (rad/s)
Mean = -0.00046
Std. Dev. = 0.00788
Median = -0.00015
400
300
250
0.05
0
-0.05
-0.1
200
0
5
10
15
150
20
time (s)
25
30
35
0.1
50
0
-0.04
-0.03
-0.02
-0.01
0
0.01
0.02
0.03
q - error value
100
+2sigma
-2sigma
Estimate error
0.05
0
-0.05
-0.1
21
40
0
5
10
15
20
time (s)
25
30
35
40
q - value (rad/s)
0.1
Kalman filter estimate
actual
0.05
i) Kalman Filter
Verification – Measurements Impact
0
-0.05
-0.1
0
5
10
15
20
time (s)
25
qq--value
error value
(rad/s)
0.05
0.05
40
actual
Sub-Threshold
(no visual – no SCC)
0
0
-0.05
-0.1
0
5
10
15
20
time
(s)
time (s)
25
30
35
40
+2sigma
-2sigma
Estimate error
0.1
q - error value
35
+2sigma
-2sigma
Kalman filter
estimate
Estimate
error
0.1
0.1
0.05
Above-Threshold
(no visual)
0
-0.05
-0.1
22
30
0
5
10
15
20
time (s)
25
30
35
40
ii) SCC Model Verification
Angular Velocity (rad/s)
0.25
Stimulus (step)
Canal Afferent Response
0.2
0.15
0.1
0.05
0
-0.05
0
5
10
15
20
25
Time (s)
30
35
40
45
0.12
Stimulus (Actual Aircraft State)
Canal Afferent Response
0.1
Ang. Velocity (p) [rad/s]
0.08
A previously developed model’s responses
(Borah et. al, 1988)
0.06
0.04
0.02
0
-0.02
-0.04
10
23
15
20
25
30
Time [s]
35
40
45
50
ii) SCC Model Verification (sub-threshold behavior)
+ The SCC does not provide accurate information in case of a sub-threshold
maneuver
 Does it provide no measurement? (is it completely inactive)
 Does it provide measurements of zero?
24
p - value (rad/s)
Kalman filter estimate
actual
0.05
0
ii) SCC Model Verification (sub-threshold behavior)
-0.05
-0.1
0
(1) The SCC
provides
of 20
zero when25
maneuver30
is sub-threshold
5
10 measurements
15
35
40
time (s)
+2sigma
-2sigma
Estimate error
p - value (rad/s)
p - error value
0.1
0.05
0.1
Pilot Expectation
Aircraft State
0
0.05
-0.05
0
-0.1
-0.05 0
-0.1
0
5
10
15
20
time (s)
25
30
35
10
15
20
25
30
35
(2) The5SCC provides
no measurement
when maneuver
is sub-threshold
40
40
time (s)
+2sigma
-2sigma
Estimate error
p - error value
0.1
0.05
0
-0.05
-0.1
0
5
10
15
20
time (s)
Sub-Threshold Above-T Sub-Threshold
25
25
30
35
40
iii) The Otolith Model Verification
-4
2
x 10
1
otolith afferent firing rate
0
sf
x
-1
Forward Acceleration
Experiment
-2
-3
-4
-5
sf_x = –θ.g – ů
-6
0
10
15
Kalman filter estimate
actual
30
35
40
9
8
0
7
0
5
10
15
20
time (s)
25
30
100
theta - error value
25
5
-5
35
40
+2sigma
-2sigma
Estimate error
50
6
5
4
0
3
-50
-100
26
20
time (s)
linear acceleration
theta - value (deg)
10
5
2
0
0
5
10
15
20
time (s)
25
30
35
40
5
10
15
20
time (s)
25
30
35
40
iii) The Otolith Model Verification
-4
6
x 10
otolith afferent firing rate
4
sf
x
2
0
-2
Pitch-up Experiment
-4
sf_x = –θ.g – ů
-6
0
5
30
35
40
4
3
2
0
0
5
10
15
20
time (s)
25
30
35
40
+2sigma
-2sigma
Estimate error
50
linear acceleration
theta - value (deg)
25
5
100
theta - error value
20
time (s)
10
-5
27
15
Kalman filter estimate
actual
15
(While we didn’t
command a
deceleration the
pitch up did
cause it)
10
1
0
-1
-2
0
-3
-50
-4
-100
0
5
10
15
20
time (s)
25
30
35
40
-5
0
5
10
15
20
time (s)
25
30
35
40
iv) The Dryden Model Verification
WMC Dryden Implementation
V - linear component gusts magnitude spectrum
Vz
0
Magnitude
Vy
20
Linear gust
verification
800
Vx
WMC Dryden Implementation
600
400
200
-20
0
0
1
2
3
4
5
6
7
8
9
10
Magnitude
Vz
0
-20
2
3
4
5
6
40
60
80
Frequency (bins)
Matlab Dryden Built-in
100
120
0
20
40
60
80
Frequency (bins)
100
120
800
Vy
20
1
20
Vx
Matlab Dryden Built-in
0
0
7
8
9
600
400
200
0
10
WMC Dryden Implementation
WMC Dryden Implementation
wx
0.05
wy
0
wz
-0.05
Angular gust
verification
W - angular component gusts magnitude spectrum
1
Magnitude
0.1
0.5
-0.1
0
0
1
2
3
4
5
6
7
8
9
10
0
20
40
60
80
Frequency (bins)
Matlab Dryden Built-in
100
120
0
20
40
60
80
Frequency (bins)
100
120
Matlab Dryden Built-in
wx
0.05
wy
0
wz
1
Magnitude
0.1
-0.05
0.5
-0.1
0
0
28
1
2
3
4
5
6
7
8
9
10
Validation of the Integrated MBO
Ability of the model to predict known problems with pilot SD.
 Specifically, to reproduce the illusions that occur due to vestibular limitations
when visual cues are lacking.

29
Impact of visual scanning. Do the visual corrections help overcome the
illusion?
Somatogyral Illusion
Above-Threshold Banking
Sub-Threshold Banking
0.05
0.05
00
-0.05
-0.05
-0.1
-0.1
00
55
10
10
15
15
20
20
time(s)
(s)
time
25
25
35
35
0.05
0.05
00
-0.05
-0.05
-0.1
-0.1 0
0
30
5
5
10
10
15
15
20
20
time (s)
time (s)
25
25
30
30
35
35
40
40
Kalman filter estimate
actual
0.05
0
-0.05
-0.1
40
40
+2sigma
-2sigma
+2sigma
Estimate error
-2sigma
Estimate error
0.1
0.1
pp- -error
errorvalue
value
30
30
p - value (rad/s)
0.1
Kalmanfilter
filterestimate
estimate
Kalman
actual
actual
0
5
10
15
20
time (s)
25
30
35
40
+2sigma
-2sigma
Estimate error
0.1
p - error value
pp--value
value (rad/s)
(rad/s)
0.1
0.1
0.05
0
-0.05
-0.1
0
5
10
15
20
time (s)
25
30
35
40
Visual Corrections on Somatogyral Illusion
Above-Threshold Banking
Pilot Expectation
Aircraft State
0.05
0
-0.05
-0.1
5
10
15
20
time (s)
25
30
+2sigma
-2sigma
Estimate error
0.05
0
-0.05
31
0
-0.05
40
0
5
10
15
20
time (s)
25
30
35
40
0
5
10
15
20
time (s)
25
20
time (s)
25
30
35
40
+2sigma
-2sigma
Estimate error
0.1
p - error value
p - error value
35
Pilot Expectation
Aircraft State
0.05
-0.1
0
0.1
-0.1
0.1
p - value (rad/s)
p - value (rad/s)
0.1
Sub-Threshold Banking
0.05
0
-0.05
-0.1
0
5
10
15
30
35
40
Somatogravic Illusion
Forward Acceleration
Pilot Expectation
Aircraft State
9
0
-5
8
0
5
15
50
25
30
35
40
7
6
5
4
3
0
x-component (ft/sec 2)
2
-50
-100
1
0
5
10
15
20
time (s)
25
30
35
0
10
15
20
time (s)
25
30
35
40
40
Deceleration
0
3.5
3
-5
-10
2.5
2
0
5
100
10
15
20
time (s)
25
30
35
40
+2sigma
-2sigma
Estimate error
50
1.5
1
0.5
0
0
-0.5
-1
-50
-1.5
-100
32
5
Pilot Expectation
Aircraft State
5
theta - value (deg)
20
time (s)
linear acceleration
theta - error value
10
+2sigma
-2sigma
Estimate error
100
theta - error value
10
5
linear acceleration
theta - value (deg)
10
0
5
10
15
20
time (s)
25
30
35
40
0
5
10
15
20
time (s)
25
30
35
40
Visual Corrections on Somatogravic Illusion
Forward Acceleration
Pilot Expectation
Aircraft State
5
9
0
8
-5
7
-10
0
5
10
15
20
time (s)
25
30
40
+2sigma
-2sigma
Estimate error
20
theta - error value
35
10
linear acceleration
theta - value (deg)
10
6
5
4
0
3
-10
2
-20
33
10
0
5
10
15
20
time (s)
25
30
35
40
1
x-component (ft/sec 2)
0
5
10
15
20
time (s)
25
30
35
40
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
Examples & Potential Design Interventions
Concerns
Model Representation
Potential Design
Intervention
Pilot distraction
No scan for some or all of
the instruments
Alerting for rare (and
rarely-sampled) flight
conditions.
Inaccurate pilot perception
of state from instruments
Inaccurate or noisy
measurement
More accurate/higher
resolution presentation of
key aircraft states.
35
Summary & Contributions
The MBO enables several analyses:
 Investigate the mechanism of spatial disorientation
 Predict the best possible pilot’s expectations of the aircraft state with a given
visual scan pattern
 Identify the pilot’s information requirements (regarding the appropriate
energy-state and attitude awareness)
 Analyze potential flight deck technology interventions and/or provide design
insights for the NextGen flight deck display designs.
36
Thank You!
37