Final Presentation
An Experimental Design of a
Foundational Framework for the
Application of Affective Computing to
Soaring Flight Simulation and Training
D O C TOR AL C AN D IDATE , CO M P U TE R SC I E NCE
SH AN NON M AR I E M O O N
M AY 22, 2017
Yanzhen Qu, Ph.D., Faculty Mentor and Chair
Dr. Richard Livingood, Ph.D., Committee Member
Dr. Cynthia Calongne, D.CS., Committee Member
Overview Of Presentation
 Proposal Recap
 Problem and Opportunity
 Research Questions
 Theoretical Framework
 Methodology
 Findings
 Performance Level
 Conclusions
 Recap of patterns and
trends
 Limitations
 Implications: Field and
Practice
 Future Research
 Closure
 Emotional Response
 Physiological Response
2
Problem Opportunity Statement
 Soaring flight (particular in competition) has
unique skills and requirements overlooked in
traditional flight simulation
 There are no soaring flight simulation training
systems that adapt to both pilot emotion and
performance
 To fill this gap, a framework foundation for
affective computing and flight is needed
3
Purpose And Significance
 Demonstrate a framework foundation to
measure pilot psychophysiological state, pilot
affect, and performance during flight task
performance in quantitative manner
 Once a foundation exists, it can be applied in
real time to create an affective soaring flight
simulator
 Potential decrease in training costs, increase in
flight safety
4
Research Questions
1.
What is the range of performance level for flight
task execution?
2.
What are the feelings reported when executing
flight tasks?
3.
What is the range of physiological values
measured during the execution of the flight task? *
*Physiological parameters measured include:
HR, HRV, EDA, PSTF
5
Conceptual Framework
6
Conceptual Framework (cont.)
 Affective Computing (Picard, 1995)
 Cognitivist Theory
 Picard, 1997
 Battarbee & Koskinen, 2005
 Boehner et al, 2007
 Affective Wearables
 Picard & Healey, 1997
 Healey & Picard, 2005
 Garbarino, Lai, Bender, Picard, & Tognetti, 2014
 Operator Affect
 Healey & Picard, 2000 & 2005 (Driving)
 Klein, 2014 (Satellite Operations),
 Mü ller & Fritz, 2015 (Software Development)
7
Conceptual Framework (cont.)
 Flight
 Simulation Effectiveness
 Lintern, Roscoe, Koonce, & Segal (1990)
 Lintern, Roscoe, & Sivier (1990)
Crowley et al(1997)
 Risukhin (2016)
 Pilot Workload / Assessment
 Roscoe & Ellis (1990)
 Dussault et al (2004)
 Dahlstrom & Halinder (2006 & 2009)
 Dussault et al (2009)
 Practical Test Standards – FAA

8
Research Methods
 Quantitative Study
 Test subjects completed a pre-experiment
questionnaire on pilot demographics
 Subjects attempted to perform a series of flight
tasks from the FAA Glider Pilot Practical Test
Standards in a soaring flight simulator
 Subjects wore physiological sensors on wrists
 Subjects reported emotional stress level during
task performance
9
Study Population
 As of 2015, FAA listed 25K+ active glider pilots
 Active is defined as a non-student pilot with a valid flight
medical, however FAA regulations do not require a medical for
glider operations
 Many glider pilots get a glider rating as a BFR and discontinue
glider flight
 Most glider pilots with recent flight time are active cross
country pilots, competition pilots, flight instructors, or in
training for a glider rating
 Many glider pilots are also considered as active pilots
of other aircraft types (airplane, rotorcraft, lighter than-air)
 Glider pilots typically operate out of dedicated
gliderports outside of Class B or Class C airspace in
rural areas
10
Sample And Sampling Procedure
 Flyers posted at Gliderports and airparks in Florida
and North Carolina
 Flyers posted to online distribution lists and social
media sites for airports, gliderports, and aviation
organizations
 Seminole-Laker Gliderport (Commercial FBO & host for
USA Seniors National Soaring Contest)
 Piedmont Soaring Society (Not-for-Profit Club)
 Women’s Soaring Society of America (Division of the
Soaring Society of American)
 Experimental Aircraft Association (Charlotte Chapter)
 99’s (Carolina Chapter) – International Organization of
Licensed Women Pilots
11
Instrumentation
 Pre-Experiment Questionnaire
 Soaring Flight Simulator (flight task
performance, IGC flight logs)
 Two E4 Sensors (physiological measurements)
 Bedford Scale (subjective emotional reporting)
12
Instrumentation: Simulator
Simulator paused during testing executing T4
13
Instrumentation: E4 Wearable
Empatica E4 Wearable Sensor Band
14
Data Collection
 Demographics from questionnaire
 Emotional stress (reported by test subjects and
recorded by observer)
 Tasks performed in simulator
 Task from standard were defined
 Performance logged by IGC loggers and recorded by
observer
 Converted to quantitative value for analysis
 Physiological measurements recorded by E4s
15
Data Analysis
 Visual review of graphed raw E4 Data
 Matlab (extract HRV from IBI and BVP data, examine EDA for
SCL and SCR)
 Observer record of task performance level
 IGC Logs reviewed
 Identify & record task initiation/ termination timestamps
 Verify observer record of task performance level
 Translate performance to quantitative scale
 Analysis & visualization
 Data imported into statistical software
 Dependent Variables examined by Task (Independent Variable)
 Additional groupings by gender, pilot ratings
16
Data Analysis: Flight Tasks
Table 1 Flight Task Definition
Task Value Definition
T1
Takeoff - includes takeoff roll until airborne and towplane lifts off
T2
Aerotow - begins after takeoff, being towed aloft by a towplane
T3
Pattern Approach - rectangular pattern and approach to landing
T4
Landing - final leg, touchdown, rollout and stop
Note: Task is the independent variable in this study
17
Data Analysis: Performance
Excerpt from Quantitative Table for One Task
Table Performance Level Definition for Task T4 (Landing) Table
Level Value
1
Definition
Landing performed to commercial pilot standard, within +/- 5 knots
of recommended approach speed, stopping short of and within 100
feet of a designated landing point, adjusts dive brake as appropriate,
maintains crosswind and directional control, smooth touch down
2
Landing performed to private pilot standard, within +10/-5 knots of
recommended approach speed, stopping short of and within 200 feet
of a designated landing point, adjusts dive brake as appropriate,
maintains crosswind and directional control, smooth touch down
Note: Values range from 1 (best) to 5 (worst)
18
Data Analysis: E4 Raw Data Example
EDA
Blood Volume
Pulse
Accelerometer
X, Y, Z
HR
Temperature
Participant Demographics
Highlights
15 Participants
8 Male, 7 Female
Age from 19 to 74
Wide range of flight
experience
#
Gender Age
Airplane Rating
Glider Rating
1
M
68
ATP
C
2
F
59
P
P
3
M
53
4
F
59
P
P
5
F
59
P
P
6
M
48
7
F
48
8
M
19
9
F
19
10 M
47
11 F
74
S
12 F
58
P
13 M
47
P
14 M
50
P
15 M
54
P
P
P
P
P
S
Note: ATP=Airline Transport Pilot, P=Private,
C=Commercial, S=Student w/Solo Endorsement
20
Findings (Performance by Pilot Rating)
Performance Level Grouped by Pilot Rating (Airplane, Glider, Glider & Airplane, Non-Rated
Pilot.
21
Findings (Performance by Gender)
Performance Level Grouped by Gender .
22
Findings (Emotion)
 Question 2: What are the feelings reported when
executing flight tasks?
Emotional Response Level (Summary Report)
23
Findings (Emotion by Gender)
Emotional Response Level (Summary Report)
24
Findings (Emotion by Pilot Rating)
Emotional Response Level Grouped by Pilot Rating
Airplane, Glider, Glider & Airplane, Non-Rated Pilot )
25
Findings (Physiological)
 Question 3: What is the range of physiological
values measured during the execution of the flight
task? *
 3.1 HR
 3.2 HRV
 3.3 EDA
 3.4 PSTF
26
Findings (Physiological: HR)
Minimum and Maximum Heart Rate per Task (Summary Report)
27
Findings (HR by Gender)
Minimum and Maximum Heart Rate Grouped by Gender
28
Findings (HR by Pilot Rating)
Minimum and Maximum Heart Rate Grouped by Pilot Rating
29
Findings (Min HR by Pilot Status)
Minimum Heart Rate Grouped by Pilot Status (Pilot=Y, Non-Rated Pilot=N)
30
Findings (Max HR by Pilot Status)
Maximum Heart Rate Grouped by Pilot Ratin
31
Findings (HRV)
Heart Rate Variability pNN50 per Task Summary Report
32
Findings (HRV by Gender)
Heart Rate Variability pNN50 Grouped by Gender
33
Findings (HRV by Pilot Rating)
Heart Rate Variability pNN50 Grouped by Pilot Rating
34
Findings (EDA SCL)
 Expected SCL pattern of roughly symmetrical rising
and falling throughout the session not seen for all
subjects.
35
Findings (EDA SCL)
 Atypical SCL Pattern for four sessions
1.
2.
3.
4.
Female G&A pilot: left typical, right session started
with SCR peak, SCL descended through the session
Female G&A pilot; left started with SCR peak, SCL
descended throughout the session, right session no
data due to EDA signal loss
Male non-rated pilot: left and right session followed
typical pattern until T3, then left SCL climbed as
right descended
Male G&A pilot: left started with SCR peak, SCL
descended throughout the session, right typical
36
Findings (EDA SCR)
EDA SCR Summary. Signal loss for some sessions, 1 left and 3 right
37
Findings (EDA SCR by Gender)
EDA Grouped by Gender. Some loss for some sessions, 1 left (female), 3 right (male)
38
Findings (EDA SCR by Gender)
 Females tended toward lower range of SCRs
Table Interquartile Range for Right Wrist SCRs by Gender per Task
Gender
T1
T2
T3
T4
Female
3/4
12 / 16
13 / 20
13 / 20
Male
4/5
14 / 20
16 / 19
15 / 19
Note: Signal loss occurred for some sessions, 1 left (female), 3 right (male)
39
Findings (EDA SCR by Pilot Rating)
EDA Grouped by Pilot Rating. Signal loss occurred for some sessions (2 G&A, 1 Non, 1 A),
40
Findings (PSTF)
 Minimal changes across all test subjects for all
tasks, less than than 1°C
Peripheral Skin Temperature Fluctuation. Example of minimal fluctuation across an
experimental session. Red lines indicate timeline markers.
41
Conclusions (Performance Level)
 Females as a group exhibited overall better
performance on tasks
 Glider rated pilots as a group outperformed
airplane only pilots and non-rated pilots
 Airplane only pilots out performed non-rated
pilots
 Non-rated pilots exhibited the overall lowest
performance
42
Conclusions (Emotion)
 Females as a group reported lower emotional
stress
 Pilots with a glider rating reported an overall
lower emotional stress level than airplane only
rated pilots or non-rated pilots
 Airplane only rated pilots reported a wider
range of emotional stress and higher maximum
values than glider rated pilots or non-pilots
43
Conclusions (Physiological: HR/HRV)
 HR
 Overall trend for HR to increase with successive
performance of tasks
 Females as a group had lower HR ranges than males
 Rated pilots as a group had lower HR ranges than
non-rated pilots
 HRV decreased as successive flight tasks were
performed across all groupings
44
Conclusions (Physiological: EDA)
 Some signal loss (addressed by new band and
electrode design released after this study was
complete)*
 Atypical SCL pattern seen for some subjects
 Females as a group exhibited fewer SCRs than male
subjects
 Baseline set of date for SCRs per Task established
* E4 Sensors can be upgraded to new design for future research
45
Conclusions (Physiological: PSTF)
 Minimal to no fluctuation emerged for
peripheral skin temperature fluctuation
 Variation of less than 1°C per subject task
46
Limitations
 Unexpected EDA signal loss
 Signal loss impacted ability to examine SCL
symmetry and atypical patterns seen
 Lack of variation in PSTF could be related to
controlled temperature in simulator flight
 Real world glider flights experience more wide range
of temperature
 Thermaling in shade under cloud streets vs. travel
between cloud streets
 Opening vents and vent windows during thermaling
47
Implications for the Field
 Demonstrated a practical framework for affective
computing and soaring flight
 Demonstrated mechanism for consistent scale for rating
performance across different flight tasks
 Demonstrated a mechanism for reporting pilot emotion
 Demonstrated a mechanism for non-intrusive
measurement of physiological indicators
 Established preliminary baselines for performance,
emotion, and physiological measure during soaring
flight task performance
 Framework and data provides a foundation that can
be expanded (additional tasks and training levels)
48
Implications for Practice
 Foundation for first ever FAA Certified Soaring
Flight Simulator integrating Affective Computing
for leading edge technology
 Potential for Soaring Competition Training
 Foundation could be extended to other types of
simulation based flight or other operator
training
49
Future Research
 Extend Framework
 Additional Tasks, & Physiological Indicators
 Additional Affect Measure (facial recognition, voice stress, eye
movement, pupil dilation)
 Additional Flight Simulation Types (seaplane, helicopter,
instrument, etc.)
 Real-time Implementation of Framework
 Adaptive Simulator: true intelligent tutoring system that adapts
to affect and performance in real-time
 Decrease in flight training times, costs
 Other Operational Domains
 Gender differences in operator performance, emotion,
and physiological response to cognitively challenging
tasks
50
Closure
 Publication Plans
 Follow-up Research
 Teaching
 Affective Simulator
 Affective Life
51
Appendix A – Demo Videos
 Glider Flight Demonstration Video
http://y2u.be/1ef753UjCZ8
 Simulator Demonstration Video
http://y2u.be/QOKCn_e0D04
Final Presentation
An Experimental Design of a
Foundational Framework for the
Application of Affective Computing to
Soaring Flight Simulation and Training
D O C TOR AL C AN D IDATE , CO M P U TE R SC I E NCE
SH AN NON M AR I E M O O N
M AY 22, 2017
Yanzhen Qu, Ph.D., Faculty Mentor and Chair
Dr. Richard Livingood, Ph.D., Committee Member
Dr. Cynthia Calongne, D.CS., Committee Member