HumanFactors
Getting the most out of Non Technical Skills (NTS) Assessment
Dr Matthew Thomas
Human Factors & Safety Management Group
University of South Australia
Level 7 Playford Building - City East
GPO Box 2471 Adelaide SA 5001
AUSTRALIA
ph:
0438 808 808
e-mail: matthew.thomas@unisa.edu.au
Kia Ora
Adelaide
Research Context
Culture, Error and NTS on the line...
TABLE TWELVE
Logistic Regression Analysis - Contributions of contextual factors and non-technical skills to crew's
failure to respond to an error during the descent-approach-landing phase.
Predictor
Captain as Error Origin
Assertiveness
Constan t
Model
2
1.555
-.849
.856
S.E.
.608
.339
.979
2
Wald's
6.531
6.253
.764
df
1
1
1
p
.011
.012
.382
e
4.735
.428
2.354
2
= 21.994, df = 2, p < .001, Nagelkerke R = .293, 70.8% correct classification.
Culture, NTS, and Error Management
If captain made the error, crew 4.7 times less likely to manage error
If crew scored high on assertiveness, 2.3 times more likely manage error
Culture, NTS, and Error Management
Strong correlations between NTS markers and Error Management
Top five markers:
Leadership / Followership
Vigilance
Assertiveness
Inquiry
Communication Environment
Instructional Use of Error and NTS...
Use of Error in Line Training
Corrected by Instructor
26
Directive from Instructor
14
Feedback from Instructor
5
Corrected by Trainee
10
None
45
0%
10%
20%
30%
40%
50%
TEM in the Simulator
Identify
Instructor Acknowledged Error
77
64
Debreifed in Simulator
56
48
Debriefed Post-Session
40
59
Debrief
Detection of Error Discussed
NTS
Error Prevention Discussed
Error Management Discussed
3
6
34
40
6
9
0%
Day One (Training)
Day Two (Check)
25%
50%
75%
100%
N = 277
Error Type
Training Focus
“Gotcha” - error due to
situational demands
Early recognition and management
strategies.
“Random” - slip or
lapse due being
human
Detection and response strategies.
“Deficit” - lack of
knowledge or skill
More practice!
“Mistake” - decisionmaking or judgement
Dig deeper and understand why.
Need to identify where decision
process went wrong.
Summary
Some good data on the role of NTS in crew
performance during normal operations
NTS are still not a real focus in recurrent
training / checking programs
CRM in the classroom and NTS in the
simulator / on the line are not yet aligned
TEM can be used as a focus for NTS in
training
Regulatory Context
CASA - Civil Aviation Safety Authority (AUSTRALIA)
13 Jan 2009
NOTICE OF FINAL RULE MAKING
Implementation of Safety Management Systems (SMS)
and introduction of Human Factors (HF) Training and
Non-Technical Skills (NTS) Assessment
Amendments to Civil Aviation Orders (CAOs) 82.3 and 82.5 (Air
Operators Certificates - RPT
CASA - Civil Aviation Safety Authority (AUSTRALIA)
RPT AOC holders must “have a program, approved by CASA, to train and
assess personnel in human factors and non-technical skills with the aim of
minimising human error.”
Human Factors (HF) means the minimisation of human error and its
consequences by optimising the relationships within systems between people,
activities and equipment.
Non-Technical Skills (NTS) means specific human competencies, including critical
decision making, team communication, situational awareness and workload
management, which may minimise human error in aviation.
CASA - Civil Aviation Safety Authority (AUSTRALIA)
17 Sep 2009
Project OS 09/15 - Minor Amendment of CAO 82.3 & 82.5 to
remove the legislative requirement for assessment and to
provide more time for Industry to conduct Human Factors
Training
Project approved.
FAA - Federal Aviation Administration (USA)
Funded some great research.
Got burnt by AQP?
Has produced some good Advisory Circulars (AC120-51E CRM).
EASA - European Aviation Safety Agency (EUROPE)
EU-OPS and EU-FCL
Requirement for assessment of CRM performance
EU-OPS 1.965 Recurrent Training and Checking — Pilots
Line checks
(i) Line checks must establish the ability to perform satisfactorily a complete line operation including
pre-flight and post-flight procedures and use of the equipment provided, as specified in the
Operations Manual.
(ii) The flight crew must be assessed on their Crew Resource Management CRM skills in accordance
with a methodology acceptable to the Authority and published in the Operations Manual. The purpose
of such assessment is to:
(A) Provide feedback to the crew collectively and individually and serve to identify retraining; and
(B) Be used to improve the CRM training system.
(iii) CRM assessment alone shall not be used as a reason for a failure of the line check.
(iv) When pilots are assigned duties as pilot flying and pilot non-flying they must be checked in both
functions.
NOTECHS
Summary
NTS Assessment is coming... (albeit very very
slowly).
Australia is going backwards as quickly as it is
going forwards.
European regulations are the most advanced,
but what about implementation?
We should be setting the standards by worlds
best practice rather than awaiting regulatory
requirements.
Getting the most out of NTS
Assessment / Evaluation
The core requirements...
Supported by standardized and
customized tools.
Supported by good resources for
instructors and crew.
BEHAVIOURAL MARKERS
TRAINING &
INTER-RATER RELIABILITY
Starts with good syllabus & scenario
design for the simulator.
INTEGRATION OF
TRAINING
Provides focus for brief and debrief
during training.
FOCUSSING ON SKILL
DEVELOPMENT
Behavioural Markers and NTS
Assessment
During the late 1990s the “Holy Grail”
was a universal standardised set of
Behavioural Markers for NTS.
NOTECHS
Behavioural
Markers for NTS
NOTECHS
Rating Scale
Training and Inter-Rater Reliability
2 day program
Training
training records
Calibration
rwg=1-( Sχ2/σE2)
Monitoring
Integration
015.677#128#92(+(1'#,'(:;
!
!
!"!#$%#&'()*+#,-./#0-1(2(2)##31241'#
!"!#$%#&'()*+#,-./#0-1(2(2)##31241'#
L(-.I+(621'#,62+-6'
,-6AA/(28#015.677
,)-!8&#779'/6!(:'6-4'/-7!7)#9/!;-4#9!9-&-!6-&'<-6!*)&#:()!54'()*!*-7*!61*1=!
-/('/--&'/(!1/14%7'7=!1/6!54'()*!7'0:41*#&!-<14:1*'#/72
$6+.B >/('/-!7:&(-!81/!#88:&!9'*)!1!7*&#/(!8&#779'/6!#&!*1'49'/6!8#0$#/-/*!'5!
*1?-#55!*)&:7*!'7!7-*!;-5#&-!;&1?-!&-4-17-2!,)-&-5#&-=!*)-!&#44'/(!*1?-#55!
$&#8-6:&-!'7!7*&#/(4%!16<'7-6!9)-/!8&#779'/67!-@8--6!AB!?/#*7!#&!
*1'49'/67!-@8--6!CB!?/#*72
015.677#,-6AA/(28#%4(8.'(2.A
"&#779'/6!(:'6-4'/-7!1&-!/#*!8#/7'6-&-6!4'0'*1*'#/72!"&#779'/6!(:'6-4'/-7!1&-!
$&#<'6-6!*#!177'7*!#$-&1*#&7!'/!-7*1;4'7)'/(!*)-'&!#9/!8&#779'/6!$#4'8'-72
,1?-#55!8&#779'/6!(:'6-4'/-7!1&-!;17-6!#/!*)-!0#7*!16<-&7-!1'&$41/-!4#16'/(!D4'()*!
9-'()*!1/6!15*!8-/*-&!#5!(&1<'*%E!1/6!177:0-!1/!-/('/-!#:*!F,G!1/6!$&#$-&!$'4#*!
*-8)/'H:-2!G/!74'$$-&%!&:/91%7=!8&#779'/6!(:'6-4'/-7!1&-!1!5:/8*'#/!#5!&:/91%!
7:&518-!8#/6'*'#/2
C42/1D#,628(+(62
015.677#128#92(+(1'#,'(:;
,-6AA/(28#,6:E62.2+#
?526+A@#F
/(+*64+#G#/(+*#/(2)'.+A
U/'*'14!&:/91%!14'(/0-/*!1/6!70##*)!7%00-*&'814!*)&:7*!1$$4'81*'#/!&-7:4*!'/!(##6!
8&#779'/6!8#/*&#4!81$1;'4'*%!6:&'/(!*1?-#552!V'()*!5#&91&6!$&-77:&-!#/!*)-!8#/*&#4!
8#4:0/!6:&'/(!*)-!'/'*'14!$)17-!#5!*1?-#55!&#44!D;-4#9!1$$&#@'01*-4%!MB!?/#*7E!
'/8&-17-7!/#7-!9)--4!7*--&'/(!-55-8*'<-/-772!W/%!6-<'1*'#/!5&#0!*)-!8-/*-&4'/-!
6:&'/(!*)&:7*!1$$4'81*'#/!7)#:46!;-!8#:/*-&-6!9'*)!'00-6'1*-!70##*)!1/6!$#7'*'<-!
8#/*&#4!'/$:*72!30##*)!&:66-&!8#/*&#4!'/$:*7!8#0;'/-6!9'*)!70144!8#/*&#4!9)--4!
'/$:*7!&-7:4*!'/!1!/#&014!*1?-#55!9'*)!/#!#<-&8#/*&#44'/(2!V1&(-!8#/*&#4!9)--4!
'/$:*7!81/!)1<-!1/!16<-&7-!-55-8*!#/!6'&-8*'#/14!8#/*&#4!/-1&!XCDT"YE!6:-!*#!*)-!
166'*'#/14!6&1(!#5!*)-!-@*-/6-6!7$#'4-&72
$6+.B J'*)!9-*!#&!74'$$-&%!&:/91%!8#/6'*'#/7=!*)-!ZT!7)#:46!('<-!7$-8'14!
1**-/*'#/!*#!-/7:&'/(!*)-!-/('/-7!)1<-!7%00-*&'8144%!;141/8-6!*)&:7*!
'/6'81*'#/72!
C6+1+(62#128#015.677
T1'/*1'/!9'/(7!4-<-4!6:&'/(!*)-!*1?-#55!&#44!;%!1$$4%'/(!8#/*&#4!9)--4!
6'7$418-0-/*!'/*#!*)-!9'/62!O:&'/(!&#*1*'#/!8#/*'/:-!*#!1$$4%!8#/*&#4!9)--4!'/!*)-!
6'7$418-6!$#7'*'#/!*#!?--$!*)-!9'/(7!4-<-4!6:&'/(!4'5*#552!,)-!1'&$41/-!'7!'/!1!
7'6-74'$!9'*)!8&#77-6!8#/*&#47!1*!*)'7!$#'/*2!W!74#9=!70##*)!&-8#<-&%!5&#0!*)'7!
7'6-74'$!'7!188#0$4'7)-6!15*-&!4'5*#55!;%!74#94%!/-:*&14'['/(!*)-!8#/*&#4!9)--4!1/6!
&:66-&!$-61472
O&%
KP!Q!KL
J-*
AN
%4A+D#M(28#128#N+-62)#,-6AA/(28#,628(+(62A
\#&!*1?-#55!'/!(:7*%!#&!7*&#/(!8&#779'/6!8#/6'*'#/7=!:7-!#5!1!)'()-&!*)&:7*!7-**'/(!
*)1/!*)-!0'/'0:0!&-H:'&-6!'7!&-8#00-/6-62!J)-/!*)-!$&-<1'4'/(!9'/6!'7!1*!#&!
/-1&!]B^!*#!*)-!&:/91%=!*)-!$#77';'4'*%!#5!9'/6!7)'5*7!&-7:4*'/(!'/!(:7*%!*1'49'/6!
8#0$#/-/*7!6:&'/(!&#*1*'#/!#&!4'5*#55!'/8&-17-72!O:&'/(!*)'7!8#/6'*'#/=!8#/7'6-&!*)-!
:7-!#5!*)&:7*!7-**'/(7!84#7-!*#!#&!1*!01@'0:0!*1?-#55!*)&:7*2!,)-!:7-!#5!1!)'()-&!
*1?-#55!*)&:7*!7-**'/(!&-6:8-7!*)-!&-H:'&-6!&:/91%!4-/(*)!1/6!0'/'0'[-7!*)-!
1'&$41/-!-@$#7:&-!*#!(:7*%!8#/6'*'#/7!6:&'/(!&#*1*'#/=!4'5*#55=!1/6!'/'*'14!84'0;2
3*1/6'/(!J1*-&Q34:7)
CN
3/#9!R!S#!T-4*'/(!II
AN
U8-!R!S#!T-4*'/(!II
CN
IJ'/67!0-17:&-6!1*!KK!5--*!DCB!0E!*#9-&!)-'()*!1/6!1$$4%!5#&!&:/91%7!CLM!5--*!
DLN0E!#&!(&-1*-&!'/!9'6*)2
II!,1?-#55!#/!:/*&-1*-6!'8-!#&!7/#9!7)#:46!#/4%!;-!1**-0$*-6!9)-/!/#!0-4*'/(!'7!
$&-7-/*2
"#$%&'()*!+ ,)-!.#-'/(!"#0$1/%2!3--!*'*4-!$1(-!5#&!6-*1'472
"<=>
&,0#!"!#$%#?03@
"#$%&'()*!+ ,)-!.#-'/(!"#0$1/%2!3--!*'*4-!$1(-!5#&!6-*1'472
HI+6;.-#"=J#>KK!
HI+6;.-#"=J#>KK!
&,0#!"!#$%#?03@
"<="
Predictions for the Future
Changes in Structure
Targeted
Integrated
Aligned
Beyond Classroom
More Simulator and Line
CFIT / ALARP
Skills. Full stop.
CRM = NTS = TEM
Still important
Skill Development
Changes in Content
Fitness for Duty
Emotions “HEA”
Fatigue / Health /...
Frustration / Stress
Confusion
Performance Protection
Adapting to Situations
Systems Thinking
Safety Emerges from
System
Some Good Resources
http://www.aavpasymposium.org
Dr Matthew J W Thomas
E-mail:
Phone:
Web:
matthew.thomas@unisa.edu.au
+61 438 808 808
http://www.unisa.edu.au/safety
UniSA © 2009