Design of a Procedure
Analysis Tool (PAT) for the
FAA Human Factors
Certification Process
NEW
Devices
Exhaustive
Certification
Process
Procedure
Analysis
Tool
Heather Johnson
Houda Kerkoub
Sky
Overhaul
Program
Raytheon
Dr. George Blaha
Shaida Latifi
Mumtahina Mahmud
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Alexis Nittoli
1
2
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
3
Context Roadmap
Flight
demand
increase
Capacity for
flights is
constant
Accidents 
human
errors as
main factors
Raytheon
Dr. George Blaha
Air traffic
modernization
Program
(NextGEN)
New regulation
(FAR25.1302)
 include human
factors in device
certification
New devices
to be installed
on aircraft
Requirement:
analysis of ALL
tasks and
functions
More devices
to be certified
Additional work
for FAA Human
Factors inspectors
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Opportunity
for improved
certification
process
Number of
available FAA
Human Factors
inspectors
constant
4
Background
 From years 1995-2030
• Passengers Per Year = 580
million to 1.2 billion
• Flights per day = 30,000 to
60,000
 US air transportation
infrastructure is aging
• Increase of delays
• Increase of cancellations
• Increase in maintenance costs
• Safety remains stagnate
Raytheon
Dr. George Blaha
Source: Subject Matter Expert
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
5
Demand and Supply of Flights
Flight demand increase while the
Capacity for flights is constant
United States Department of
Transportation –
U.S Air Carrier Traffic
Statistic. July 2014
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
6
Modernization Program NextGEN
Terminal
Enroute

Transform ground navigation into
satellite navigation

Surface
Air Traffic Control will have “satellitebased surveillance, navigation, and
network-centric operations”

NextGEN $40 billion(expected by
2025)

Projected Money saved through
NextGEN will be $123 billion
through 2030
Sources: Calvin L. Scovel III, 2013, Gerald L. Dillingham, Ph.D. 2012
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
7
Modernization Program = More Devices
Portfolios
NextGEN
9 Implementation Portfolios
1.
2.
3.
4.
5.
6.
7.
8.
9.
Improved Surface Operations
Improved Approaches and Low Visibility
Closely Spaced, Parallel, Converging, and
Intersecting Runway Operations
Performance Based Navigation (PBN)
Time Based Flow Management
Collaborative Air Traffic Management
Automation Support for Separation
Management
On-Demand NAS Information
Environment and Energy
Raytheon
Dr. George Blaha
Operational
Improvements
OI 104123 –Time Based Metering
Using RNAV and RNP Route
Assignments
OI 108209 – Increase Capacity and
Efficiency Using Area Navigation
and Required Navigation
Performance
OI 109315 – Implement NextGen
Environmental Engine and Aircraft
Technologies
OI 109318 – Environmental Engine and
Aircraft Technologies
New Devices
Aircraft equipage
with PBN
equipment
Aircraft equipage
with
environmentally
friendly engine
(ex: C Series
engine and
FADEC)
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
8
More Devices to Certify
 The overall number of OIs relating to
certification of new devices is
projected to peak from 2013-2018.
 Each OI is associated with its set of
Research action (R) (approximate
ratio 1:2)
 3.5 years of certification time per R
Gawdiak et al, January 2010
Raytheon
Dr. George Blaha
 approximately 1.75 years of
certification time per OI
 For example, In 2014, 60 OIs are
expected to take 100+ years to
complete
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
9
Accidents: Flight vs. Crew Errors
4%
4%
6%
13%
56%
17%

Source: Statistic analysis of airplane accidents, Copyright 2005-2014
Source: Human Error and Commercial Aviation Accidents:
A Comprehensive, Fine-Grained Analysis using HFACS; FAA
Raytheon
Dr. George Blaha
Flight Crew

Aircraft
Weather
Miscellaneous
Maintenance
Airport/ATC
Cognitive demand
imposed on pilot during
flight is due to
complexity of the
environment
Flight crew errors
account for 56% of the
causes of accidents in a
year
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
10
New Federal Aviation Regulation (FAR) on Certification
FAR 25.1302
AC 25.1302
Complements existing
regulations to ensure the
“design reduces or avoids
crew errors” and addresses
crew capabilities and limitations”
in task performances
Regulation
focused on
Human Factors
Guidance for the design and
methods of compliance (MOC) for
installed equipment on airplanes to
minimize errors by the crew and
enable crew to detect and manage
errors.
AC 25.1302 (b)
Supports the crew in planning
and decision making, reduces
errors by making decision
requirements
Certification:
Process that
officially
recognizes if
device meets FAA
standards.
Raytheon
Dr. George Blaha
AC 25.1302 (b2)
1. Heavily based
on Task
Performance
2. First
regulation
requiring
examination of
ALL tasks
Enforces accessible and usable control
and information by the flight crew in a
manner consistent with urgency,
frequency and task duration.
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
11
Regulation Guidelines (Advisory Circular)
4 MOC:
•
•
•
•
Bench Test
HITL
Simulator Testing
Inspector Testing
Source: AC 25.1302
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
12
Tasks from Standard Operating Procedures (SOP)
 Standard Operating Procedures (SOP): covers all
operational demands that are involved for the safety
of the flight
 Test Case: Swiss European RJ100
Sections 2 & 3
describe flight
procedures
Raytheon
Dr. George Blaha
Item
Count
Procedures
104
Tasks
642
Sub-tasks
1,263
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
13
Certification vs. Time on Task
Initial Approach Procedure
Hold at Present Position Task
1. Press HOLD
2. Press 6R
3. Review the default holding
patterns/revise as necessary
4. Press EXEC to activate holding
patterns
Procedure
Tasks of Pilot
Operator Actions
Monte Carlo Simulation: Histogram
Number of
Operators
performing
task
Probability of failure to
complete the task
Source: Procedure from RJ 100 FMS Pilot’s Guide REV 2.0, 2010
Raytheon
Dr. George Blaha
The Histogram indicates
that operators can fail if not
completed in time.
Time to complete the
task [sec]
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Decomposition of Tasks: Task Specification
Language (TSL)
•
•
1
2
14
General decomposition follows TSL framework
Decomposition allows the mapping of finite actions to
statistical distributions
Stimulus
Identify Task
Select Function
3
Access Function
Enter
4
Confirm / Verify
5
6
Raytheon
Dr. George Blaha
Monitor
Hold at Present Position Task
1. Press HOLD
2. Press 6R
3. Review the default holding
patterns/revise as necessary
4. Press EXEC to activate holding
patterns
The title of the RTE HOLD page shows
ACT (active) when the holding pattern
is being flown
Source: Procedure from RJ 100 FMS
Pilot’s Guide REV 2.0, 2010
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Decomposition of Tasks: Task Specification
Language (TSL)
15
• General decomposition follows TSL
framework
• Decomposition allows the mapping of finite
actions to statistical distributions
Stimulus
1 Identify Task
2
3
Select Function
Access Function
4
Enter
Confirm / Verify
5
6
Monitor
Stimulus:
ATC call: Swiss XX Hold at Present Position, 1 nm
legs, right turns, ECHT 1220Z
TSL Matching of corresponding procedure
Pilot Identifies need to hold at present position
Pilot mentally selects the HOLD and 6R buttons
from the Multifunction Control Display Unit
(MCDU)
Pilot Accesses the HOLD function followed by the
6R function
Pilot Enters selection (press buttons)
Pilot verifies the entry is correct
Pilot presses the EXEC button
Pilot monitors the change in the HOLD page
Source: Procedure from RJ 100 FMS Pilot’s Guide REV 2.0, 2010
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Additional Work for Inspectors vs. Number of
Inspectors Constant
GAP
𝟏.𝟕𝟓∗# 𝒐𝒇 𝑶𝑰𝒔
•
•
•
1.75 years per OI using 1:2 ratio
FAR 25.1302 is the first regulation that requires checking ALL
functions in a new device  more labor (inspectors and DER)
required per new device certification
The inspectors’ capacity for new devices certification is very
small compared to the amount of devices seeking certification
as a result of the NextGEN technology.
Policy Impacts of Implementing NextGen Research: SUMMARY REPORT, January 2010
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
16
Opportunity
•
•
•
•
Many new devices are coming into the market because of Modernization Program
Regulations are shifting to make certification a more thorough process
Current capacity is not sufficient
All the above create opportunity for a Procedure Analysis Tool (PAT)
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
17
18
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
19
Stakeholder Analysis – Broken System
FA
A
Tension
DE
R
Raytheon
Dr. George Blaha
Broken System
DERs perform certification test
Tension
report to FAA inspector during
certification process while
being employed by applicant
certification
(Aviation
Department of Systems Engineering and Operations Researchfor
SYST
490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification
Process
Manufacturer)
20
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
21
Problem Statements vs. Proposed Solutions
Problem
Solution
Stakeholders tension created by the fact
that the DER (employed by the applicant)
reports at the same time to the FAA
inspector during the certification process
PAT will create a win-win by relieving
stress from DERs and giving inspectors
objective and quantifiable measures to
evaluate functions
Expensive and time consuming
Simulation will drastically reduce the
certification time and associated costs
Lack of cockpit interaction in the current
certification process
PAT will model the flight operators’
interactions within the cockpit environment
using the SOPs
Current Means of Compliance are
subjective
PAT will generate decision output based on
human based model with clear fail/pass
threshold
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
22
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Mission Requirements
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
23
Objectives Hierarchy
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
24
25
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
26
Technology Requirements
Technology
Feature
GUI
Database
Lookup
Table
Sequence
Diagram
✓
Sequence
Diagram
Operator Action
✓
✓
✓
Operator Action
Category
✓
✓
✓
Distribution
✓
✓
Monte Carlo
Simulation
✓
Raytheon
Dr. George Blaha
Algorithm
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
✓
✓
27
Prototype: Procedure Analysis Tool (PAT)
 Physical Process Modeled  Perceptual, Cognitive, and Motor
Process
Time to Complete Task
Perform a Procedure
(SOP)
# Operators
Standard
Operating
Procedure
Human Factors
Metrics database
Perception,
Cognition, Motor.
Time to
Complete Task
Note: Human Factors Metrics are statistical distributions expressed in [sec].
Ex time to see a cue: Norm~(.263, 0.011)
Modeling Psychological Refractory Period (PRP) and Practice Effect on PRP with Queuing Networks and Reinforcement Learning Algorithms, Liu et al.
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
28
Prototype
 Step1: decompose SOP into tasks, sub-tasks, operators actions (OA)
and identify devices  build a sequence diagram
DER
 Step2: categorize (OA) into categories  Operator Actions
Categories (OAC) from the system database database
 Step3: map the OA to their corresponding statistical distribution
according to their Operator Action Category
 Step4: for each task, aggregate the times (random variables
generation) of all Operator Actions
PAT
 Step5: perform Monte Carlo simulation and build a frequency
distribution
 Step6: identify the resulting type of distribution and the threshold
for pass/fail. Ex: 3σ for a 99.7% confidence interval
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Prototype – Step1
 Holding at the present position of the aircraft:
1. Press HOLD
2. Press 6R to hold at the present position of
the aircraft
3. Review the default holding pattern
parameters and revise if necessary
4. Press EXEC to activate the holding pattern
The title of the RTE HOLD page shows ACT
(active) when the holding pattern is being
flown
When the holding pattern is being flown, the
EXIT HOLD option shows on the RTE HOLD
page for exiting the holding pattern
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
29
Prototype – Step 1
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
30
31
Prototype – Step 2 &3
KLM Category
button push
TSL Identify (Decision)
TSL Select
flight controls manipulation (feet)
flight controls manipulation (hands)
listen to ATC Clearance
Readback to ATC
Multi Crew Coordination - callout
Multi Crew Coordination - readback
memory checklist retrieval
thrust levers manipulation
visual instrument check
provide visual cue
Distribution
Normal(0.3, 0.01)
Normal(3, 1)
Normal(3, 1)
Lognormal(1, 0.2)
Lognormal(1.5,0.2)
Exponential(1, 0.1)
Exponential(0.9, 0.1)
Normal(2,1)
Normal(1.5, 0.2)
Lognormal(1, 0.05)
Triangular(0.1,2,3.5)
Normal( 0.632, 0.011 )
Normal( 0.5, 0.01 )
Table – Operator Action Categories (OAC) & Statistical Distributions
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
32
Prototype – Step4
Random values
generated according to
pre-defined statistical
distributions
Table: Tasks Mapping to OAC and Distributions
sequence #
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
sequence description (from Magic Draw)
SWISS XX Hold at present position, 1 nm leg, right turn,
ECHT 1220Z
Hold at present position, 1 nm leg, right turn, ECHT
1220Z. SWISS XX
decide: Present position, existing wypt in flightplan, or
specific position
dcide to hold present position
memory item
HOLD mode key
press the HOLD button
R6 key
press R6
review default holding patterns
EXEC key
press EXEC
EXEC light OFF
change of AC state
RTE HOLD shows ACT
Raytheon
Dr. George Blaha
entity performing task category
task time
ATC
Listen to ATC Clearance
1.033462118
flightcrew
Readback to ATC
0.919539452
flightcrew
flightcrew
checklist
automation
flightcrew
automation
flightcrew
flightcrew
automation
flightcrew
automation
aircraft
automation
Decision (TSL Id)
TSL Select
memory checklist retrieval
provide visual cue
button push
provide visual cue
button push
visual instrument check
provide visual cue
button push
provide visual cue
n/a
visual instrument check
Task Time
2.746652897
3.025068908
1.105461353
1.105461353
0.309876543
1.105461353
0.309876543
0.639847381
1.105461353
0.309876543
1.105461353
1
0.639847381
16.46135453
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
33
Prototype – Step 5
Distribution 1
for OAC 1
Distribution 2
for OAC 2
Distribution n
for OAC n
…
Run 1: t1,1 + t1,2 +… + t1,n = Trun1
Run 2: t2,1 + t2,2 +… + t2,n = Trun2
replications
Monte Carlo simulation of
Procedure Time
Procedure composed of n tasks
….
….
Time to complete the tasks [sec]
Run m : Run m: tm,1 + tm,2 +… + tm,n = Trun m
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
34
Monte Carlo Simulation
Number of
Pilots
Completing
Hold at PP
Task
Probability of failure
to complete the tasks
Time to complete the tasks
[sec]
Devices associated with high probability of failure to
complete the tasks will be unqualified for certification
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
35
Human Factors Database Distributions

Statistical distribution from
regression model (ex:
Communication)

Regression model from data
collection

Experiment design:

48 radio calls using
ATC phraseology


5 team members
Data collection time as
a function of number of
words

Regression models results
will be validated using
actual pilot operations.
Arriving:
Pilot: Lancaster Tower, Cherokee
8121K, 10 [miles] southwest at
2,500 inbound for landing with
[information] Sierra.
Tower: Cherokee 8121K, Lancaster
Tower, report entering left
Department of Systems Engineering and Operations Research SYST 490 – 2014
downwind Runway 31.
Dr. George Blaha Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Raytheon
36
Prototype – Statistical Distributions Elicitation
 Mathematical principles used to derive distributions:
 Button push Prototype – Step3 & 4
• Fitts’s Law
•
•
•
•
•
Time
W= target width
D= distance between target and user
T= movement time
a= intercept of the fitted line
b= slope of the fitted line
D
W
Index of difficulty
Location
(distance from the operator)
Experiment design:
4 main compartments will be located
5 team members will reach out and
time performance
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
37
Validation Plan
Two sets of validations will take place:
1. Validation of OAC Human Factors distributions (provided by
regressions) against literature review and experiments
2. Survey of RJ100 pilots on an “unpractical” device and run
the simulation on it
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
38
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
39
PAT: Monte Carlo Simulation
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
40
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
41
Business Case Assumptions
 Alternative 1:
 70% of the certification process is used for human in the loop tests
 30% of the certification process is used for inspection and bench tests
 Alternative 2:
 70% of the certification process is used for procedure analysis tool
that was proposed
 20% of the certification process is used for human in the loop tests
 10% of the certification process is used for inspection and bench tests
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
42
Cost Regarding Means of Compliance
Inspection
Human in the Loop
Procedure Analysis Tool
The years after software license paid:
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
43
$300,000,000.00
$250,000,000.00
$200,000,000.00
$150,000,000.00
$100,000,000.00
$50,000,000.00
$0.00
Cumulative Current
System Cost/Year
Cumulative Alternative
Cost/OI per Year
2008
2010
2012
2014
2016
2018
2020
2022
2024
2026
2028
2030
Cost in Millions
Savings in Cost
Years
3000000
2000000
1000000
2030
2028
2026
2024
2022
2020
2018
2016
2014
2012
2010
0
2008
Number hours
saved
Savings in Time
Cumulative Hours
Cumulative Hours
Current System
Alternate System
CumulativeHours
Hours
Cumulative
Alternative
Current
System
Years
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
44
Agenda








Context
Stakeholders Analysis
Problem Statement vs. Proposed Solution
Objectives Hierarchy and Requirements
Prototype: Procedure Analysis Tool (PAT)
PAT: Monte Carlo Simulation
Business Case - Alternatives
Management
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
45
Work Breakdown Structure
Procedure
Analysis Tool
Management
and Research
Planning
Context
Analysis
Problem/Need
Statement
Concept of
Operations
Requirements
Results
Documentation
Analysis
Budget
Risk Analysis
Deliverable
Preparation
Poster
Stakeholder
Analysis
System
Alternatives
Conclusion
Paper
Time Sheets and
Weekly
Accomplishment
Summary
Raytheon
Dr. George Blaha
Modeling and
Simulation
Recommendation
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Conference
Preparation
46
Critical Path
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
47
Budget
Wage/hour for team = $45
GMU Overhead = 2.13
Total Hourly Rate = $95.85
Total Estimated Time
803 Hours
Cumulative Planned Value
$76,967.55
Best Case Scenario
$65,422.42
Worst Case Scenario
$88,512.68
Total Cumulative Actual Time
775 hours
Total Cumulative Actual Cost
$74,283.75
Cumulative Earned Value
$73,847.63
Overall Cost Variance
$13,414.21
Overall Schedule Variance
0.96
Cost($)
Earned Value
100000
90000
80000
70000
60000
50000
40000
30000
20000
10000
0
Cumulative PV
Cumulative AC
Earned Value
Best Case
Worst Case
1 2 3 4 5 6 7 8 9 10 11 12
Weeks
CPI vs. SPI
CPI
0.994
SPI
0.959
Raytheon
Dr. George Blaha
Ratio
1.5
1
CPI
0.5
SPI
0
1
2
3
4
5
6 7 8
Weeks
9 10 11 12
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
48
Risk Priority Diagram
Descriptor
Severity (S)
Likelihood (L)
Ability to Detect (D)
Risk Priority Number
(RPN)
Team Member unable to
contribute (sickness or
school overload)
5
8
3
120
Unable to categorize a
subtask into an Operators
Action Category
5
6
1
30
Unable to find a distribution
relating to the Operator
Action Category in time
7
7
1
49
Software limitations; may
not be compatible or not
available at deliverables
5
4
2
40
Severity (S) of each failure: 1 represents failure with no effect and 10 represents very severe
and hazardous failure
Likelihood (L) of each failure occurring: 1 indicates that failure is rather remote and not likely
to occur and 10 indicates that failure is almost certain to occur
Detection (ability of to detect) – 1 used when monitoring and control systems are almost
certain to detect the failure and 10 where it is virtually certain the failure will not be detected
Risk Priority Number (RPN) – Multiply S, L, and D together
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
49
Risk Mitigation
 Have multiple software options in case limitations are
encountered software.
 If unable to categorize a subtask or find distribution relating
to an Operators Action Category, assumptions will be made
to come up with dummy distributions.
 Knowledge will always be shared among team members
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Special Thanks
 Dr. Sherry
 Dr. Donohue
 Dr. Laskey
 Dr. Berg
 Dr. Adelman
 Dr. Costa
 Siamak Khaledi
 David Hughes
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
50
51
Questions?
Raytheon
Dr. George Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
52
Current Means of Compliance Continued…
Cost of Human in the Loop
Raytheon
Dr. George Blaha
Cost of Inspection
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
Cost for Procedure Analysis Tool
Raytheon
COST
PER
Dr. George
Blaha
Department of Systems Engineering and Operations Research SYST 490 – 2014
Design
of a Procedure Analysis Tool (PAT) for the FAA Human Factors Certification Process
YEAR
53