Federal Aviation
Administration
FAA FLIGHT TEST
RISK
MANAGEMENT
Presented to: Flight Test Safety Workshop
Melbourne, Florida
By: Rod Huete, FAA Test Pilot
John Hed, FAA Flight Test Engineer
Date: May, 2008
Outline
• FAA Flight Test Organization/ Mission
• Safety Challenges as Regulatory Agency
– Specific to type of certification activity
– Generic
• Accidents/Incidents
• Flight Test safety / Risk Management
Program
• Flight Test Safety Database Demo
• Conclusions
FAA Flight Test Risk management
Federal Aviation
Administration
2
Organization
Department of Transportation
FAA
Airports
Air Traffic
Aviation Safety
Aircraft Certification Service
Rotorcraft
Directorate
Small A/C
Directorate
Regional ACO
FAA Flight Test Risk management
Transport A/C
Directorate
Regional ACO
Engine/Propeller
Directorate
Regional ACO
Federal Aviation
Administration
3
Aircraft Certification Service
• The Aircraft Certification Service is the
FAA organization responsible for:
– Certification approval of:
• New and modified aircraft produced by U.S.
Manufacturers (Type Certificates (TC))
• U.S. modifications to aircraft (Supplemental Type
Certificates (STC))
– Validation of foreign certified aircraft operated by
U.S. operators
– Continuing airworthiness of:
• Aircraft with U.S. TCs
• Aircraft operated by U.S. operators
FAA Flight Test Risk management
Federal Aviation
Administration
4
Aircraft Certification Office (ACO)
Responsibilities
• Flight test evaluation of:
–
–
–
–
Performance & Handling qualities
Systems performance (e.g. AFCS)
Aircraft Flight Manual (AFM) procedures
Flight crew human factors
• Approval of Aircraft Flight Manual (AFM)
• Assist in development of new flight test
requirements, advisory material and guidance
FAA Flight Test Risk management
Federal Aviation
Administration
5
TC Projects
• Normally working with well established
design and testing organizations (at least for
the major manufacturers):
– Experienced engineering staff in all disciplines
– Excellent knowledge of products
– Sound understanding of flight test requirements and
advisory material
– Experienced flight test personnel
– Comprehensive procedures for flight testing
FAA Flight Test Risk management
Federal Aviation
Administration
6
OEM’s Flight Testing
• Establishes the limit flight envelope through
progressive, incremental development
• Specifically assigned project pilots (pilot and
copilot) thoroughly familiar with test aircraft, special
limitations and procedures, and status of aircraft
systems
– Handling pilot can generally concentrate on the test
– Copilot can generally concentrate on monitoring aircraft flight
and system status (in addition to ATC, navigation and other
aspects)
– Varied level of responsibility accorded to Flight Test
Engineers/Observers
• Manufacturer’s testing encompasses most required
certification tests
FAA Flight Test Risk management
Federal Aviation
Administration
7
New Aircraft Approvals – Safety Concern
• Generally “end point” with little or no work-up
• Simulators have not always been available to
support certification programs
• Limited use of telemetry
• FAA pilot not as familiar with aircraft as the OEM
pilot
• High workload on:
– FAA pilot; and,
– Manufacturer’s pilot who has to monitor aircraft status, ATC and
navigation, as well as test progress
• Potential for mishandling at limiting conditions
FAA Flight Test Risk management
Federal Aviation
Administration
8
STC Projects
• Many display, cockpit or avionics upgrades
–
–
–
–
•
•
•
•
•
Conversions to glass cockpits
Enhanced vision
Synthetic vision
Electronic flight bag
Passenger to Cargo conversions
Passenger/Cargo to firefighting
Winglets and other drag reduction
Engine upgrades
…….
FAA Flight Test Risk management
Federal Aviation
Administration
9
STC Approvals – Safety Concern
• Little or no development flight testing
• Applicant’s tests often not done by experienced
flight test personnel
• Generally the applicant’s pilot will not be a Test
Pilot:
– Will usually have a good operational understanding of the
aircraft; but,
• May not understand the need for the test
• May not understand the test
• Can be resentful
• Higher probability of unexpected results during FAA
certification tests
FAA Flight Test Risk management
Federal Aviation
Administration
10
Validation Projects
• For all major Transport Category aircraft to
be operated by U.S. operators, FAA conducts
a limited flight test assessment to:
– Determine compliance with retained/unique FAA
technical and/or design related operating
requirements
– Become familiar with the performance, handling and
operating characteristics
– Obtain sufficient knowledge of aircraft to support:
• Continuing airworthiness responsibilities
• U.S. modification activity
FAA Flight Test Risk management
Federal Aviation
Administration
11
Validation Approvals - Safety Concerns
• Low FAA crew proficiency in aircraft type.
• OEM may have very different safety
procedures.
• Limited interaction with OEM so crews not
familiar with each other.
– Often language barriers
– Procedure differences
FAA Flight Test Risk management
Federal Aviation
Administration
12
Generic Safety Challenges
•
•
•
•
Unforeseen event (e.g. independent failure)
Pilot blunder (e.g. inappropriate control input)
Pilot misjudgment (e.g. landing rate of descent)
Loss of situational awareness (e.g. due to fixation
with test)
• Flight Test Engineer or crew blunder (e.g. incorrect
airspeeds, incorrect config.)
• Aircraft deficiency (e.g. incomplete or
misunderstood development tests)
• Continuing tests in adverse atmospheric conditions
and/or inappropriate hours (e.g. due to schedule
pressure)
FAA Flight Test Risk management
Federal Aviation
Administration
13
How We Dealt With These Challenges
In The Past
•
•
•
•
•
Safety process non existent as a formal process
No Flight / Test Safety Officers identified
Test safety dependent on individual test pilot / FTE
experience or “company” processes
Risk assessment / management frequently done real
time
• No collection / distribution of lessons learned
FAA Flight Test Risk management
Federal Aviation
Administration
14
How We Now Deal With These Challenges
• Created a formal safety process (FAA Order
4040.26)
• Flight test safety officers assigned at ACOs
• Formal safety training required
– CRM, Survival, Physiological…
• Sharing of safety practices and lessons
learned
– Safety meetings
– Event reporting
– Flight test safety database
FAA Flight Test Risk management
Federal Aviation
Administration
15
FAA Flight Test Risk management
Federal Aviation
Administration
16
Typical
FAA Accidents and Incidents
FAA Flight Test Risk management
Federal Aviation
Administration
17
FAA Flight Test Risk management
Federal Aviation
Administration
18
FAA Flight Test Risk management
Federal Aviation
Administration
19
FAA Flight Test Risk management
Federal Aviation
Administration
20
FAA Flight Test Risk management
Federal Aviation
Administration
21
FAA CERTIFICATION INCIDENT SUMMARY
DATE
1986
1991
UNK
UNK
UNK
4 / 96
1997 (apprx)
1998 (?)
2 / 99
4 / 99
8 / 99
8 / 99
10 / 99
11 / 99
1 / 00
TYPE
AIRCRAFT
Single Eng
Seaplane
Biz Jet
Biz jet
Small
Helicopter
Pawnee
Small
Helicopter
Large
Helicopter
Single Eng
Ag Cat
biplane
Large
Helicopter
Transport
Single Eng
Medium
Helicopter
Single Eng
DESCRIPTION
REMARKS
Fatality during bailout from unrecoverable spin
Inverted spin during aft c.g. stall test
Aft c.g. stall test, departed controlled flight w/no spin
chute
RWY performance tire failure
MIDO found large bolt wedged in flight controls
Accident
Incident
Incident
Fatality during airspeed calibration test
Cabin step caution light inaccurate
Accident
Incident
Main rotor fuselage strike during Vne throttle chop
Incident
Main gear door bent on simulated engine failure on T/O
test
Aircraft flipped over, nose first, on landing
Incident
Tail rotor strike on landing during H-V test
Incident
Tail strike on auto pilot go-around test
Main gear door bent during Vd test
Hard landing during HV test
Incident
Incident
Incident
Rudder jammed during spin test
Incident
FAA Flight Test Risk management
Incident
Incident
Incident
Federal Aviation
Administration
22
FAA Common Threat Areas
• Infant Mortality
• Runway Environment / Obstructions
• Lack of Confirmation of Applicant’s Test
performance
• Pre-test Airworthiness Inspection
(Conformity)
FAA Flight Test Risk management
Federal Aviation
Administration
23
Current FAA Flight Test safety / Risk
Management Program
• Organizational structure based on system
safety model
• FAA order 4040.26, Flight Safety Program
–
–
–
–
–
Flight Test Risk Management Process
Incident Reporting
Local Quarterly safety meeting
Flight Test Briefing Guide
Accident Response Plan
FAA Flight Test Risk management
Federal Aviation
Administration
24
Current FAA Flight Test safety / Risk
Management Program – Cont’
• Regulatory requirement for Risk Assessment
• Initial and recurrent safety training
– Flight Test Safety course
– Crew Resource Management (CRM)
• Flight Test Safety Database
FAA Flight Test Risk management
Federal Aviation
Administration
25
Organizational Structure
Flight Program Oversight
Committee (FPOC)
Flight Program Manager
(FPM)
ACO Flight Test Manager /
Flight Program Coordinator
(FPC)
Facility FPC
(Optional)
Flight Program Executive
Operations
Manual
Flight Program Flight Safety Officer
(FPFSO)
Local
Operations
Manual
Facility FSO *
FAA Flight Test Risk management
Federal Aviation
Administration
26
FAA Flight Test Risk Management Process
• Risk Assessment required for all tests
– Normally done by development of THAs
• Done by ACO, or Part of acceptable company
process
• TIA signature authority raised to appropriate
level
FAA Flight Test Risk management
Federal Aviation
Administration
27
Incident Reporting/ Database
FAA Flight Test Risk management
Federal Aviation
Administration
28
Flight Test Safety Database
Update
An Online Resource for
Flight Test Safety Planning
FAA Flight Test Risk management
Federal Aviation
Administration
29
Database Background
• Great deal of ‘corporate knowledge’ out there
but not documented and thus ‘dies’ with the
owner.
• Flight Testers are spread out all over the
country. Everyone did safety differently.
• Order 4040.26A required implementation of a
risk management process that:
– Identifies hazards and probabilities
– Identifies/implements mitigating actions
– Estimate and accept risk
FAA Flight Test Risk management
Federal Aviation
Administration
30
Database Background
• Joined up with NASA to develop an online
database of flight test safety info.
• NTPS contracted to gather FAA data.
– Retrieve data from authorities and manufacturers
– “Consolidate” it into master records for each test &
hazard
• Other groups formed to gather military,
development and other data.
– Icing, High AOA, Propulsion…
FAA Flight Test Risk management
Federal Aviation
Administration
31
Database Status & Update
• Substantially complete with FAA Part 23 & 25
data input. Working on Parts 27/29 (Rotary)
• Continuing to gather military and
developmental data
• Continuous small interface improvements
• NEW – Input module now online so you can
input your own data or suggest edits of
existing data.
– Continuing to improve this with “offline” method also
FAA Flight Test Risk management
Federal Aviation
Administration
32
Flight Test Safety Database
http://pbma.nasa.gov/ftsdb/Home.aspx
On-line Demo
FAA Flight Test Risk management
Federal Aviation
Administration
33
Flight Test Safety Database Summary
• The matured THA database IS a wide array of
test safety ideas and suggestions
– But it is NOT an FAA-mandated solution
– And it is NOT an auto-safety planning device
• The matured hazards, causes and mitigations are
necessarily generic
• Your test will have unique problems and will require
unique solutions
• The THA database can be an excellent starting point
FAA Flight Test Risk management
Federal Aviation
Administration
34
Conclusion
• As a regulatory Certificating Agency we
face many flight test challenges
• Past experience has demonstrated that
we are vulnerable to safety
risks/hazards
• We have learned from our mistakes
• We are meeting the challenge by
applying proven safety practices
FAA Flight Test Risk management
Federal Aviation
Administration
35
FTSDB Home Page
FAA Flight Test Risk management
Federal Aviation
Administration
36
FTSDB Login
FAA Flight Test Risk management
Federal Aviation
Administration
37
FTSDB Main Page
FAA Flight Test Risk management
Federal Aviation
Administration
38
FTSDB Home Page
Vmca
FAA Flight Test Risk management
Federal Aviation
Administration
39
FTSDB Results Page
FAA Flight Test Risk management
Federal Aviation
Administration
40
FTSDB Combined THA
FAA Flight Test Risk management
Federal Aviation
Administration
41
FTSDB Results Page
FAA Flight Test Risk management
Federal Aviation
Administration
42
FTSDB Single THA
FAA Flight Test Risk management
Federal Aviation
Administration
43
FTSDB Update Page
FAA Flight Test Risk management
Federal Aviation
Administration
44
FTSDB New THA
FAA Flight Test Risk management
Federal Aviation
Administration
45
FTSDB Single THA
FAA Flight Test Risk management
Federal Aviation
Administration
46
FTSDB THA Input Wizard
Summary
FAA Flight Test Risk management
Federal Aviation
Administration
47