Project on Unmanned Aircraft in the NAS
Final Review Panel Meeting
1
Integration of Unmanned Aircraft
into the National Airspace System
A Project Course by
Carnegie Mellon University
Dept. of Engineering and Public Policy
Dept. of Social and Decision Sciences
May 1, 2007
2
Expert Review Panel













Tom Curtin, AUVSI
Bret Davis, AUVSI
Lexa Garrett, America West Airlines
Jim Geibel, GAO
David Gerlach, FAA
Tom Henricks, Aviation Week
Ramon Lopez, Aurora Flight Sciences
Edmond Menoche, GAO
Rene Rey, FAA
Melissa Rudinger, Aircraft Owners & Pilots Assn.
James Sizemore, FAA
Larry Thomas, GAO
Dyke Weatherington, DoD/OSD
3
Purpose of CMU Project Courses in
Technology and Policy
 Analyze a “real world” policy problem
involving technology
 Combine diverse information and analytic
frameworks to derive policy insights
 Learning objectives:
 Problem decomposition, structuring and
formulation
 Interdisciplinary problem solving
 Communication
 Teamwork
4
Examples of past project courses
Title
Year
Safety and the Light Truck Craze: Who Wins? Who
Loses? Who Cares?
2000
Environmental Impacts of E-commerce - A case study
of book purchasing
2000
Sustaining Pittsburgh's Vital Services When the Power
Goes Out
2004
Wireless Communications Systems for Emergency
Responders
2004
Hybrids and Diesels in the American Automobile Fleet
2005
The Impact of Spyware
2005
Safety of Implanted Cardiac Devices
2007
5
Contributors to our UAS project
 20 undergraduates majoring in:
 Engineering
 Social Science
 Business Administration




3 Ph.D. student managers
3 faculty advisors
Expert review panel
Other experts
6
Background for this Project
 Increasing demand for UA
 Military (many current uses)
 Civilian (many potential uses)
 Federal Aviation Administration (FAA)
is developing a roadmap for
integrating UA into the NAS
 A few of the issues to be addressed:
 Safety and reliability
 Public acceptability
 Market viability
7
Analysis Areas
 Economics
 How cost-effective are UA compared to alternative
means of providing specific services?
 Risk, Technology and Standards
 What are the regulatory implications of different
approaches to “equivalent level of safety?”
 Public Awareness and Perceptions
 Are risks of UA of greater public concern than risks
of manned aircraft?
 Governance
 How can the current system for deliberation and
decision-making on UA access be improved?
8
Project Outcomes
 ~16 person-months of research completed
across the four focus areas
 Economic model of market viability
 Risk model of fatality implications of UA
introduction
 Better understanding of public awareness &
risk perception
 Actor & “roadblock” analysis yields insight
on deliberative process for UA integration
 Regulatory & policy recommendations
9
Economics
Team Members:
Nathan Diorio-Toth
Feng Deng
Reiko Baugham
Victoria Morton
Brad Brown
Team Manager:
Ryan Kurlinski
10
Purpose
 Assess the market viability of UAS
applications using relative cost
effectiveness
 Assess the effect of various
regulatory measures on the market
viability of UAS applications
11
Goals
 Develop UAS cost model
 Cost components





Airframe
Communications
Insurance
Pilot
Etc.
 Apply cost model to chosen applications
and alternatives to compare cost
 Examine sensitivity of overall cost to
changes in each cost component
 Estimate cost implications of different
regulatory measures and technology
improvements
12
UAS Applications
 Weather Reconnaissance
 Alternative:
 WC-130J Hercules: high-wing, medium range
aircraft
 Pipeline monitoring
 Alternative:
 Concentric sensors: pressure sensitive sensors
 Localized Surveillance
 Alternative:
 Traffic Helicopter: e.g. Bell JetRanger
13
Analysis Method
 Used triangular distributions to assign
probable ranges to each input cost
 From this, generated a Probability
Density Function
 Probability Density Function shows the
entire range of possible costs with the
associated likelihood of each cost
 Allows analysis of the most probable cost
advantages
14
Importance Analysis
Contribution of
uncertainty in
each input to
uncertainty in
total cost
Triangular
probability
distributions of
all input variables
Economic
Model
15
Weather Reconnaissance
 Analyzed the use of Aersonde UAS for
Weather Reconnaissance vs. the use
of the WC-130J Hercules
 Aerosonde UAS currently in use for
Weather Reconnaissance
 Hercules WC-130J currently in use by
Keesler Air Force Base
16
Results: Weather Reconnaissance
Probability Density
12u
Probability Density of UAS
Cost Advantage
10u
8u
6u
4u
2u
($/flight hour)
0
0
100K
200K
300K
400K
500K
600K
700K
Alt Cost-UAS Cost
17
Importance in Alt Cost-UAS Cost
Results: Weather Reconnaissance
Importance Analysis of Model Inputs
1
0.8
0.6
Mission Hours
per Year
Operational
Lifetime Com-Link Cost
0.4
0.2
0
Component costs
Manpower
Com-Link Cost
Operational lifetime
Cost per gallon
Insurance Rate
Safety Technology Cost
Hours per year
Gallons per Hour
Alt Cost-UAS Cost Inputs
18
Results: Weather Reconnaissance
 Key Results:
 UAS more cost effective than current
manned alternative
 Most important inputs in determining overall
cost effectiveness:
 Mission hours per year
 Com link cost
 Operational lifetime
 Currently available sense-and-avoid
equipment cause significant decrease in cost
effectiveness, but does not cause the UAS to
be more expensive than the manned
alternative
19
Pipeline Monitoring
 Analyzed the use of the Aero Environment
AeroPuma vs. the use of concentric wire
sensors ($6+/m)
 Note the difference in monitoring style
 UAS monitors using thermal imaging with each
pass and relays pertinent leak info to docking
stations
 Concentric sensors constantly monitor pipeline
and relay information
20
Results: Pipeline Monitoring
 Key Results:
 UAS cheaper depending on number in
use
 Important to note difference in
monitoring styles between UAS and
concentric sensor
 Important inputs:
 Relay/Docking station cost
 Number of UASs in use
21
Localized Surveillance
 Application based on the surveillance of a
1km2 area for a short time (~1-3 hours)
 Considered the use of a Cyber Defense
Systems CyberBUG vs. the use of a traffic
helicopter
 For model inputs, considered monitoring a
large traffic accident over 2 hours
 For policy considerations, analyzed the
addition of mandated sense-and-avoid
hardware to the UAS
22
Results: Localized Surveillance
Probability Density
PDF of Cost per Mission for UAS Compared
with Manned Alternative
Note: no meaningful overlap
2000 4000
6000 8000 10K
Cost per Mission ($)
12K
14K
16K
23
Probability Density
Results: Localized Surveillance
PDF of Cost per Mission for UAS Compared
with Manned Alternative with High-Range
Fixed Cost Variance
Note: still no meaningful overlap
2500 5000 7500 10K 12.5K 15K 17.5K 20K 22.5K
Cost per Mission ($)
24
Results: Localized Surveillance
Probability Density
PDF of Cost per Mission for a Larger UAS
Capable of Carrying Sense-and-Avoid
Equipment Compared with the Cost of
Manned Alternative
Note: Significant overlap
indicating that UAS would
likely no longer be a viable
alternative to manned craft
0
10K
20K
30K
40K
50K
60K
70K
Cost per Mission ($)
25
Input Importance for Cost Per Mission
Results: Localized Surveillance
Missions
per Year
Mission
Related Costs
Flight Hours Per
Mission
Importance of inputs.
Input Costs
26
Results: Localized Surveillance
 Key Results:
 UAS less expensive in almost every case
 Levelized cost for manned more sensitive than
to utilization hours & discount rate than cost
for unmanned
 UAS cost effectiveness reduced significantly
by requirement for sense-and-avoid hardware
 Important inputs:
 Missions per year
 Discount rate
 Flight hours per mission
27
Policy Implications
 Analyzed the effect of the following policies:
 Mandated insurance premiums
 Mandated use of A/N hardware

(Increased fixed cost)
 Mandated record-keeping practices

(Increased yearly cost)
 Mandated airframe materials

(Increased fixed cost)
 Mandated minimum amount of safety
equipment

(Increased fixed cost)
 Mandated pilot/operator training
28
Policy Implications: Results
 All policies except mandated sense-and
avoid hardware had little effect on the
cost advantage of UAS over manned
alternative
 Required sense-and-avoid hardware
greatly affects cost-effectiveness,
however
 Localized Surveillance and Pipeline
Monitoring would no longer be viable as
larger, much more expensive UAS would be
necessary
29
Risk, Technologies, & Standards
Team Members:
Samiah Akhtar
Jonathan Cornell
Nicole Hayward
Will Kim
Nick Misek
Doug Robl
Team Manager:
Keith Florig
30
Purpose
 Derive a risk model to explore how risk is
related to UAS numbers, dimensions, and flight
zones
 Research on elements of risk mitigation such as
human factors, sense and avoid
 Exploration of alternative incident reporting
systems
Predator
Source:
http://www.fs.fed.us/psw/news/PSW_
News/2005_09/images/uav.gif
31
Technology and Risk Outline
 Goals
 Risk Modeling
 Purpose
 Assumptions & Approach
 Findings
 UAS Risk Mitigation
32
Risk Modeling Purpose
 Provide a way of modeling that creates some
groundwork for future modeling
 Use model to compare relative risk calculations
 Pointer to the future, not the answer
 Points of interest
 Mid-air vs. single-craft crash
 Effect of sense and avoid technology
 UAS to displace manned aircraft
Source:
http://www.maximog.com/ima
ges/sublevel/uav_left.jpg
33
Risk Modeling Assumptions





Uniform national model
Uniform traffic density
Uniform ground population density
Uniform aircraft per type
Appropriate for:


VFR traffic
Rural, less populated areas
 NOT Appropriate for:



Urban settings
Airports
High traffic densities
34
Risk Model
35
Risk Modeling Approach
Number of midair collisions:
N    [1  P( A)]  D 2  S  31  10 6 sec/ year
N
= total number of aircraft
in defined airspace
ρ
= aircraft traffic density
D
= diameter of plane
(wingspan)
S
= average aircraft speed
P(A) = probability of avoidance
(Used for calibration)
VFR operations only
UAV Picture Source:
http://www.evworld.com/press/spider_lion_uav.jpg
36
Risk Modeling: UAs displacing Manned
Expected Annual Fatalities vs. % of Airspace Unmanned
Expected Annual Mortality
14
12
10
8
6
4
2
0
0%
Small risk
from
10%
20%
unmanned
at lower
Risk
from
extrema
unmanned at
low levels less
than decreased
risk from
manned
30%
40%
50%
60%
70%
80%
90%
100%
% of Total Aircraft is Unmanned
Single-craft
crashes still
present less
risk than midairs
Manned Mid-Airs
Manned Single-Craft
Unmanned Mid-Airs P(Avoid)=0
Unmanned Mid-Airs P(Avoid)=0.5
Unmanned Single-Craft 100 failures/100k flight hrs
Unmanned SIngle-Craft 10 failures/100k flight hrs
Unmanned Single-Craft 500 failures/100k flight hrs
37
Risk Modeling: Mid-Air vs. Single-Craft
Expected Annual Mortality vs. % UASs Added to NAS
Expected Annual Mortality
40
35
30
At some point,
manned risk
surpasses
unmanned risk
25
20
15
10
5
0
0%
20%
At low
numbers,
sense and
avoid has
little effect
40%
60%
80%
100%
120%
% UASs Added
140%
Single-Craft 50 failures/100k flight hrs
Single-Craft 200 failures/100k flight hrs
Mid-Airs P(Avoid)=0
Mid-Airs P(Avoid)=0.25
Mid-Airs P(Avoid)=0.5
Mid-Airs P(Avoid)=0.95
Single-Craft From Substituting Manned
Mid-air from Substituting Manned
160%
180%
200%
Single-Craft
generally less
risk than mid-air
38
Risk Modeling Conclusions




Mid-air collisions generally have more risk
than single-craft crashes
Displacing small to moderate amounts of
manned craft represents decrease in risk
Smaller, less reliable UAs can present less risk
than larger more reliable manned aircraft
For small numbers of UAs in low traffic
densities, sense and avoid has small effect
39
Technology and Risk Outline
 Goals
 Risk Modeling
 UAS Risk Mitigation
 Human Factors
 Sense and Avoid
40
Human Factors Implications
 Risks - Caused Most Number of Accidents
 “Sensory Isolation” [McCarley et al]
 UAS operator does not receive same
sensory cues as manned aircraft operator
 Automation
 Malfunction of automated components
controlled by the UAS operator
 Operator Hand-Off
 Issues with handing off control of vehicle
from one operator or crew to another
41
Human Factors Implications
 Recommendations
 Training and Procedures
 Up to date training as new technology
advancements arise
 Ensure that operator has accurate
knowledge of automated components
within UAS
 Multimodal displays
 Prevent sensory isolation
 Allow for audio, visual and speech control
 Example: simulated cockpit
42
Detect, Sense and Avoid
 Risks
 Market impact of single fatal collision
 Lack of standardization among DSA
systems
Sense and Avoid
Technology
Cost*
Size*
Weight*
Power
Usage*
Pros
Cons
IFF Transponder
$500-3000
1.5m2
6 kg
30 watts,
nominal
Low-cost,
Transponder
requirement
ADS-B
Transponder
$2000-6000
2.0m2
8 kg
50 watts,
nominal
Compatible
with NGATS
High
power,
transponder
EO/IR Sensors
$50,000
200,000
4.0m2
5-10 kg
10-20
watts
Detection
in
IFR and VFR,
no
metal
requirement
Short detection
range,
high
cost
Synthetic Aperture
Radar
$10,000 50,000
3.0m2
20 kg
80 W
All-weather
Weight,
power
-
high
43
Detect, Sense and Avoid
 Recommendations
 Create regulations specific to size,
weight, application etc
 Testing Periods
 Phased Integration
44
Technology and Risk Outline




Issues
Goals
Risk Modeling
UAS Risk Mitigation
 Reporting systems
45
Current Reporting Systems
 Two Options
 NTSB Reporting (as required by FAR) - Accident
 NASA ASRS Voluntary Reporting - Incident
 Current Implementation
 NTSB mandates detailed information when:
 Flight control system malfunction, Illness of
crewmember, Turbine Engine Failure, Inflight fire, Mid-air collision or Damage in
excess of $25,000 to other property
 ASRS System is anonymous and does not have
any reporting requirements
46
Reporting Recommendations
 Initially mandate reporting of all accidents and incidents
 Re-evaluate strategy after testing period
NTSB
- NTSB information helps FAA
to assess standards
- NTSB provides useful
information on UAS failures
- FAA responds with rules for
reporting incidents.
- UAS responds with improved
design and engineering
Communication
Triangle
FAA
UAS
manufacturers
47
Public Awareness & Perceptions
Team Members:
Darian Ghorbi
Jenny Kim
Mark Peterson
Laura Seitz
Patrick Snyder
Team Manager:
Pete Tengtrakul
48
Statement of Purpose
 Add the element of public perception to the
discussions of UAS in the NAS
 Motivation: the fact that there has never been
a formal presentation of public perception on
the topic
 Findings: useful for the creation of regulations
and policy implications
49
Objectives
 Compare public perceptions of the risks
concerning manned and unmanned
aircraft
 Find demographic groups with certain
risk and benefit patterns of UAs
 Research implications of opinion of UAs
 Create survey to aid in completing
objectives
50
Hypotheses
 Perceived Risk
- Manned < Remotely Piloted < Autonomous
 Ground vs. Air
- More risk of UAS perceived in air
 Prior Knowledge vs. Risk Perception
- Prior knowledge, associate less risk
 Benefit vs. Risk Perception
- Higher benefit, lower risk
 Education vs. Risk Perception
- Technical education, associate less risk
 Age vs. Risk Perception
- Older participants more cautious
 Frequency of Flight
- Those that fly frequently, associate less risk
51
Layout of Survey


First Page
 Provide information about UAS
 Autonomous
 Remotely Piloted
 Gauge previous knowledge
 Source
Last Page
 Demographics
 Gender
 Age
 Education
 Frequency of Flight
 Voting (identify opinions of those that are politically
engaged)
 Pilot
52
Layout of Survey
Application
•Traffic Monitoring
•Pipeline Monitoring
•Disaster Relief
•Border Patrol
Questions
•
Quick Response
•
Benefit
•
Picture of UAS
application
•
Description of UAS
•Physical Information
•
Stakeholder
•
Public
Risk
•
Ground
•
Air
7 Point Scale
•
1 - Much Less
•
4 - Same
•
7 - Much More
•Current application
53
Obtaining Surveys
Method
Mon Valley
NGO
Word of
Mouth
Total
# of Surveys
79 (56%)
62 (44%)
141
Coding: numerical code assigned
Screening: data obtained from those under 16 years
of age were not counted
54
Statistical Methods
 Paired T-tests
 Across applications
 ANOVA
 Significance of mean
 Regression
 Correlations
 Demonstrated the strength of the variables (risk
and benefit)
55
Results:
Descriptive Statistics
Demographic Variable
Mean
Scale
Previous Knowledge
Gender
Age
Educa tion
Register to Vote
Freque ncy of Fli ght
Licensed Pil ot
0.53
0.44
36.04
2.33
0.66
2.51
0,01
1= heard, 0 =never heard
1=Male, 0=Female
Years
1=High School,4= Gradua te
1=Registered,0=not
1=never, 5=more than 12x
1=Licensed,0=not
Variable
Remotely Piloted
Autonomous
Support
.75
(0.45)
4.65
(1.57)
4.14
(1.58)
4.36
(1.51)
4.55
(1.39)
.58
(0.50)
4.29
(1.77)
4.01
(1.66)
4.81
(1.66)
4.87
(1.68)
Benefit to Stakeholders
Benefit to Society
Risk to people on Ground
Risk to people in the Air
56
Perceived Risk (1=Low, 7=High)
Perceived Relative Risks Between Remotely
Operated vs. Autonomous
7
6
5
4
Pipeline
T raffic
Border
Disaster
3
2
1
Remotely operated UA
Autonomous UA
0
UA Applications
Autonomous applications are viewed to have more risk in
comparison to remotely operated UAs.
57
Relative Risks Across Applications
Perceived Risk (1=Low, 7=High)
7
6
5
4
3
Pipeline
Traffic
Border
Disaster
2
1
Remotely operated UA
Autonomous UA
0
UA Applications
Traffic Monitoring has the highest perceived risk.
58
Perceived Benefit (1=Low, 7=High)
Relative Benefit Across Applications
7
6
5
4
3
Pipeline*
Border
Disaster
T raffic
2
1
Remotely operated UA
Autonomous UA
0
UA Applications
The more risky the public perceived the application, the
less benefit they associated with the application.
59
Relative Perceived Risk to People on
Ground vs. Air
Perceived Risk (1=Low, 7=High)
7
6
5
4
Pipeline*
Traffic*
Border*
Disaster*
3
2
1
Risk to people on Ground
Risk to people in the Air
0
UA Applications
There is no difference between risk perceived on ground vs.
air. Also, there is no difference between perceived benefit
between stakeholders and society.
60
Demographics and Risk
Age
N
Mean
StDev
Under 25
56
4.340
1.541
36-35
28
4.679
1.786
36-45
13
5.192
1.665
46-55
14
4.964
1.216
56-65
21
5.310
1.512
Over 65
9
2.333
1.871
Those over the age of 65 perceived UAs as least risky and
least beneficial; the mean value is insignificant.
61
Risk Perception Conclusions





Unmanned aircraft risk > manned
Autonomous risk > Remotely piloted
No difference:
 Risk: Ground vs. Air
 Benefit: Stakeholders vs. Society
54% heard of UAS
 Need education programs
 78% of those that heard of UAS obtained information from
television
 The more familiar, the more comfortable
Traffic Monitoring - higher risk
 Fear of operating around high population density areas
62
Impact on Policy
 Limit flight path/area
 Limited population density
 Implement education/outreach
programs
63
Future Insights
 Limitations:
 Time, Resources, and Budget
 Sample
 National scale-different regions
 Future Surveys:





Compare UAS to other risky technologies
Size of aircraft
Privacy concerns
Economics concerns
Lengthened
64
Governance
Team Members:
Nora Darcher
Norma Espinosa
Scott Fortune
Andrea Fuller
Team Manager:
Leonardo Reyes-Gonzalez
65
Purpose
 Evaluate current system of
governance for UAS integration
against principles of good governance
 Suggest measures that could improve
the governance process
66
Analysis






Principles of Good Governance
Rules for FAA governance
Historical Technologies
Actor Interactions
Roadblocks
Cost and Benefits for each actor
67
Characteristics of Good Governance
http://www.unescap.org/pdd/prs/ProjectActivities/Ongoing/gg/governance.asp
68
Governance requirements on FAA
 OMB rule requires FAA standards
adoption procedures to have the
following:





Openness
Balance of interest
Due process
Appeal process
Consensus
69
Historical Analysis
 How did the governance system handle
the introduction of new technologies?
Technology
What We Learned
Supersonic Transport Public Interest NGOs can
have a large impact
(1960s)
Automation of Radar
System
(1970s)
Microwave landing
system (1970s)
Incremental changes are
easier than changing entire
system at once
Intl. adoption of US
standard is advantageous
to US technology firms
70
Actor Analysis
 Objective:
 Provide a systematic assessment of the actors
involved in integrating UAs into the NAS
 Process:
 Identified key actors, examined their goals and
looked at problem from each actor’s perspective
71
Actor Analysis
Resistant
Neutral
For
Hesitant
72
Roadblock Analysis
 Objective: prioritize problems
inhibiting the integration of UAs
 Categories




Technological
Organizational
Infrastructural
Public Concern
73
Roadblock Analysis
74
Roadblock Analysis
# of actors
6
Airspace Access
5
4
Equivalent/Acceptable
Level of Safety
3
2
1
Transponders
Data Acquisition
0
Low
High
Complexity
75
Most obvious needs
 Defining an equivalent/acceptable level of
safety
 Allowing UAS operations in scarcely-used
airspace to facilitate testing and
development for civil and commercial
applications.
 Potentially large public concern about UAS
safety argues for proactive public
involvement in deliberations
76
Conclusions
77
Summary of Conclusions




Economics
Risk, Technologies, and Standards
Public Awareness and Perception
Governance
78
Economics
 Some civil UAS applications seem highly
competitive with alternatives
 Initial policy ought to be tailored to the
most commercially viable applications
 Cost models show that (i) costs are most
sensitive to hours of utilization, (ii) safety
equipment has modest cost effect, except
for small systems using sense and avoid
 Foreign UAS firms may develop an
advantage if they gain airspace first
79
Risk, Technologies, and Standards
 For some applications in some classes of airspace,
unmanned aircraft result in fewer fatalities than
manned aircraft used for the same task
 Sense and avoid is important only in airspace with
significant traffic density
 Low risk areas could be used for experimentation and
testing without posing a high risk to those on the
ground or in other aircraft
 A mandatory incident reporting system has potential
to greatly improve both airworthiness and human
factors reliability
80
Public Awareness and Perception
 All UAS applications surveyed were
considered more risky and less beneficial
than the manned alternative
 Traffic monitoring perceived as most risky
(likely due to flight over dense population)
 About half of participants had heard of UAs
 Those more familiar with UAS technology
perceive less risk
81
Governance
 Integration problem is more complex than
many people realize
 Incremental approach allows for policy
experimentation at low risk (e.g., sparsely
populated areas/airspace)
 Standards need to be established to
provide benchmark and incentive for
manufacturing
 Attention to public perception and
involvement can greatly influence unfolding
of UAS issue
82
Thank You for Coming!
Questions?
83