Project on Unmanned Aircraft in the NAS
Mid-semester Review Panel Meeting
1
Expert Review Panel








Tom Curtin, AUVSI & ONR
Jim Geibel, GAO
Tom Henricks, AW&ST
Ramon Lopez, Aurora Flight Sciences
Molly Macauley, Resources for the Future
James Sizemore, FAA
John Walker, RTCA SC-203
Dyke Weatherington, DoD/OSD
2
Integration of Unmanned Aircraft
into the National Airspace System
A Project Course by
Carnegie Mellon University
Dept. of Engineering and Public Policy
Dept. of Social and Decision Sciences
March 5, 2007
3
Purpose of CMU Project Courses in
Technology and Policy
 Analyze a “real world” policy problem
involving technology
 Combine diverse information and analytic
frameworks to derive policy insights
 Learning objectives:
 Problem decomposition, structuring and
formulation
 Interdisciplinary problem solving
 Communication
 Teamwork
4
Examples of past project courses
Title
Year
Safety and the Light Truck Craze: Who Wins? Who
Loses? Who Cares?
2000
Environmental Impacts of E-commerce - A case study
of book purchasing
2000
Sustaining Pittsburgh's Vital Services When the Power
Goes Out
2004
Wireless Communications Systems for Emergency
Responders
2004
Hybrids and Diesels in the American Automobile Fleet
2005
The Impact of Spyware
2005
Safety of Implanted Cardiac Devices
2007
5
Contributors to our UAS project
 20 undergraduates majoring in:
 Engineering
 Social Science
 Business Administration
 3 Ph.D. student managers
 3 faculty advisors
 Expert panel review
6
Role of Expert Review Panel
 At mid-semester:
 Review project team’s progress and
plans
 Suggest changes in project direction
 Suggest better ways to obtain data and
conduct analysis
 At semester’s end:
 Provide feedback on draft of final report
7
Background for this Project
 Increasing demand for UASs
 Military (many current uses)
 Civilian (many potential uses)
 Federal Aviation Administration (FAA)
is developing a roadmap for
integrating UASs into the NAS
 A few of the issues to be addressed:
 Safety and reliability
 Public acceptability
 Market viability
8
Project Strategy
 Pursue a set of analyses that will advance
understanding of the UAS-NAS integration
problem, making best use of the skills of
project members, while being realistic about
the project's resource and time constraints
9
Analysis Teams
 Technology and Risk
 What are the regulatory implications of different
approaches to “equivalent level of safety?”
 Public Awareness and Perception
 Are risks of UAS of greater public concern than
risks of manned aircraft?
 Economics
 How cost-effective are UASs compared to
alternative means of providing the same service?
 Governance
 How can the current system for deliberation and
decision-making on UAS access be improved?
10
Technology and Risk
Team Members
Samiah Akhtar
Jon Cornell
Nicole Hayward
Will Kim
Nick Misek
Doug Robl
Manager
Leonardo Reyes-Gonzalez
11
Outline
 Issues that we plan to address
 Our approach
 Deliverables
12
Issues to be addressed
1.
What are the implications of FAA’s “equivalent level of
safety” (ELOS) goal for target UAS failure rates?
2.
What are current UAS failure rates and how much can
they be reduced by safety technologies such as
transponders, see and avoid capacity, ADS-B
transceivers, AN hardware, etc?
3.
How can the FAA/industry measure and track the
safety performance of unmanned aircraft?
13
1. Implications of Equivalent Level of Safety
 ELOS: FAA’s goal is that risks to people in the air
and on the ground from unmanned aircraft must
be no greater than those from manned aircraft.
 Practical Interpretation: “Equal” risk means
equal number of catastrophic events nationwide.
Why?
 Low probability events causing large numbers of
casualties have been most important drivers of policy
 e.g. 1956 Airplane collision over Grand Canyon
14
1. Implications of Equivalent Level of Safety
UA catastrophic risk model
Catastrophes
caused by
+
midair
collision
Catastrophes
caused by
UAS
malfunction
State and
control variables
=
Annual UA
catastrophes
Baseline annual
manned
catastrophes
15
UA catastrophic risk model
Midair collision
catastrophe rate
Wingspan & mass
# of UAs in airspace
# of other vehicles in airspace
Population flying
Population density on ground
Malfunction
catastrophe rate
Mass
# of UAs in airspace
Reliability (Failures/Hr*)
Population density on ground
ELOS
constraint:
Catastrophes
per year for
manned
aircraft
*Office of the Secretary of Defense. Unmanned Air
Vehicle Reliability Study. Feb 2003.
16
2. Technologies to Mitigate Risk
 How much can a given
technology reduce risk per
flight hour below the status
quo?
http://www.competitionaircraft.com/ultrali
ght-aircraft-parts.html
 Transponders, see and avoid
capacity, ADS-B, AN hardware,
flight-real operator consoles.
http://www.americanchampionsales.com/Assets/I
mages/ACA%20options/MVC-834F_JPG.jpg
17
2. Technologies to Mitigate Risk
700
Hunter
600
Pioneer
Class A or B Mishaps per 100,000 Hours
Global
Hawk
500
400
Shadow
F-16
300
200
100
Predator
I-Gnat
0
100
from Weatherington 2006
1,000
 Identify main causes
of failure gap between
UAS and manned
aircraft (e.g., human
factors, equipment
failure)
 Assess potential of
given technology to
reduce particular
failure modes
U-2
10,000
100,000
1,000,000
Cumulative Flight Hours
18
3. Tracking UAS safety incidents
 Nuclear power industry vastly
improved safety culture after TMI by
sharing and analyzing safety
information
 For the UAS industry, what kinds of
information collection, sharing, and
analysis is feasible?
 Accessible to FAA? to the public?
 Voluntary or regulatory mandate?
19
Intended Deliverables
 Analysis of how different UA flight
variables (e.g. numbers, size, failure
rates, population flown over) can be
varied to achieve the ELOS goal
 Description of the latest UAS
technologies to mitigate risk, and
estimates of their effectiveness
 A discussion of alternative safety
incident reporting systems
20
Review Panel Input
 How significant is catastrophe rate as the ultimate
driver of aviation safety performance?
 Which safety technologies are most promising for
large UAs? Small UAs?
 How are accident and near-accident data currently
collected for UASs?
21
Public Awareness and
Perception
Team Members
Darian Ghorbi
Jenny Kim
Mark Peterson
Laura Seitz
Patrick Snyder
Manager
Pete Tengtrakul
22
Statement of Purpose
 Add the element of public perception to the
UAS-NAS discussions
 Motivation: the fact that there has never been
a formal presentation of public perception on
the topic
 Findings: useful for the creation of regulations
23
Objectives
 Compare public perceptions of the risks
of manned and unmanned aircraft
 Find demographic groups with certain
risk/benefit patterns
 Research implications of opinions
regarding UASs
 Create and administer survey to aid in
completing objectives
24
Hypotheses







Autonomous vs. Remotely Piloted
- Autonomous perceived as more risky
Ground vs. Air
- UASs perceived as more risky in air
Prior Knowledge vs. Risk Perception
- Prior knowledge, associate less risk
Benefit vs. Risk Perception
- Higher benefit, lower risk
Education vs. Risk Perception
- Technical education, associate less risk
Age vs. Risk Perception
- Older demographics more cautious
Frequency of Flight
- Arguments for both sides
25
Layout of Survey
 First Page
 Provide information about UASs
 Autonomous
 Remotely Operated
 Gauge previous knowledge
 Source
26
Layout of Survey
Application
•Traffic Monitoring
•Pipeline Monitoring
•Disaster Relief
•Border Patrol
Questions
•
Quick Response
•
Benefit
•Agriculture Monitoring
•
Picture of UAS
application
•
Description of UAS
•Physical Information
•
Stakeholder
•
Public
Risk
•
Ground
•
Air
7-Point Scale
•
1 - Much Less
•
4 - Same
•
7 - Much More
•Current Application
27
Layout of Survey
 Last Page
 Demographics
Gender
Age
Education
Frequency of Flight
Voting (identify opinions of those that are
politically engaged)
 Pilot (may have drastically different
opinions)





28
Executing Surveys
 Method
 In person
 Potential Locations
 Pittsburgh Shopping Malls
 Incentives
 Raffle, Carnegie Mellon trinkets, $2 Bills
 Quantity Desired
 ~300 surveys needed for adequate
statistical power to determine the
different risks and benefits
29
Sample Results
Perceived Risk (1=Low, 7=High)
"Perceived Risk of UASs used in Border Patrol application, by age demographic”
7
6
5
4
3
2
1
0
18-25
26-35
36-45
46-60
61+
Age (years)
**Not real data, created to illustrate analysis
30
Sample Results
"Perceived Risks between Remotely Operated and Autonomous UAS applications"
Perceived Risk (1=Low, 7=High)
7
6
5
Traffic
4
Pipeline
Border
Agriculture
3
Disaster
2
1
Remotely operated UASs
Autonomous UASs
0
UASs Applications
**Not real data, created to illustrate analysis
31
Review Panel Input
 Which relationships are of most interest?
 Any attributes of subjects or UASs overlooked?
 How will this information be useful in making
decisions?
 Methods of assessment other than survey?
32
Economic Perspective
Team Members
Reiko Baugham
Brad Brown
Feng Deng
Nathan Diorio-Toth
Victoria Morton
Manager
Ryan Kurlinski
33
Outline







Purpose Statement
Goals
UAS Applications
Preliminary Cost Model
Problems/Issues
The Next Step
Questions
34
Purpose Statement
 Assessing the market viability of
specific UAS applications using an
analysis of relative cost effectiveness
 Analyzing how different regulatory
measures could affect costs and
therefore the viability of a UAS
market for a particular application
35
Goals
 Develop UAS cost model
 Capital vs. Operating costs
 Varying UAS sizes
 Compare with both ground based and aerial
alternatives
 Determine appropriate measure of cost
effectiveness
 Apply cost model to chosen applications to
assess market viability
 Identify and examine areas of cost model
most sensitive to change
 Different regulatory measures
 Improvements to technology
36
UAS Applications






Animal tracking
Border Patrol
Local surveillance
Pipeline monitoring
Traffic monitoring
Weather
reconnaissance
Source: www.cpb.gov
37
Preliminary Cost Model
Adapted from a presentation by Ronald Schaefer
38
Preliminary Cost Model
 Equipment Manufacture




Basic aircraft (no additional equipment)
Ground control station
Support vehicle
Additional communication hardware/satcom rental
 System Operation and Maintenance
 Manpower
 Training
 Salary
 Mission Related Expenses
 Insurance
 Task specific equipment (special sensors)
 Fuel
 Miscellaneous/Other
39
Problems/Issues
 Cost data is hard to get
 Many manufacturers are unwilling to
disclose specific cost information




Government contract confidentiality
Market confidentiality
Nefarious uses
Not a potential client
 Developing detailed cost model
 Is anything missing?
40
The Next Step
 Add more detail to the cost model
 Make more application specific
 Obtain more cost data for UAS and
their alternatives for each application
 Determine effectiveness measures for
each application
 Determine if cost effectiveness is best
measure to use or if there is a better
alternative
41
Review Panel Input
 Where else do we look for cost data?
 Specific people, organizations, publications?
 Is our cost model complete?
 Any broad categories missing?
 Any important sub-categories missing?
 Are there alternative measures to cost
effectiveness?
 Allow to better or more easily assess market
viability?
 What are good effectiveness measures for our
chosen applications?
42
Governance
Team Members
Nora Darcher
Norma Espinosa
Scott Fortune
Andrea Fuller
Managers
Keith Florig
Leonardo Reyes-Gonzalez
43
Purpose and Goal
 Group purpose is to analyze the
current system of governance
regarding UAS integration and
ultimately make recommendations to
improve the system considering the
principles of good governance
 Goal is to advance discussion on
whether and how to integrate into the
NAS
44
Principles of Good Governance
 8 Characteristics of Good Governance
Source: http://www.unescap.org/pdd/prs/ProjectActivities/Ongoing/gg/governance.asp
45
Stakeholders
 Identify:
 stakeholders as well as their roles and
objectives
 alliances and competitors
 channels and conventions of interaction
 Goal is to gain an understanding of
how the governance system for the
airspace functions
46
Stakeholders
Descriptive model of stakeholder interactions regarding UAV access to the NAS
ATC
technology
FAA non-UAS
regs
NTSB
UAS
standards
organizations
FAA UAS
regs
Media
attention
Non-UAV
traffic
density
Airlines
FAA budget
UAS
technology
frontier
UAV
accident
history
Congress
(oversight &
budget)
Air safety
NGOs
Pilots
Non-Defense
Govt Users
Civilian
users
UAS vendors
UAS lobby
DoD/DHS
47
Roadblock Analysis
 Analyze system to determine
roadblocks to UAS integration
 Organize roadblocks:
 First categorize list:
 Technological – Sense and avoid
 Organizational – Certification standards
 Infrastructural – Air traffic control
technology
 Public Concern – Privacy infringement and
safety
48
Roadblock Analysis (cont’d)
 Next prioritize list
 Most important to solve
 Easiest to solve
 Perform historical analysis
 Introduction of ultralights
 Exempt from many aircraft standards
 No license or training required for pilots
49
Expected Results
 Make recommendations to address
issues on the list by improving
“governance system”
 Possible recommendations include:
 Funding initiatives
 Broader stakeholder participation
 Database development and information sharing
 Increased staffing
 Need for outside help or advice
 Providing technical assistance
 Specific policy initiatives
 Creation of NTSB-type agency for UAS crashes
50
Review Panel Input
 Are there any up and coming initiatives in UAS
governance?
 Which dimensions of governance are most in
need of improvement (e.g., transparency,
availability of channels for stakeholder
deliberation)?
51
Discussion Session
52
Technology & Risk Questions
 How significant is catastrophe rate as the ultimate
driver of aviation safety performance?
 Which safety technologies are most promising for
large UAs? Small UAs?
 How are accident and near-accident data currently
collected for UASs?
53
Public Perception Questions
 Which relationships are of most interest?
 Any attributes of subjects or UASs overlooked?
 How will this information be useful in making decisions?
 Methods of assessment other than survey?
54
Economic Questions
 Where else do we look for cost data?

Specific people, organizations, publications?
 Is our cost model complete?


Any broad categories missing?
Any important sub-categories missing?
 Are there alternative measures to cost effectiveness?

Allow to better or more easily assess market viability?
 What are good effectiveness measures for our chosen
applications?
55
Governance Questions
 Are there any up and coming initiatives in UAS
governance?
 Which dimensions of governance are most in
need of improvement (e.g., transparency,
availability of channels for stakeholder
deliberation)?
56
General Questions?
57
Thank You For Coming!
58