National Aeronautics and Space Administration
Usage of Decision Analysis Methods
Outside of a Classroom Environment
by Aerospace Researchers
Sharon Monica Jones
NASA Langley
Rafael E. Landaeta, C. Ariel Pinto and Resit Unal
Old Dominion University
James T. Luxhøj
Rutgers University
Hampton Roads Area INCOSE Conference on Decision Analysis and Its
Applications to Systems Engineering, Newport News, VA (November 17-18, 2009)
www.nasa.gov
Outline
•
Background/Definitions
•
Data Collection Process
•
Results
•
Concluding Remarks
National Aeronautics and Space Administration
Jones,, et al. (2009)
2
Problem Definition
•
Aerospace technology managers need:
 To predict future technologies
 To assess progress toward R&D goals
•
Aerospace technology portfolio decisions are difficult because:
 Very little time to acquire background data
 Uncertainty factors (e.g., politics, global economy, environment, funding)
•
Prescriptive decision analysis methods
 Have been used for financial portfolio assessment
 Value for policy related decisions has been questioned
National Aeronautics and Space Administration
Jones, et al (2009)
3
Ralph Keeney’s Suggestions for Making
Better Decision Makers*
•
Develop concepts, tools and procedures to help decision makers
“My experience is that many people, including well-educated people, have a very difficult
time in structuring their decisions. They can get mixed up about the difference between
fundamental concepts such as alternatives and objectives.”
•
Use real decisions, not just laboratory problems in decision research
“We have learned a great deal from all the laboratory settings where decision experiments
have been conducted. There have also been some research studies of real decision
problems. I feel there is much more to be gained by having more of this type of research.”
•
Teach people what they can and will learn and use
“…hundreds and thousands of people have had at least a course that included a substantial
part on decision analysis and very few have probably ever conducted a formal decision
analysis. Once we find out what people can and will learn and use, that should constitute
the basis for much of our teaching of decision analysis.”
*Source: Ralph L. Keeney, “Making Better Decision Makers”, Decision Analysis, 1:4 (2004)
National Aeronautics and Space Administration
Jones,, et al. (2009)
4
Decision Analysis Usage in Aerospace
Portfolio Development
•
Aerospace managers have investigated the use of decision analysis methods for portfolio
investment decisions:
 Commercial Aviation Safety Team (CAST)
 NASA Aviation Safety Program
 Future Aviation Safety Team (FAST)
•
These technology assessments involved:
 Resource commitments (e.g., employee time, travel money, software acquisition, training)
 Assumption that decision analysis methods would improve aerospace technology
assessment process
National Aeronautics and Space Administration
Jones,, et al. (2009)
5
Technology Assessment
“ A process for measuring the impact of established or new technologies”
*Hans Mohr, “Technology Assessment in Theory and Practice”, Society for Philosophy and Technology, 4:4 (Summer 1999)
National Aeronautics and Space Administration
Jones, et al (2009)
6
Aerospace Technology Assessment
Three different processes for examining impact of a set of technologies
Technology assessment
Technology forecasting
Aerospace Technology
Assessment
Technology foresight
National Aeronautics and Space Administration
Jones, et al (2009)
7
Conceptual Model *
1. Identify Context &
Understand Objectives
2. Identify Alternatives
3. Decompose & Model the
Problem
a. Model of Problem Structure
b. Model of Uncertainty
c. Model of Preferences
Decision
Analysis
Methods
Aerospace Technology Assessment
4. Choose the Best
Alternative
5. Sensitivity
Analysis
6. Implement the
Chosen Alternative
*Adapted from Robert T. Clemen, Making Hard Decisions, 2nd Edition (1995)
National Aeronautics and Space Administration
Jones, et al (2009)
8
Purpose of Study
What is known:
What is unknown:
Decision analysis methods
in financial portfolio
assessment
Decision analysis methods
for policy related decisions
Decision experiments in
laboratory settings
Decision analysis methods
in real decision problems
Technology assessment in
medical R&D
Technology assessment in
aerospace
National Aeronautics and Space Administration
Jones, et al (2009)
9
Study Overview
•
Population was aerospace researchers with experience in one or more of the following:
 Aerospace program/project management
 Aerospace technology assessment
 Aerospace technology selection
 Aerospace R&D portfolio development
•
Methods that were investigated in the study
 Decision trees
 Influence diagrams
 Criteria aggregation methods
 Explicit tradeoff approaches
•
Participants were questioned about their usage of these methods for aerospace technology
assessment
National Aeronautics and Space Administration
Jones, et al (2009)
10
Decision Trees
Value of XYZ stock
increases
Buy shares of XYZ
stock
Value of XYZ stock
unchanged
Earn money
Total amount
of money
unchanged
Value of XYZ stock
decreases
Lose money
Don’t buy XYZ
stock
= decision node
Total amount
of money
unchanged
= chance node
= consequence node
National Aeronautics and Space Administration
Jones, et al (2009)
11
Influence Diagrams
chance node
Price of XYZ
stock at
future point
Buy XYZ stock?
decision node
National Aeronautics and Space Administration
Change in amount
of money
consequence node
Jones, et al (2009)
12
Criteria Aggregation Methods
• Methods in which two sets of aggregated indices are developed and used to evaluate the
alternatives in the decision problem
• Methods in the category include:
 Analytical Hierarchy Process (AHP)
 Weighted Sum Model (WSM)
n
A*WSM-score =
max
i

aij,wj, for 1=1,2,3,….n
j=1
where,
A*WSM-score
n
aij
wj
=
=
=
=
the WSM of the best alternative
the total number of criteria
the score of the i-th alternative in terms of the j-th criterion
the weight of importance of the j-th criterion
Example of Simple Weighted Sum Model *
*E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study(1995)
National Aeronautics and Space Administration
Jones, et al (2009)
13
Explicit Tradeoff Approaches
• Decision analysis methods that are based on value functions
• Methods in this category include:
 Multi-Attribute Utility Theory (MAUT)
 Simplified Multi-Attribute Rating Approach (SMART)
National Aeronautics and Space Administration
Jones, et al (2009)
14
Excluded Decision Analysis Methods
•
Avoided decision analysis methods that are not popular in U.S.
•
Real world applications are complex with large amounts of uncertainty
•
Specific decision analysis methods that were excluded from study:
 Outranking methods (e.g., ELECTRE, TOPSIS)
 Optimization methods
 Analytic network process (ANP)
National Aeronautics and Space Administration
Jones, et al (2009)
15
Data Collection and Analysis Process
Refine List of
Candidate
Survey
Participants
Develop List
of Pilot
Participants
Conduct Pilot
Survey
Review &
Analyze Pilot
Results
Modify Survey
Instrument
Develop WebBased Survey
Instrument
Conduct
Survey
Analyze
Results
Legend
 Survey Development
 Data Collection
 Data Analysis
National Aeronautics and Space Administration
Jones, et al (2009)
16
Web-Based Instrument Development
•
Several web-based services examined
•
Questions developed based on several sources:
 Short surveys at professional meetings
 Validated research in decision analysis literature
•
Identities of survey participants remained anonymous
National Aeronautics and Space Administration
Jones, et al. (2009)
17
Pilot Survey
•
Conducted with subset of population (10 persons)
•
Think aloud cognitive interviewing techniques used
 Solicitation of all thoughts and comments
 Manual recording of information during completion of online survey
 De-identification of subjects in final documentation
National Aeronautics and Space Administration
Jones, et al. (2009)
18
Survey Instrument Modification
•
Questions were modified, added or eliminated from the survey based on input from:
 Pilot survey comments
 Data analysis of pilot survey data
 Additional comments from other reviews (e.g., ODU IRB)
•
Number of survey questions reduced from 70 to 65
National Aeronautics and Space Administration
Jones, et al. (2009)
19
Data Collection Overview
•
Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review
Boards (IRB’s)
•
E-mail invitation was sent to 260 persons
•
154 total visits to survey website
 16 partial responses
 99 completes surveys
•
Out of the 99 completed surveys
 76% male, 24% female
 Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates
 72% employed as government civil servants
National Aeronautics and Space Administration
Jones, et al. (2009)
20
Job Functions
National Aeronautics and Space Administration
Jones, et al. (2009)
21
Aerospace Experience
National Aeronautics and Space Administration
Jones, et al. (2009)
22
Decision Trees
National Aeronautics and Space Administration
Jones, et al. (2009)
23
Influence Diagrams
National Aeronautics and Space Administration
Jones, et al. (2009)
24
Criteria Aggregation Methods
National Aeronautics and Space Administration
Jones, et al. (2009)
25
Explicit Tradeoff Approaches
National Aeronautics and Space Administration
Jones, et al. (2009)
26
Usage of Decision Analysis Methods
Outside of a Classroom Environment
National Aeronautics and Space Administration
Jones, et al. (2009)
27
Categories of Non-ATA Usage of DA
Outside of a Classroom Environment
National Aeronautics and Space Administration
Jones, et al (2009)
28
Additional Questions
National Aeronautics and Space Administration
Jones, et al. (2009)
29
Additional Questions (cont’d)
National Aeronautics and Space Administration
Jones, et al. (2009)
30
Additional Questions (cont’d)
National Aeronautics and Space Administration
Jones, et al. (2009)
31
Additional Questions (cont’d)
National Aeronautics and Space Administration
Jones, et al. (2009)
32
Additional Questions (cont’d)
National Aeronautics and Space Administration
Jones, et al. (2009)
33
Likelihood of Future Usage of DA Methods
National Aeronautics and Space Administration
Jones, et al. (2009)
34
Concluding Remarks
•
This is a subset of the total data
 There are many additional questions in the study
 More formal analysis of the data was conducted using structural equation modeling
techniques to test a set of hypotheses
•
Survey participants believed that the successful use of decision analysis methods depends on:
 Selection criteria in the decision model
 Experience of the person that implements the method
 Reliability of the input data
•
Training/education does not guarantee future use of a decision analysis method
National Aeronautics and Space Administration
Jones, et al. (2009)
35
Questions?
National Aeronautics and Space Administration
36
National Aeronautics and Space Administration
Your Title Here
37