Enterprise Modeling
Improving the Analysis of Alternatives (AoA) Process
for IT Solutions Government Contract Proposals
Saad El Beleidy
Peyman Jamshidi
Jared Kovacs
Gabriel Lewis
Presented at:
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
2
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
3
IT Solutions Government
Contractors
Project Sponsor:
AoA: develop complex IT
solution, crucial to proposal
15-25 Proposals per year, of
varied sizes
Federal Contract Spending
decreased $40 billion from
2009 to 2011
0.8
0.4
0
0
2
4
6
8
10
AoA Duration (Weeks)
Federal Contract
Spending (Billions)
Proposal Development is
critical to gaining revenue
Likelihood
Civilian and National Security (CNS) Division, Vangent, Inc.
$560
$530
$500
2009
2010
2011
Fiscal Year
Adapted from the
Federal Times, 2011
4
Process Level 1
Proposal Development Process
Government
Entity
Requirements
Selected Solution
Acquisitions
Committee
Solicitation
Proposals
Government Contractor
Technical
Solution
Development
Technical
Solution
Development
/ AoA
Bid
Decision
Proposal Writing
Budget & Management
Highly competitive: 5-10 competitors for each proposal
5
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Mean Duration
(% AoA)
Ranked List of
Alternatives
60%
20%
10%
30%
40%
40%
20%
0%
Define the
Problem Domain
Define Evaluation Explore Alternate Evaluate Solutions
Criteria
Solutions
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
6
Task Category Classification
1. Labor Intensive:
Little expertise required: parsing, documenting.
Time Variability = Low (±22%)
2. Decision Making:
Expert required: difficult decisions, judgment calls.
Time Variability = High (±54%)
3. Experience Recall:
Expert required: subjectively making judgments based on
previous experience. Time Variability = Medium (± 44%)
4. Networking:
Personal dialogue and collaboration with co-workers and others.
Time Variability = High (±100%)
7
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
8
Process Level 3
Phase 1: Define Problem Domain
Task
Category
% of
Phase
Labor
Intensive
25%
Decision
Making
23%
Experience
Recall
21%
Networking
31%
Data acquired via knowledge
elicitation from stakeholder SMEs
9
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
10
Process Level 3
Phase 2: Define Evaluation Criteria
Task
Category
% of
Phase
Labor
Intensive
42%
Decision
Making
20%
Experience
Recall
18%
Networking
20%
Data acquired via knowledge
elicitation from stakeholder SMEs
11
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
12
Process Level 3
Phase 3: Explore Alternate Solutions
Task
Category
% of
Phase
Labor
Intensive
36%
Decision
Making
15%
Experience
Recall
13%
Networking
36%
Data acquired via knowledge
elicitation from stakeholder SMEs
13
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
14
Process Level 3
Phase 4: Evaluate Solutions
Task
Category
% of
Phase
Labor
Intensive
38%
Decision
Making
21%
Experience
Recall
23%
Networking
18%
Data acquired via knowledge
elicitation from stakeholder SMEs
15
Current Issues in AoA
1. Limited and variable availability of
past research due to proprietary
restrictions
Probability
0.5
0.25
0
0%
50%
100%
2. Limited and variable applicability
of past research once gained
Probability
Relative Expected Availability
0.5
0.25
0
0%
3. Variable difficulty of AoA—need for
past research
50%
100%
Relative Expected Applicability
16
6. Time spent for AoA results is an
entirely overhead cost
100%
50%
0%
0%
50%
100%
Relative Expected Availability
100%
Probability
5. One employee handles entire
AoA, so potentially parallel
processes are conducted in
series
AoA Output
Relative
Quality
4. Quality of AoA suffers from the
information’s lack of availability
and applicability
AoA Output
Relative
Quality
Current Issues in AoA
50%
0%
0%
50%
100%
Relative Expected Applicability
0.8
$16,000
0.4
$6000
$32,000
0
0
2
4
6
8
10
AoA Duration (Weeks)
17
Key Stakeholder Goals
Solutions Architects
Perform Analysis of Alternatives
– Maximize productivity
– Avoid overtime
– Maximize labor rewards
Capture Managers
Manage the transition from opportunity discovery to contract award.
Oversees bid strategies, pricing, and teaming
– Maximize probability of winning contracts
• Increase AoA discriminability and quality
Proposal Managers
Develop and manage the proposal plan and schedule
– Maximize proposal/AoA throughput
18
Key Stakeholder Interactions
Interactions
Proposal Demand
Solution Demand
Tension:
Managers and
Solutions Architects
Limited time and
personnel resources
to conduct AoAs.
19
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
20
Problem Statement
During a time of national economic downturn,
federal contract spending cuts have led to a
decrease in available contract revenue and an
increase in competition between government
contractors.
These factors have increased the time sensitivity
of proposal development, specifically in the AoA
process.
21
Need Statement
There is a need for Analysis of Alternatives
process improvements to reduce the mean time
duration by at least 33%, and the variability
by 25%, while maintaining or increasing AoA
proposed solution quality and keeping
maximum costs below $100,000 per AoA.
Need validated with key stakeholders
22
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
23
Design Alternative Approach
AoA Phase
Parallel
Tasks
Task
Category
% AoA
Variability
Define Problem
Domain
6
Labor
Intensive
35.2%
Low ±22%
Define Evaluation
Criteria
4
Decision
Making
19.2%
High ± 54%
Explore Alternate
Solutions
5
Experience
Recall
26.3%
Med ± 44%
Evaluate
Solutions
4
Networking
19.2%
High ±100%
Optimize AoA Staffing Levels
Information Management System
– Target Parallel Tasks
– Reduce mean time duration
– Target efficiency increases
– Reduce duration and variability
24
Optimize AoA Staffing Levels
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
---
-10%
---
+10%
1 Additional Solutions Architect collaborates to conduct AoA
Reduce mean time duration of AoA
– Additional resource to conduct parallel tasks
– Increases size of social network
– Potential for conflict in making decisions
Cost: Approx. $200,000 salary including benefits per year
Five year cost: $1,000,000
25
Information Management
Alternatives
Implementing a File Management System
– Database searching capability improves availability of information in the AoA
– Increases efficiencies of task categories, improves re-use of past research
Implementing a Content Management System
– Enhanced capabilities greatly improves availability of information in the AoA
– Increases efficiencies of task categories, improves re-use of past research
Maintaining a Sanitized Repository
– Adds new content to the information pool
– Changes high-variability task categories to low-variability categories
26
File Management System
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+10%
+10%
+15%
+5%
Benefits
–
–
–
–
Integrates with current system
Organized file structure
Promotes collaboration
Easily scalable
Drawbacks
– Requires permissions for file access
– Limited search functionality
Cost: Intravation
Initial Cost:
100 User License
GDIT(Parent Company) has 25 user license;
$1000 one time fee per active user
First Year Cost: $77,000
Annual Maintenance: $1600
Five Year Cost: $83,000
27
Content Management System
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+15%
+15%
+20%
+10%
Benefits
– Robust searching and indexing
– Preconfigured user roles based upon
content–access needs
– Authentication, check in/out, tracking
– Workflow management
Drawbacks
– High complexity
• High learning curve
• Expensive Technical Support
Cost: Documentum
Initial Cost:
100 User License Cost: $110,665
System Cost:
First Year Cost: $129,000
Annual Maintenance: $19,000
Five Year Cost: $203,000
28
Sanitized Document Repository
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+15%
+10%
+10%
+5%
Low
High
Med
High
35%  40%
20%  17%
19%  17%
26%
Variability
% of AoA
Original New
Benefits
– Quality of information improvement
• Promotes availability
• Increases applicable content
– Virtually eliminates security risks
– Provides quicker access to data
– Minimal technical support
Drawbacks
– Low initial benefit
Cost
Estimate 12 labor hours per AoA
Five Year Cost: $150,000
29
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
30
Simulation Design
Monte Carlo Discrete Event Simulation of the AoA
1000 Replications
Each Replication covers 25 Proposals (1 year)
Key Model Assumptions:
1.
2.
3.
4.
Solutions Architects work on one task at a time
Only one proposal is being worked on at any given time
The four task categories adequately capture AoA time consumption
All tasks are of equal importance to the quality of AoA output
31
Simulation Design
AoA Process
Definition
Task Category
Efficiencies
AoA Mean
Duration
AoA Difficulty
Simulation Model
Information
Availability
Information
Applicability
AoA Duration
Variability
Quality
Metric
Number of
Technologies
32
Simulation Video
Task Time Delay Equation

𝐷𝑒𝑙𝑎𝑦 =
𝑇𝐶
𝐷 ∗ 𝐸 ∗ 𝑊 ∗ 𝑉 ∗ 𝑇𝑉
[
]
𝑁
: Task Categories (Labor Intensive, Decision Making, Experience Recall,
Networking)
: Inherent Task Delay
: Task Category Efficiency Index (% Expected Time)
: Task Category Weight (% of task with specific task category)
: Task Category Variability Factor (RV)
: Technology Variability (RV)
: Number of Technologies/AoAs in the Proposal
34
Task Quality Metric Equation
𝑄 = 0.6 ∗ 𝐴𝑣 + 0.4 ∗ 𝐴𝑝
𝐴𝑣
: Availability Factor (RV)
𝐴𝑝
: Applicability Factor (RV)
0.6,0.4
: Stakeholder weights
35
Design of Experiment Matrix
Run
Configuration
1
2
3
4
5
6
7
8
9
10
11
12
Baseline
A1
A1, A2
A1, A2, A3
A1, A2, A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
A3
A4
A1
x
x
x
x
x
x
-
Alternatives
A2
A3
x
x
x
x
x
x
x
x
x
x
-
A4
x
x
x
x
A1: Staffing Levels
A2: Sanitized
Repository
A3: File
Management
A4: Content
Management
36
Contents
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Results and Conclusions
37
Simulation Results
Percent Decrease in AoA Mean
Time Duration
Percent Decrease in Mean Duration
60.00%
49.47%
50.00%
51.66%
43.08%
43.91%
40.00%
45.42%
36.53%
33.00%
30.00%
20.10%
20.00%
23.38%
13.97%
11.15%
10.11%
10.00%
0.00%
A1
A1, A2
A1, A2, A1, A2,
A3
A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
A3
A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
38
Simulation Results
Percent Decrease in AoA Time
Duration Variability
Percent Decrease in Duration Variability
60.00%
50.00%
50.00%
50.00%
37.50%
37.50%
40.00%
37.50%
34.38%
30.00%
25.00%
18.75%
20.00%
12.50%
10.00%
6.25%
6.25%
6.25%
A3
A4
0.00%
A1
A1, A2
A1, A2, A1, A2,
A3
A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
39
Simulation Results
Quality Metric
0.84
Using a Sanitized Repository
is
the
only
proposed
alternative that directly
affects quality.
Quality Metric
0.82
0.8
0.78
0.74
Baseline
0.76
~10%
0.74
0.72
Increase in Quality
0.7
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
40
Stakeholders’ Utility Function
Utility
Mean
Duration
Duration
Variability
AoA Output
Quality
Solution
Quality
.238
.048
.143
.571
Value Hierarchy obtained via Stakeholder values elicitation
weighted with the Swing Weight Method
41
Alternative Utility Ranking
Rank
Alternative
Configuration
Utility
1
2
3
4
5
6
7
8
9
10
11
12
A1, A2, A4
A1, A2
A1, A2, A3
A2, A4
A2
A1, A4
A2, A3
A4
A1, A3
A1
Baseline
A3
4.25
3.95
3.90
3.90
3.54
3.49
3.28
3.18
3.14
3.09
2.50
2.37
A1: Optimized Staffing Levels
A2: Sanitized Repository
A3: File Management
A4: Content Management
Meet The Need
Don’t Meet The Need
42
Sensitivity Analysis
Percent Change in Criteria Weight
Necessary to Change Utility Rank
Alternative
Configurations
A1, A2
A1, A2, A3
A2, A4
Utility Function Criteria
Mean Time
+33%
+34%
+60%
Usability
+41%
+48%
+34%
Integrability
+69%
+119%
+60%
43
Cost-Benefit Analysis
5
4.5
A1, A2, A4
4
Utility
3.5
A1, A2
A2, A4
A2
A2, A3
A4
3
A1
A1, A2, A3
A1, A4
A1, A3
Baseline
2.5
A3
2
Meet the
Duration and
Variability
Reduction Need
Do Not Meet the
Duration and
Variability
Reduction Need
1.5
1
-$5,000.00
$15,000.00
$35,000.00
$55,000.00
$75,000.00
$95,000.00
Cost per AoA
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
44
Cost-Benefit Analysis
5
4.5
Closest to Desirable Region
4
Utility
3.5
A1, A2
A2, A4
A2
A2, A3
A4
3
A1
A1, A2, A4
A1, A2, A3
A1, A4
A1, A3
Baseline
2.5
A3
2
Meet the
Duration and
Variability
Reduction Need
Do Not Meet the
Duration and
Variability
Reduction Need
1.5
1
-$5,000.00
$15,000.00
$35,000.00
$55,000.00
$75,000.00
$95,000.00
Cost per AoA
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
45
Recommendations
Optimize Staffing Levels and Maintain a
Sanitized Repository (A1, A2)
–
–
–
–
–
–
Percent Reduction in Time Duration: 44%
Percent Reduction in Duration Variability: 38%
Percent Increase in Quality: 10%
Total Utility: 3.95
Max Expected Cost per AoA: $50,000
Total Implementation Cost: $230,000/year
46
Recommendations
(Potential Value, if Cost Reduced)
Optimize Staffing Levels, Maintain a
Sanitized Repository, and Implement a
Content Management System (A1, A2, A4)
–
–
–
–
–
–
Percent Reduction in Time Duration: 52%
Percent Reduction in Duration Variability: 50.00%
Percent Increase in Quality: 10%
Total Utility: 4.25
Maximum Expected Cost per AoA: $78,000
Total Implementation Cost: $111,000 plus $249,000/yr
47
Questions?
48
Backup Slides
49
Project Plan
WBS – Top Level
Enterprise
Modeling
Project
1.0 Project
Definition
2.0 Requirements
Development
5.0 Results
Analysis
3.0 Solution
Development
4.0 Modeling
and Testing
6.0 Communications
and Management
Syst 495-Spring 2012
50
Context
Output of AoA: Ranked Set of Alternatives
AoA output for “Case
Management Platforms”
(Vangent, Inc)
Vendors Considered:
Custom Developed
Siebel
IBM
Microsoft
Total Scores
Syst 490-Fall 2011
51
Stakeholders’ Value Hierarchy
AoA Process
Improvement
Objective
Maximize
Usability
Maximize
Integrability
Maximize
Tailorability
Maximize Tech
Support
.190
.167
.119
.071
Reduce Mean
Time
Reduce Time
Variability
Maximize AoA
Quality
Maximize
Scalability
.048
.143
.024
.238
Value Hierarchy obtained via Stakeholder values elicitation
weighted with the Swing Weight Method
Syst 495-Spring 2012
52
Information Management
Alternatives
Define the
Problem Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Retrieve Past Criteria from
similar proposals
Retrieve Past Proposal
Requirements
Evaluate
Solutions
Provide reference of past
analyses
Maintain List of SME’s
Retrieve Past Alternative Material
Syst 495-Spring 2012
53
Optimizing Staffing Levels
Alternative
Define the
Problem Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Time Reduction
Parallel Processes
Syst 495-Spring 2012
54
Percent Decrease
60.00%
50.00%
40.00%
33%
30.00%
25%
20.00%
10.00%
0.00%
Mean Time (Filled)
Variability (Hollow)
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
55
Alternatives Efficiency Summary
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Baseline
---
---
---
---
Optimizing
Staffing
Levels
---
-10%
---
+10%
Sanitized
Repository
+15%
+10%
+10%
+5%
File
Management
+10%
+10%
+15%
+5%
Content
Management
+15%
+15%
+20%
+10%
Alternative
Syst 495-Spring 2012
56
Alternatives’ Efficiency Indexes
Labor Intensive
Decision Making
Experience Recall
Networking
1.2
Efficiency Index
1.1
1
0.9
0.8
0.7
0.6
Baseline
A1
A1, A2 A1, A2, A1, A2, A1, A3 A1, A4
A3
A4
A2
A2, A3 A2, A4
A3
A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
57
Process Level 1
Proposal Development Process
Government
Entity
Requirements
Selected Solution
Acquisitions
Committee
Government Contractor
Solicitation
Proposal
TechnicalSolution
Solution Development
Technical
Development
Bid
Decision
Proposal Writing
Budget & Management
Proposal Development is critical to gaining revenue
Highly competitive: 5-10 competitors for each proposal
Syst 495-Spring 2012
58
Process Level 2
Technical Solution Development
Analysis of
Alternatives
(AoA)
Solicitation
Alternatives
Analysis
Ranked List of
Alternatives
Integration
of Assets
Assets for
Proposal
Solution to
be Proposed
Model developed based on team research and knowledge
elicitation from Vangent sponsors
Syst 495-Spring 2012
59
Process Level 3
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
60