Enterprise Modeling
Improving the Analysis of Alternatives (AoA) Process
for IT Solutions Government Contract Proposals
Saad El Beleidy
Peyman Jamshidi
Jared Kovacs
Gabriel Lewis
Contents
!
!
!
!
!
!
!
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Design of Experiment
Results
Project Plan and Budget
Syst 495-Spring 2012
2
Contents
!   Context and Stakeholder Analysis
!
!
!
!
!
!
Problem and Need
Design Alternatives
Simulation Design
Design of Experiment
Results
Project Plan and Budget
Syst 495-Spring 2012
3
IT Solutions Government
Contractors
Project Sponsor:
!   15-25 Proposals per year, of
varied sizes
!   Complex proposals, large scope
of technology
!   Federal Contract Spending
decreased $40 billion from
2009 to 2011
0.8
0.4
0
0
2
4
6
8
10
AoA Duration (Weeks)
Federal Contract
Spending (Billions)
!   Proposal Development is
critical to gaining revenue
Likelihood
Civilian and National Security (CNS) Division, Vangent, Inc.
$560
$530
$500
2009
2010
Fiscal Year
2011
Adapted from the
Federal Times, 2011
Syst 495-Spring 2012
4
Process Level 1
Proposal Development Process
Government
Entity
Requirements
Selected Solution
Acquisitions
Committee
Solicitation
Proposals
Government Contractor
Technical
Solution
Development
Technical
Solution
Development
/ AoA
Bid
Decision
Proposal Writing
Budget & Management
Highly competitive: 5-10 competitors for each proposal
Syst 495-Spring 2012
5
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Mean Duration
(% AoA)
Ranked List of
Alternatives
60%
20%
10%
30%
40%
40%
20%
0%
Define the Problem Define Evaluation Explore Alternate Evaluate Solutions
Domain
Criteria
Solutions
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
6
Task Category Classification
1. Labor Intensive:
Little expertise required: parsing, documenting.
Time Variability = Low (±22%)
2. Decision Making:
Expert required: difficult decisions, judgment calls.
Time Variability = High (±54%)
3. Experience Recall:
Expert required: subjectively making judgments based on
previous experience. Time Variability = Medium (± 44%)
4. Networking:
Personal dialogue and collaboration with co-workers and others.
Time Variability = High (±100%)
Syst 495-Spring 2012
7
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
8
Process Level 3 (AoA)
Phase 1: Define Problem Domain
Task
Category
% of
Phase
Labor
Intensive
25%
Decision
Making
23%
Experience
Recall
21%
Networking
31%
Data acquired via knowledge
elicitation from stakeholder SMEs
Syst 495-Spring 2012
9
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
10
Process Level 3 (AoA)
Phase 2: Define Evaluation Criteria
Task
Category
% of
Phase
Labor
Intensive
42%
Decision
Making
20%
Experience
Recall
18%
Networking
20%
Data acquired via knowledge
elicitation from stakeholder SMEs
Syst 495-Spring 2012
11
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
12
Process Level 3 (AoA)
Phase 3: Explore Alternate Solutions
Task
Category
% of
Phase
Labor
Intensive
36%
Decision
Making
15%
Experience
Recall
13%
Networking
36%
Data acquired via knowledge
elicitation from stakeholder SMEs
Syst 495-Spring 2012
13
Process Level 2
Top Level AoA Process
Solicitation
Define the
Problem
Domain
Define
Evaluation
Criteria
Explore
Alternate
Solutions
Evaluate
Solutions
Ranked List of
Alternatives
High-level “Decision and Analysis Resolution” (DAR) model
provided by Vangent sponsors
Syst 495-Spring 2012
14
Process Level 3 (AoA)
Phase 4: Evaluate Solutions
Task
Category
% of
Phase
Labor
Intensive
38%
Decision
Making
21%
Experience
Recall
23%
Networking
18%
Data acquired via knowledge
elicitation from stakeholder SMEs
Syst 495-Spring 2012
15
1.  Limited and variable availability of
past research due to proprietary
restrictions
Probability
Current Issues in AoA
0.5
0.25
0
0%
50%
100%
2.  Limited and variable applicability
of past research once gained
Probability
Relative Expected Availability
0.5
0.25
0
0%
3.  Variable difficulty of AoA—need for
past research
Syst 495-Spring 2012
50%
100%
Relative Expected Applicability
16
6.  Time spent for AoA results is an
entirely overhead cost
Syst 495-Spring 2012
Probability
5.  One employee handles entire
AoA, so potentially parallel
processes are conducted in
series
AoA Output
Relative
Quality
4.  Quality of AoA suffers from the
information’s lack of availability
and applicability
AoA Output
Relative
Quality
Current Issues in AoA
100%
50%
0%
0%
50%
100%
Relative Expected Availability
100%
50%
0%
0%
50%
100%
Relative Expected Applicability
0.8
$16,000
0.4
$6000
0
0
2
$32,000
4
6
8
10
AoA Duration (Weeks)
17
Key Stakeholder Goals
!   Solutions Architects
Perform Analysis of Alternatives
–  Maximize productivity
–  Avoid overtime
–  Maximize labor rewards
!   Capture Managers
Manage the transition from opportunity discovery to contract award
Oversees bid strategies, pricing, and teaming
–  Maximize probability of winning contracts
•  Increase AoA discriminability and quality
!   Proposal Managers
Develop and manage the proposal plan and schedule
–  Maximize proposal/AoA throughput
Syst 495-Spring 2012
18
Key Stakeholder Interactions
Interactions
Proposal Demand
Solution Demand
Tension:
Managers and
Architects
Limited time and
personnel resources
to conduct solutions
development.
Syst 495-Spring 2012
19
Contents
!   Context and Stakeholder Analysis
!   Problem and Need
!
!
!
!
!
Design Alternatives
Simulation Design
Design of Experiment
Results
Project Plan and Budget
Syst 495-Spring 2012
20
Problem Statement
During a time of national economic downturn,
federal contract spending cuts have led to a
decrease in available contract revenue and an
increase in competition between government
contractors.
These factors have increased the time sensitivity
of proposal development, specifically in the AoA
process.
Syst 495-Spring 2012
21
Need Statement
There is a need for Analysis of Alternatives
process improvements to reduce the mean time
duration by at least 33%, and the variability
by 25%, while maintaining or increasing AoA
proposed solution quality and keeping
maximum costs below $100,000 per AoA.
Need validated with key stakeholders
Syst 495-Spring 2012
22
Contents
!   Context and Stakeholder Analysis
!   Problem and Need
!   Design Alternatives
!
!
!
!
Simulation Design
Design of Experiment
Results
Project Plan and Budget
Syst 495-Spring 2012
23
Design Alternative Approach
AoA Phase
Parallel
Tasks
Task
Category
% AoA
Variability
Define Problem
Domain
6
Labor
Intensive
35.2%
Low ±22%
Define Evaluation
Criteria
4
Decision
Making
19.2%
High ± 54%
Explore Alternate
Solutions
5
Experience
Recall
26.3%
Med ± 44%
Evaluate Solutions
4
Networking
19.2%
High ±100%
!   Optimize AoA Staffing
Levels
–  Target Parallel Tasks
–  Reduce mean time duration
!   Information Management
System
–  Target efficiency increases
–  Reduce duration and variability
Syst 495-Spring 2012
24
Optimize AoA Staffing Levels
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
---
-10%
---
+10%
1 Additional Solutions Architect collaborates to conduct AoA
!   Reduce mean time duration of AoA
–  Additional resource to conduct parallel tasks
–  Increases size of social network
–  Potential for conflict in making decisions
!   Cost: Approx. $200,000 salary including benefits per year
Five year cost: $1,000,000
Syst 495-Spring 2012
25
Information Management
Alternatives
Division-wide database system to facilitate storage, retrieval and
sharing of technical solution material in AoA
!   Stakeholders currently use Microsoft Sharepoint for file sharing
–  Insufficient level of access to past data, lacking product features
–  Low quality system that is underutilized
!   Alternatives aim to:
–  Reduce mean duration of AoA
•  Enable re-use of past technical solution material
–  Reduce time variability
•  Change high variability task types to low variability task type
–  Maintain or increase quality
•  Increase availability of information
Syst 495-Spring 2012
26
File Management System
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+10%
+10%
+15%
+5%
!   Benefits
–
–
–
–
!   Cost: Intravation
Integrates with current system
Organized file structure
Promotes collaboration
Easily scalable
Initial Cost:
100 User License
GDIT(Parent Company) has 25 user license;
$1000 one time fee per active user
!   Drawbacks
–  Requires permissions for file access
–  Limited search functionality
First Year Cost: $76,615
Annual Maintenance: $1615
Five Year Cost: $83,075
Syst 495-Spring 2012
27
Content Management System
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+15%
+15%
+20%
+10%
!   Benefits
!   Cost: Documentum
–  Robust searching and indexing
–  Preconfigured user roles based upon
content–access needs
–  Authentication, check in/out, tracking
–  Workflow management
!   Drawbacks
–  High complexity
•  High learning curve
•  Expensive Technical Support
Initial Cost:
100 User License Cost: $110,665
System Cost:
First Year Cost: $129,080
Annual Maintenance: $18,415
Five Year Cost: $202,740
Syst 495-Spring 2012
28
Sanitized Document Repository
Task
Category
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Efficiency Gain
+15%
+10%
+10%
+5%
Low
High
Med
High
35%  40%
20%  17%
19%  17%
26%
Variability
% of AoA
Original New
!   Benefits
!   Drawbacks
–  Quality of information improvement
•  Promotes availability
•  Increases applicable content
–  Virtually eliminates security risks
–  Provides quicker access to data
–  Minimal technical support
–  Low initial benefit
!   Cost
Estimate 12 labor hours per AoA
Five Year Cost: $150,000
Syst 495-Spring 2012
29
Contents
!   Context and Stakeholder Analysis
!   Problem and Need
!   Design Alternatives
!   Simulation Design
!   Design of Experiment
!   Results
!   Project Plan and Budget
Syst 495-Spring 2012
30
Simulation Design
!   Monte Carlo Discrete Event Simulation of AoA
!   1000 Replications
!   Each Replication covers 25 Proposals (1 year)
!   Key Model Assumptions:
1.  Solutions Architects work on one task at a time
2.  Only one proposal is being worked on at any given time
3.  The four task categories adequately capture time
consumption in AoA
4.  All tasks are of equal importance to the quality of AoA output
Syst 495-Spring 2012
31
Simulation Design
AoA Process
Definition
Task Category
Efficiencies
AoA Mean
Time
AoA Difficulty
Information
Availability
Simulation Model
AoA Time
Variability
Quality
Metric
Information
Applicability
Number of
Technologies
Syst 495-Spring 2012
32
Task Time Delay Equation
: Task Categories (Labor Intensive, Decision Making, Experience Recall,
Networking)
: Inherent Task Delay
: Task Category Efficiency Index (% Expected Time)
: Task Category Weight (% of task with specific task category)
: Task Category Variability Factor (RV)
: Technology Variability (RV)
Syst 495-Spring 2012
: Number of Technologies/AoAs in the Proposal
33
Task Quality Metric Equation
: Availability Factor (RV)
: Applicability Factor (RV)
: Stakeholder weights
Syst 495-Spring 2012
34
Contents
!
!
!
!
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
!   Design of Experiment
!   Results
!   Project Plan and Budget
Syst 495-Spring 2012
35
Design of Experiment Matrix
Run
Configuration
1
2
3
4
5
6
7
8
9
10
11
12
Baseline
A1
A1, A2
A1, A2, A3
A1, A2, A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
A3
A4
A1
x
x
x
x
x
x
-
Alternatives
A2
A3
x
x
x
x
x
x
x
x
x
x
Syst 495-Spring 2012
A4
x
x
x
x
!   A1: Staffing Levels
!   A2: Sanitized
Repository
!   A3: File
Management
!   A4: Content
Management
36
Alternatives Efficiency Summary
Labor
Intensive
Decision
Making
Experience
Recall
Networking
Baseline
---
---
---
---
Optimizing
Staffing
Levels
---
-10%
---
+10%
Sanitized
Repository
+15%
+10%
+10%
+5%
File
Management
+10%
+10%
+15%
+5%
Content
Management
+15%
+15%
+20%
+10%
Alternative
Syst 495-Spring 2012
37
Alternatives’ Efficiency Indexes
Labor Intensive
Decision Making
Experience Recall
Networking
Efficiency Index
1.2
1.1
1
0.9
0.8
0.7
0.6
Baseline
A1
A1, A2 A1, A2, A1, A2, A1, A3 A1, A4
A3
A4
A2
A2, A3 A2, A4
A3
A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
38
Contents
!
!
!
!
!
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Design of Experiment
!   Results
!   Project Plan and Budget
Syst 495-Spring 2012
39
Simulation Results
Percent Decrease in AoA Mean
Time Duration
Percent Decrease in Mean Duration
60.00%
49.47%
50.00%
51.66%
43.08%
43.91%
40.00%
45.42%
36.53%
33.00%
30.00%
20.10%
20.00%
23.38%
13.97%
11.15%
10.11%
10.00%
0.00%
A1
A1, A2
A1, A2, A1, A2,
A3
A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
A3
A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
40
Simulation Results
Percent Decrease in AoA Time
Duration Variability
Percent Decrease in Duration Variability
60.00%
50.00%
50.00%
50.00%
37.50%
37.50%
40.00%
37.50%
34.38%
30.00%
25.00%
18.75%
20.00%
12.50%
10.00%
6.25%
6.25%
6.25%
A3
A4
0.00%
A1
A1, A2
A1, A2, A1, A2,
A3
A4
A1, A3
A1, A4
A2
A2, A3
A2, A4
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
41
Simulation Results
Quality Metric
0.84
Using a Sanitized Repository
is
the
only
proposed
alternative that directly
affects quality.
Quality Metric
0.82
0.8
0.78
0.74
Baseline
0.76
0.74
10.2%
0.72
Increase in Quality
0.7
Alternative Configurations
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
42
Stakeholders’ Utility Function
Utility
Mean
Duration
Duration
Variability
AoA Output
Quality
Solution
Quality
.238
.048
.143
.571
Value Hierarchy obtained via Stakeholder values elicitation
weighted with the Swing Weight Method
Syst 495-Spring 2012
43
Alternative Utility Ranking
Rank
Alternative
Configuration
Utility
1
2
3
4
5
6
7
8
9
10
11
12
A1, A2, A4
A1, A2
A1, A2, A3
A2, A4
A2
A1, A4
A2, A3
A4
A1, A3
A1
Baseline
A3
4.25
3.95
3.90
3.90
3.54
3.49
3.28
3.18
3.14
3.09
2.50
2.37
A1: Optimized Staffing Levels
A2: Sanitized Repository
A3: File Management
A4: Content Management
Syst 495-Spring 2012
Meet The Need
Don’t Meet The Need
44
Sensitivity Analysis
Percent Change in Criteria Weight
Necessary to Change Utility Rank
Alternative
Configurations
A1, A2
A1, A2, A3
A2, A4
Utility Function Criteria
Mean Time
+33%
+34%
+60%
Usability
+41%
+48%
+34%
Syst 495-Spring 2012
Integrability
+69%
+119%
+60%
45
Cost-Benefit Analysis
5
4.5
A1, A2, A4
4
Utility
3.5
A2
3
A2, A4
A1, A2
A2, A3
A4
A1
A1, A2, A3
A1, A4
A1, A3
Baseline
2.5
Do Not Meet the
Duration and
Variability
Reduction Need
A3
2
Meet the
Duration and
Variability
Reduction Need
1.5
1
0.5
0
-$5,000.00
$15,000.00
$35,000.00
$55,000.00
$75,000.00
$95,000.00
Cost per AoA
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
46
Cost-Benefit Analysis
5
4.5
Closest to Desirable Region
4
Utility
3.5
A2
3
A2, A4
A1, A2
A2, A3
A4
A1
A1, A2, A4
A1, A2, A3
A1, A4
A1, A3
Baseline
2.5
Do Not Meet the
Duration and
Variability
Reduction Need
A3
2
Meet the
Duration and
Variability
Reduction Need
1.5
1
0.5
0
-$5,000.00
$15,000.00
$35,000.00
$55,000.00
$75,000.00
$95,000.00
Cost per AoA
A1: Optimized Staffing Levels; A2: Sanitized Repository;
A3: File Management; A4: Content Management
Syst 495-Spring 2012
47
Recommendations
!   Optimize Staffing Levels and Maintain a
Sanitized Repository (A1, A2)
–  Percent Reduction in Time Duration: 43.91%
–  Percent Reduction in Duration Variability: 37.50%
–  Percent Increase in Quality: 10.21%
–  Total Utility: 3.95
–  Max Expected Cost per AoA: $50,000
–  Total Implementation Cost: $230,000/year
Syst 495-Spring 2012
48
Recommendations
(Potential Value, if Cost Reduced)
!   Optimize Staffing Levels, Maintain a
Sanitized Repository, and Implement a
Content Management System (A1, A2, A4)
–  Percent Reduction in Time Duration: 51.66%
–  Percent Reduction in Duration Variability: 50.00%
–  Percent Increase in Quality: 10.17%
–  Total Utility: 4.25
–  Maximum Expected Cost per AoA: $78,000
–  Total Implementation Cost: $111,000 plus $249,000/yr
Syst 495-Spring 2012
49
Contents
!
!
!
!
!
!
Context and Stakeholder Analysis
Problem and Need
Design Alternatives
Simulation Design
Design of Experiment
Results
!   Project Plan and Budget
Syst 495-Spring 2012
50
Project Plan
!   WBS – Top Level
Enterprise Modeling
Project
1.0 Project
Definition
2.0 Requirements
Development
3.0 Solution
Development
5.0 Results
Analysis
4.0 Modeling
and Testing
6.0 Communications
and Management
Syst 495-Spring 2012
51
Project Budget
$
Total Project Expected Cost
Total Burden per Person
Management Reserve
$130,000.00
$62.40
$8,000.00
Budget Cost Work Performed (BCWP)
$132,500.00
Actual Cost Work Performed (ACWP)
$137,500.00
Budget Cost Work Scheduled (BCWS)
$124,800.00
Cost Performance Index (CPI)
Schedule Performance Index (SPI)
Syst 495-Spring 2012
0.96
0.94
52
Project Budget
BCWP
ACWP
BCWS
$160,000.00
Management Reserve
$140,000.00
$120,000.00
Cost $100,000.00
$80,000.00
$60,000.00
$40,000.00
$20,000.00
$0.00
1
5
9
13
17
21
Time (Week)
Syst 495-Spring 2012
25
29
33
37
53
Questions?
Syst 495-Spring 2012
54
Backup Slides
Syst 495-Spring 2012
55
Appendix A:
!   Output of AoA: Ranked Set of Alternatives
AoA output for “Case
Management Platforms”
(Vangent, Inc)
Vendors Considered:
!   Custom Developed
!   Siebel
!   IBM
!   Microsoft
Total Scores
Syst 490-Fall 2011
56
Appendix B:
Stakeholders’ Value Hierarchy
Stakeholders’ Value
Hierarchy
Maximize
Usability
Maximize
Integrability
Maximize
Tailorability
Maximize Tech
Support
.190
.167
.119
.071
Reduce Mean
Time
Reduce Time
Variability
Maximize AoA
Quality
Maximize
Scalability
.143
.024
.238
.048
Value Hierarchy obtained via Stakeholder values elicitation
weighted with the Swing Weight Method
Syst 495-Spring 2012
57
Appendix C: Simulation Design - 1
Syst 495-Spring 2012
58
Appendix C: Simulation Design - 2
Simulation
Initiation
Syst 495-Spring 2012
59
Appendix C: Simulation Design - 3
AoA Phase 1:
Define the
Problem
Domain
Syst 495-Spring 2012
60
Appendix C: Simulation Design - 4
AoA Phase 2:
Define
Evaluation
Critera
Syst 495-Spring 2012
61
Appendix C: Simulation Design - 5
AoA Phase 3:
Explore
Alternate
Solutions
Syst 495-Spring 2012
62
Appendix C: Simulation Design - 6
AoA Phase 4:
Evaluate
Solutions
Syst 495-Spring 2012
63
Appendix C: Simulation Design - 7
Simulation
Variable
Assignments
Syst 495-Spring 2012
64
Appendix C: Simulation Design - 8
Simulation
Calculations
Syst 495-Spring 2012
65