The Statistical Analysis of the Impact of
Adequate Testing on the Defense
Systems Development
Zaw P. Tun
GWU, EMSE PhD Candidate
Dr. Shahram Sarkani, Ph.D., P.E.
Dr. Thomas Mazzuchi, D.Sc.
April 13, 2015
Overview








Purpose
Background
Focus Area
Data Extraction
Source of Data
Proposed Methodology
Conclusion
Future Work
Introduction
 Purpose
• Analyze the cost growth trend to determine if adequate testing plays an
essential role in successful development of new systems in the DOD
acquisition environment
 Background
• DOD is managing $1.68 trillion worth of acquisition programs
• DOD’s acquisition costs has increased by $135 billion over the past 2 years for
98 major defense acquisition programs
• GAO found that most of DOD’s portfolio total cost growth in past 2 years
occurred in the DOD’s largest programs, which are all in production
– This means programs experience significant problems and changes well after the
programs and their costs should have stabilized
Focus Areas
Source: GAO (2011), Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP
 GAO has continuously stated that in order to mitigate risks of
uncovering problems in the later phases of the development
cycle, knowledge based testing should be conducted at key
junctures, specifically around the Engineering and
Manufacturing Development (EMD) phase
Data Extraction
Mission
Need
Statement
Data form SAR, GAO, and DOT&E Annual Report, etc.
Between MS-B and MS-C
• Appropriate TRL level (GAO knowledge point 1)
• PDR (GAO knowledge point 1)
• CDR (GAO knowledge point 2)
• Minimum design changes after this point
• Manufacturing process should be stabilized before
entering Production phase (GAO knowledge point 3)
• Should not have significant cost growth at this
point
Data form SAR, GAO, and DOT&E Annual Report, etc.
Beyond MS-C
• RDT&E cost growth should be minimum
Significant RDT&E cost growth in the Production and Development Phase is the indicator
of insufficient T&E activities in earlier phases of the acquisition
Primary Source of Data

Selected Acquisition Reports (SARs)
• RAND conducted several research studies based on SAR data and identified numerous
“disadvantages" that an analyst must be aware of in order to maintain the validity of the
data . Most prevalent problems are:
o
o
o
o
o
o
o
o
•
Failure of some programs to use a consistent baseline cost estimate
Exclusion of some significant elements of cost
Exclusion of certain classes of major programs (e.g., special access programs)
Constantly changing preparation guidelines
Inconsistent interpretation of preparation guidelines across programs
Unknown and variable funding levels for program risk
Cost sharing in joint programs
Reporting of effects of cost changes rather than their root causes
Using SARs provides key “advantages”
o SARs conform to a strict reporting format, providing consistency to the data
o Those who create SAR reports receive annual SAR training, which adds to the consistency of the
data
o SARs are presented to Congress, so the level of scrutiny that SARs receive in the review process
bolsters both the consistency and accuracy of the documents. Databases in general contain
inaccuracies, but a database built from SAR data arguably withstands scrutiny better than most
Supplementary Data Sources
 DOT&E Annual Reports
•
•
•
•
•
Current Activities
Cost data
Issues
Milestone dates
Recommendations
 GAO Reports
•
•
•
•
Technology, Design, and Production Maturity
Cost Data
Milestone Data
Issues
Conceptual Model
Performance Indicators
Proposed Methodology
 Data Extraction
• Base year RDT&E dollars from SAR chosen for analysis, since these dollars
exclude estimated inflationary effects
• This format facilitates conversion of the various base years of individual
estimates into a single base year, making possible easy comparison across
programs
 Data Analysis
• Relationship between early adequate testing and program success
– RDT&E cost growth Prior/Post MS-C  Schedule Slip and Unit Cost Growth
– Performance indicators  Schedule Slip and Unit Cost Growth
Using SEM for Data Analysis
 Powerful multivariate techniques – Specialized version of
other analysis methods
 Advantages over Multiple Regression
• SEM programs provide overall tests of model fit and individual
parameter estimate tests simultaneously
• Regression coefficients, means, and variances may be compared
simultaneously, even across multiple between subjects groups
• Measurement and confirmatory factor analysis models can be used to
purge errors, making estimated relationships among latent variables
less contaminated by measurement error
• Ability to fit non-standard models, including flexible handling of
longitudinal data, databases with auto correlated error structures
(time series analysis), and databases with non-normally distributed
variables and incomplete data
An Example of Cost Growth Due to Inadequate Early Testing
GAO Early Knowledge Based Testing Activities
•
Compare Baseline and Current Status(Cost and Schedule)
•
RDT&E Cost increase in Production Phase indicate
that more problems discovers during that phase
•
It is an indicator that program did not perform
adequate early testing
Preliminary Results
Average % Growth
 Preliminary data analysis shows that
171.65
Aircraft
aircraft acquisition programs has
Helicopter
139.89
Ship
116.94
highest average percentage of total
Missile
95.35
Satellite
90.92
acquisition cost growth compared to
Land Vehicle
84.82
C3I
42.12
others programs.
 Recently DOT&E conducted a systematic review of recent
67major acquisition programs that experienced delays
• 56 programs (or 84 percent) had performance problems in
testing (either DT, OT, or both)
 56 programs were categorized and cost growth were analyzed
and compared with other programs in the same category
 Preliminary results shows that programs with inadequate
testing in the earlier phase of the development has higher
cost growth
Inadequate Testing Cost Growth
Other Cost Growth
Aircraft
244.94%
114.64%
Missile
116.26%
77.92%
n
16
8
8
11
9
6
13
Conclusion
 Knowledge based testing is essential in
detecting problems early in the acquisition
phase
 Program success depends on the knowledge
obtained during earlier tests in the
development of the system
 Structural Equation Modeling (SEM) is a
powerful statistical technique to analyze
program success factors from the T&E
perspective.
Future Work
 Complete Data collection and data base
construction
 Refine Performance Indicators and Conceptual
Model based on further research
 Perform data analysis using Structural
Equation Modeling (SEM) to test model and
find the correlation between program success
and “Performance Indicators”
Contact Information
Zaw P. Tun
(407) 595-4684
zph2k@gwmail.gwu.edu
Shahram Sarkani, Ph.D., P.E
(888) 694-9627
sarkani@gwu.edu
Thomas A. Mazzuchi, D.Sc.
(202) 994-7541
mazzu@gwu.edu
Backup
 References
•
•
•
•
•
•
•
Arena, M., Leonard, R., Murray, S., & Younossi, O. (2006). Historical Cost Growth of Completed
Weapon System Programs. Santa Monica CA: RAND
GAO (2011). Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP
GAO (2011). WEAPONS ACQUISITION REFORM :Actions Needed to Address Systems Engineering and
Developmental Testing Challenges, GAO-11-806
Hough, Paul G. (1992). Pitfalls in Calculating Cost Growth from Selected Acquisition Reports. Santa
Monica CA: RAND
Jarvaise, J., Drenzer, J., & Norton, D. (1996). The Defense System Cost Performance Database: Cost
Growth Analysis Using Selected Acquisition Reports. Santa Monica CA: RAND
Sipple , V. (2002). ESTIMATING ENGINEERING COST RISK USING LOGISTIC AND MULTIPLE
REGRESSION, Master Thesis, AFIT.
The office of the Director, Operational Test and Evaluation (DOT&E) (2011). FY 2011 Annual Report