MDA Journey to Mission Success
March 3, 2008
Randolph Stone
Director-Quality, Safety & Mission Assurance
Missile Defense Agency
Approved for Public Release
08-MDA-3305 (29 FEB 08)
Agenda
• Reasons for Quality
• Ground-Based Midcourse Defense System (GMD)
- History
- Major Problems
- Impact
- Systems Engineering Improvements
- Cradle-to-Grave Approach
- Lessons Learned
- GM Successes
• Kinetic Energy Interceptor (KI)
- Award Fee
- Knowledge Points
- MDA Assurance Provisions
Approved for Public Release
08-MDA-3305 (29 FEB 08)
2
Reasons for Quality
Approved for Public Release
08-MDA-3305 (29 FEB 08)
3
Great Prophet 2
Approved for Public Release
08-MDA-3305 (29 FEB 08)
4
Taepo-Dong-1
Approved for Public Release
08-MDA-3305 (29 FEB 08)
5
Ground-Based Midcourse Defense
Sea-Based X Band Radar
Approved for Public Release
08-MDA-3305 (29 FEB 08)
Ground-Based Interceptor Emplacement
6
GM History
• Program Description
- National Missile Defense (NMD) Program began in 1996
under BMDO program
- Realigned as the Ground-Based Midcourse Defense
System (GMD) under MDA in 2002
– A key component of the nation’s multi-layered
Ballistic Missile Defense System
– The GMD includes sensors, command systems, fire
control systems, and Ground Based Interceptors
(GBI)
- In spiral development – 3 phases simultaneously
– Design and Development, Operational, Support
Approved for Public Release
08-MDA-3305 (29 FEB 08)
7
GM Major Problems
• Directed to produce and field while still in the development
& test phase
- Rapid deployment – enormous pressure on PD to meet
program schedule
- Led to high-risk decisions regarding schedule, product
quality, and program cost
• Acquisition reform – removed Mil Stds, relied on industry
“best practices”
- Traceability and sub-contractor flow-down
- High percentage of issues developed with sub-contractors
– Unauthorized changes, poor communication, lack of
prime oversight
Approved for Public Release
08-MDA-3305 (29 FEB 08)
8
GM Major Problems
• EKV team focused on technical aspects of design instead of
producability of end item
- No production control, constant engineering changes
- Schedule slips, nonconformances and quality escapes
• Commercial off the Shelf (COTS)
- No formal qualification in design and development for
COTS flight computer
- Computer reset – Root Cause unknown for over a year
- COTS became logistically expensive; would have been
easier to build
Approved for Public Release
08-MDA-3305 (29 FEB 08)
“Why Not
Not Fly?”
Fly?”
“Why
9
Impacts
• GM experienced a string of failures between 2000 and 2005
- “Interceptor production slowed as the year progressed
primarily because technical problems were
discovered…these problems were traced back to poor
oversight of subcontractors, too few qualification tests, and
other quality assurance issues.” GAO Report, March 2006
GMD Test Failures
Approved for Public Release
08-MDA-3305 (29 FEB 08)
10
IFT-13c
Example
Countdown automatically aborted due to a
built-in-test anomaly that detected excessive
drops of Electronic Control Unit (ECU)
messages.
Cause: Roundoff error propagation caused a
timing mismatch. When the SIGI and FC clocks
crossover, timing jitter occurs and occasionally
data is sent to the ECU when it is busy, resulting
in no responses. To avoid a recurrence, need to
raise the tolerance level for missed messages -changing one line of computer software code.
IFT-13c – December 2004
Approved for Public Release
08-MDA-3305 (29 FEB 08)
11
IFT-14
Example
One of the last steps in the launch sequence is to
open the silo doors and retract the horizontal
stabilizers that hold the missile in place. Sensors
indicated that one of the three lateral support
group mechanisms had not retracted, causing
the missile to abort launch
Cause: Corrosion and dissimilar metals
(galvanic corrosion) caused the failure. One
mechanism failed to latch due to wrong crush
blocks
Approved for Public Release
08-MDA-3305 (29 FEB 08)
IFT-14 – February 2005
These Tests
Tests Were
Were Wake-Up
Wake-Up Calls
Calls
These
12
MDA Systems Engineering Improvements
• MDA began to take a hard look at its quality practices
- Uncovered test and development problems through audits of
EKV and OBV
– Example: “Functional performance test not consistently
conducted during environmental testing”
- Commissioned 2005 Independent Review Team (IRT) study
– Dr. William Graham, Dr. Bill Ballhaus, MG (R) Bill Nance
– Confirmed not enough Ground Testing, Deficiency in
performance verification and systems engineering
– Recommended:
—Strengthen systems engineering
—Hold contractors accountable
Approved for Public Release
08-MDA-3305 (29 FEB 08)
Complete
Halttoto
Flight
Tests
These
Tests Were
Wake-Up
Calls
Complete
Halt
Flight
Tests
13
MDA Systems Engineering Improvements
• MDA began to use a back-to-basics systems engineering
approach
- Mission Readiness Task Force (MRTF)
– Improved readiness through systematic proven processes
— System Verification, Critical Design Reviews, System
Function Reviews
– Improved test rigor & gained confidence in testing procedures
- Restructured Contracts & Award Fee Processes
– “Paid for Failures”
- Program Management began supporting Quality
– Strategic Improvements: Infusing quality in contract,
empowering Mission Assurance Organization
– Tactical Improvements: Test Like You Fly, Dedicated
Ground Test Missiles, End-to-End Testing
Approved for Public Release
08-MDA-3305 (29 FEB 08)
Should
It Fly?”Calls
These“Why
Tests Were
Wake-Up
14
MDA Cradle-to-Grave Approach
System Functional Review
Software Specification Review
Continuous Activities
-Evaluate human interfaces, controls, displays
-S/W safety issues & Audits
- Mission Assurance Reps (MARs) on-site
Preliminary Design Review
Critical Design Review
-Subsystem and functional issues
Test Design Reviews (TDR)
(Gov. & Industry sites)
-Voting Member
- Mission Assurance Audits
Mission Operations Review
-Design Performance
-Producibility and risk areas
System Verification Review
Joint Services IM Munition WG
Interface Control WG
- Award Fee Boards
-Interface integration activities
--Independent Readiness Review Teams
-Ensures integration and compatibility
--Mission Readiness Reviews
-Resolve technical issues
Acquisition Strategy Panel
Design Certificate Review
-MAP, PMAP on Contract
Safety Hazards Assessment
-Mission Assurance Criteria
Development Test for Risk Reduction
-Award Fee Requirements
-FMECA, Sneak Analysis
Interoperability and/or Integration Test
AF
TEST LIKE YOU FLY
KV
DV
KI
SN
AL
IP
BC
AB
TH GM SN
LAUNCH
ACTIVITIES
AF
INTEGRATED
TEST AND
EVALUATION
PRODUCTION
DESIGN
AND
DEVELOPMENT
CONTRACT
AF
BMDS System Safety Working Group
System Requirements Review
AF
LAUNCH
TC
Mission Assurance Audits
Peer Reviews
-Surface correctable problems early
-Identify risk
-Ensure Best Practices
AF
Award Fee Board
Contract Award Fee Review - Annually
Voting Member
Approved for Public Release
08-MDA-3305MDA
(29Programs
FEB 08)
-Average 6 MA audits annually
Pre-Shipment Reviews
Pedigree Review
Flight Readiness Review
BMDS Parts & Materials Control Board (PMCB)
Flight Operations Review
-Voting Member
-Element PMP Control Boards (PMPCB)
Pre-Flight Readiness Review
-Voting Member
Launch Readiness Review
Tests and Inspections
Mission Readiness Review
-FOD Prevention, NDT, Acceptance Inspection
-4 to 6 weeks Prior to launch
-Mandatory Gov. Inspections (MGIs)
-Voting Member
Failure Review Board
BMDS Safety Officers (BSOs)
Material Review Board
-6 BSOs at Colo. Spgs.
Supplier Road Maps
MDA Executive Level Flight Test Reviews
Stakeholder Initiatives
-Series of 4 Flight test reviews leading to launch
Integrated Product Teams (IPT)
-Voting Member
15
FTG-02
Approved for Public Release
08-MDA-3305 (29 FEB 08)
16
GM Lessons Learned
• Good Quality practices from the
beginning will benefit in the long run
Intercept
• It is hard to change once the program
has started
- Hard to change culture
- Costly to change design
NO INTERCEPTOR
FAILURES
These Tests Were
Wake-Up SINCE
Calls 2005
17/17 Successful Intercepts
Approved for Public Release
08-MDA-3305 (29 FEB 08)
17
KEI – Kinetic Energy Interceptor
KEI Launcher
• Mission Assurance Philosophy Built-In
- Contracts & Program Culture
• In Design and Development Phase
- 1st and 2nd stage static fire tests
- KEI Fire Control and Communication
• First Flight Test – FY09
KEI Command Center
Kinetic Energy Interceptor
Approved for Public Release
08-MDA-3305 (29 FEB 08)
MDA’s New
Program
BuiltWake-Up
with Mission
Assurance
These
Tests Were
Calls
18
Kinetic Energy Interceptor
• Contract integrated with high-level of Mission Assurance
- Program Integration with Mission Assurance
Government Personnel
– Matrixed People from MA Organization
– KIQ: QS Direct Line
- Contractors held fully accountable for mission success
- Real-time incorporation of lessons learned
– Not sitting in a database
– Experience from across BMDS utilized
- Early Integration Product Team (IPT) Approach with all
suppliers
• Contract Structured with Award Fees
Government
Involvement
from theCalls
Beginning
These Tests
Were Wake-Up
Approved for Public Release
08-MDA-3305 (29 FEB 08)
19
KEI Award Fee Boards
• Unique Award Fee Criteria – Based on Knowledge Points and
Mission Assurance
Mission Assurance Actions and Metrics
- Schedule comes 2nd
• Find problems and identify issues
early and often
• Increased scrutiny on performance
- No longer “paid for failures”
• Set of steps to determine accurate award fee
- Performance Monitor – Including Mission Assurance Personnel
- Award Fee Review Board – Including Mission Assurance
Personnel
- Award Fee Advisory Board – Including Mission Assurance
Management
- Fee Determining Official – MDA Director
Incentives
to “Get
It Right
the First
Time”
These Tests
Were
Wake-Up
Calls
Approved for Public Release
08-MDA-3305 (29 FEB 08)
20
Award Fee Criteria – Knowledge Points
Kinetic Energy
Booster
• MDA Director’s Knowledge points
- Preplanned event that provides objective
evidence for key decision making
- Each added commitment hinges on knowledge
gained about critical risks
• Completed KEI Knowledge Points
- Demonstrate ONIR Accuracy for Boost Phase
Fire control (April 2006)
- Successful 1st and 2nd stage static firings
• Future Knowledge Points
- Booster Flight Tests (2008)
Approved for Public Release
08-MDA-3305 (29 FEB 08)
21
Award Fee Criteria – MDA Assurance Provisions
MAP Requirements Sections
MAP Rev A
3.1 Management
3.2 Design and Development
3.3 Software and Firmware
3.4 Technical and Mission Assurance
Reviews
3.5 Reliability, Maintainability, Availability
3.6 Parts and Materials Control Program
3.7 Integrated Test and Evaluation
Program
3.8 Test, Measuring, and Diagnostic
Equipment and Standards
3.9 Interface Management
3.10 Configuration Management
3.11 Control of Nonconforming Items and
Materials
3.12 Fabrication and Quality
3.13 Supplier Management
3.14 Safety
160These
Pages Tests
– Over
1800Wake-Up
“Shall” Statements
Were
Calls
Approved for Public Release
08-MDA-3305 (29 FEB 08)
22
Mission Assurance Toolbox
CEO
Days
Supplier Roadmaps
Pedigree
Reviews
Lessons Learned
Mission Assurance
And Safety Audits
MDA Assurance
Provisions
MDA Pedigree
Review Certification
Government /
Industry
Partnerships
Industry
Stakeholder
Initiatives
MDA
Mission Assurance
Representatives
MDA
Advisories
Approved for Public Release
08-MDA-3305 (29 FEB 08)
MDA Parts,
Materials
And
Processes
Plan
Memorandum
Of
Agreement
(MOA)
DCMA QA
Letter of
Delegation
QSMA
Unfettered
Access
23
FTG-03a
Approved for Public Release
08-MDA-3305 (29 FEB 08)
24
Questions?
Approved for Public Release
08-MDA-3305 (29 FEB 08)
25