DoD Systems and Software Engineering
Taking it to the Next Level
25 Oct 2006
Mark D. Schaeffer
Director, Systems and Software Engineering
Office of the Deputy Under Secretary of Defense (A&T)
1
Office of the Under Secretary of Defense
Acquisition, Technology and Logistics
USD, Acquisition
Technology & Logistics
DUSD, Acquisition &
Technology
Technical Advisor,
Interoperability
Dir, Systems of
Systems Management
Defense Procurement
and Acquisition Policy
Dir, Systems and
Software Engineering
Industrial
Programs
Dir, Portfolio
Systems Acquisition
Small Business
Programs
Defense Acquisition
University
Defense Contract
Management Agency
Flatter, Leaner, Empowered!
2
Systems and Software Engineering
Mission Statement
 Shape acquisition solutions and promote early technical planning
 Promote the application of sound systems and software engineering,
developmental test and evaluation, and related technical disciplines
across the Department's acquisition community and programs
 Raise awareness of the importance of effective systems engineering
and drive the state-of-the-practice into program planning and
execution
 Establish policy, guidance, education and training in collaboration
with academia, industry, and government communities
 Provide technical insight to program managers and leadership to
support decision making
Evolving System Engineering Challenges
3
Systems Engineering Revitalization Framework
Policy
Acquisition
Community
SE and T&E
Communities
Guidance
Industry
Associations
Academic
Community
Program
Support
E&T
Driving Technical Excellence into Programs!
4
Systems Engineering
Revitalization Effort
 Issued Department-wide Systems Engineering (SE) policy
 Issued guidance on SE, T&E, and SE Plans (SEPs)
 Instituted system-level assessments in support of DAB, OIPT,
DAES, and in support of programs
 Established SE Forum to ensure senior-level focus within DoD
 Integrating DT&E with SE policy and assessment functions--focused
on effective, early engagement of both
 Instituting a renewed emphasis on modeling and simulation in
acquisition
 Working with Defense Acquisition University to revise curricula
(SPRDE, T&E, PQM, LOG, PM, ACQ, FM, CONT)
 Leveraging close working relationships with industry and academia
Necessary but not sufficient!
5
Vision for Systems Engineering
and Software
Systems
Engineering
Revitalization
 Competencies Improved
 Delivered Product Suite
• Courseware
• Policy/Guidance
• Program Support
methods
 Elevated Stature
 Raised Awareness
 Positive Influence
Systems and Software
Engineering
Centers of Excellence
 World class leadership
 Broaden to Software Engineering,
System Assurance, Complex
Systems-of- Systems
 Responsive and agile, proactive to
changing customer needs
 Focused technical assistance,
guidance, and workforce education
and training
. . . the Technical Foundation
that Enables Acquisition Excellence
6
Why the Focus on Software…
 Research investment has been static or declining
• DARPA computer science R&D funding 50% ↓ (’01 ’04, universities)
 Requirements growth 10X (% functionality) ’60s -’00s
 Need vs. skilled/clearable workforce - gaps increasing
 President’s Information Technology Advisory Committee
Report, February 2005
• Identifies SW as “major vulnerability”
• Recommends priority attention: “Secure Software
Engineering and Software Assurance” and “Metrics,
Benchmarks, and Best Practices”
 Cost, schedule and performance issues
Software is an increasingly, important factor
7
Systems and Software Engineering in
Programs Reduces Costly Mistakes
33% historical RDT&E
RDT&E Mistakes
Cost Growth
Under estimating engineering effort is Major source of error
Percent Cost Growth (CG)
35%
137 Systems
Applied to
$222.8B RDT&E FYDP**
30%
25%
20%
15%
10%
5%
0%
Other Mistakes
1/3 *
SSE
impact
Yields a
Potential
$73.52B
RDT&E Cost Growth FYDP
ILS Factors; Spares & Support
Schedule Slips/Management Factors
Engineering/Test/Development
Production Assumptions & Estimation
* SSE positive impact on just
1/3 of RDT&E mistakes (11%)
% of RDT&E Total
-5%
Source: DoD Cost Avoidance Study (CAIG) 10 year ongoing
Yields a
Potential
$24.51B
RDT&E Cost Avoidance FYDP
**SAR data for MAIS and MDAP programs under OSD Systems Engineering Oversight
8
State of Systems Engineering
Director, Systems & Software Engineering
Director, Systems &
Software Engineering
Mark Schaeffer
Deputy Director
Enterprise Development
Bob Skalamera
SES
CORE COMPETENCIES
• SE Policy
• SE Guidance
• SE in Defense Acquisition
Guidebook
• Technical Planning
• Risk Management
• Reliability &
Maintainability
• Contracting for SE
• SoS SE Guide
• SE Education and Training
• DAU SE Curriculum
• SPRDE Certification Rqmt
• Corrosion
• R-TOC
• Value Engineering
Deputy Director
Developmental Test
& Evaluation
Chris DiPetto
SES
CORE COMPETENCIES
• DT&E Policy
• DT&E Guidance
• T&E in Defense
Acquisition Guidebook
• TEMP Development
Process
• DT&E Education and
Training
• DAU DT&E Curriculum
• DT&E Certification Rqmt
• Joint Testing, Capabilities
& Infrastructure
• Targets Oversight
• Acq Modeling & Simulation
• Energy
• DSOC/Acq Tech Task Force
Est.
Aug 06
SES
Deputy Director
Software Engineering &
System Assurance
Mark Schaeffer (Acting)
SES
CORE COMPETENCIES
• SWE and SA Policy
• SWE and SA Guidance
• SoS, SA Guides
• SWE and SA Education and
Training
• DAU SW Acq Curriculum
• Continuous Learning
Modules for SWE, SoS, SA
• Software Engineering
• Acquisition Support
• Software Engineering
Institute (SEI)
• Process Improvement
• CMMI Sponsor
Deputy Director
Assessments & Support
Dave Castellano
SES
CORE COMPETENCIES
• Support of ACAT I and
Other Special Interest
Programs (MDAP, MAIS)
• Assessment Methodology
(Program Support
Reviews - PSRs)
• T&E Oversight and
Assessment of Operational
Test Readiness (AOTR)
• Systems Engineering and
Developmental Test
Planning and Support
• Lean/6-Sigma Training/Cert
• DoD/National Software
Investment Strategy
Acquisition program excellence through sound systems and software engineering
9
DoD Software Center of Excellence
Organizational Tenets
 Support Acquisition Success
• Ensure effective and efficient software solutions across the
acquisition spectrum of systems, SoS and capability portfolios
 Improve the State-of-the-Practice of Software Engineering
• Advocate and lead software initiatives to improve the state-ofthe-practices through transition of tools, techniques, etc.
 Provide Leadership, Outreach and Advocacy
• Implement at Department and National levels, a strategic plan
for meeting Defense software requirements
 Foster Software Resources to meet DoD needs
• Enable the US and global capability to meet Department
software needs, in an assured and responsive manner
Promote World-Class Leadership for Defense
Software Engineering
10
What We’re Finding
and
What We’re Hearing
11
Top 10 Emerging Systemic Issues
1. Management
• IPT roles, responsibilities, authority, poor communication
• Inexperienced staff, lack of technical expertise
2. Requirements
• Creep/stability
• Tangible, measurable, testable
3. Systems Engineering
• Lack of a rigorous approach, technical expertise
• Process compliance
4. Staffing
• Inadequate Government program office staff
5. Reliability
• Ambitious growth curves, unrealistic requirements
• Inadequate “test time” for statistical calculations
6. Acquisition Strategy
• Competing budget priorities, schedule-driven
• Contracting issues, poor technical assumptions
7. Schedule
• Realism, compression
8. Test Planning
• Breadth, depth, resources
9. Software
• Architecture, design/development discipline
• Staffing/skill levels, organizational competency (process)
10. Maintainability/Logistics • Sustainment costs not fully considered (short-sighted)
• Supportability considerations traded
Major contributors to poor program performance
12
Top 5 SE Issues
NDIA Workshop – July 2006
1. Key systems engineering practices known to be effective are not
consistently applied across all phases of the program life cycle.
2. Insufficient systems engineering is applied early in the program life
cycle, compromising the foundation for initial requirements and
architecture development.
3. Requirements are not always well-managed, including the effective
translation from capabilities statements into executable
requirements to achieve successful acquisition programs.
4. The quantity and quality of systems engineering expertise is
insufficient to meet the demands of the government and the defense
industry.
5. Collaborative environments, including SE tools, are inadequate to
effectively execute SE at the joint capability, system of systems
(SoS), and system levels.
NDIA Systems Engineering Division - Top 5 SE Issues
July 26-27, 2006
13
Top 7 Software Issues
NDIA/DoD Workshop – August 2006
1. The impact of requirements upon software is not consistently
quantified and managed in development or sustainment.
2. Fundamental system engineering decisions are made without full
participation of software engineering.
3. Software life-cycle planning and management by acquirers and
suppliers is ineffective.
4. The quantity and quality of software engineering expertise is
insufficient to meet the demands of government and the defense
industry.
5. Traditional software verification techniques are costly and ineffective
for dealing with the scale and complexity of modern systems.
6. There is a failure to assure correct, predictable, safe, secure
execution of complex software in distributed environments.
7. Inadequate attention is given to total lifecycle issues for COTS/NDI
impacts on lifecycle cost and risk.
NDIA Systems Engineering Division - Top SW Issues
August 24-25, 2006
14
Defense Software Strategy Summit
 OSD led Summit co-hosted by NDIA Oct 18-19, 2006
• Build on the Top 7 SW Issue Workshop Findings
 Nearly 100 participants from across DoD, industry and
academia
 Identified numerous issues, barriers and actionable
recommendations across:
• Software acquisition and sustainment
• Software policy
• Human capital
• Software engineering practices
 Next steps:
• Final report forthcoming
• Prioritize work, develop action plans, and move out
15
Challenges Remain
 Implementing a DoD vision and strategy for software
 Component and Industry adoption and effective implementation of
sound SE practices as early as possible in the system life cycle
 SE Working Integrated Product Teams (SE WIPTs)
 Retention and development of technical acumen in an aging and
shrinking acquisition workforce
 Meeting all requests for technical support to programs
 SE support to Acquisition Initiatives stemming from the QDR
 Continue to evolve “high visibility” initiatives:
- Energy
- System-of-Systems
- CMMI
- Modeling & Simulation
- Defense Safety
Oversight Council
- System Assurance
16
How do we get there?
DoD
Centers of
Excellence
17
CMMI: New Release and Next Steps
Issues:
 Integrity of CMMI appraisals
 Misperception and misuse of the CMMI by acquirers
Actions:
 Implemented changes to the CMMI v1.2 product suite to ensure:
• Integrity of appraisals
• Quality of the product suite
• Education of acquirers
• Opportunities for streamlining where appropriate
 Developing a CMMI model for Acquirer process improvement
• Partnership with General Motors
• Stakeholders cross DoD, Govt Agencies and Industry
 Writing a CMMI guidebook
• Help acquirers understand what CMMI is and is not
 Conducting study of actual process implementation post-Level 5
19
System Assurance
Issues:
 Vulnerability of our systems to malicious tampering or access
 Numerous assurance, protection and safety initiatives that are not
aligned
Actions:
 Developing a comprehensive System Assurance strategy
 Promoting nationwide collaboration
 Identifying standards activities to address system vulnerabilities
 Developing a Handbook for Engineering System Assurance
• Guidance for PMs and Engineers on how Systems Engineering
practice can be applied to mitigate system vulnerability to
malicious control/tampering
20
System-of-Systems Engineering
 DUSD (A&T) directed OSD-led effort to develop and publish
System-of-Systems (SoS) Systems Engineering guide
• 6-month effort addressing areas of agreement across community
• Initial focus on SoS with stated requirements and organizations
responsible for execution
• Addresses DAG technical process and considerations for
technical management across system life cycle
- Focused on systems engineering challenges characteristics of SoS
and suggested approaches
• Audience: Program Managers and Lead/Chief Engineers for
SoS acquisition programs, legacy systems, and constituent
programs
Draft of initial version of guide is out for review
21