Systems Engineering = Best Practices
Nicholas M. Torelli
Deputy Director, Human Capital and Specialty Engineering
Systems and Software Engineering (SSE)
Office of the Deputy Under Secretary of Defense
for Acquisition and Technology
Presented to
2009 Systems Engineering Summit:
Harnessing Best Practices
Sponsored by the
Huntsville Regional Chapter of INCOSE,
US Army, RDECOM, AMRDEC, DAU, and NASA
March 3, 2009
DoD Vision for Systems Engineering
Systems engineering principles and
disciplines are fully accepted and
assimilated into the DoD acquisition
workforce positioning the DoD for
acquisition excellence leading to a
stronger national defense.
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
2
SSE’s 2009 Priorities
• Acquisition work force development
• Renewed focus on early application of systems
engineering to affect affordability and total ownership cost
• Increased level of visibility for our role in developmental
test and evaluation
• Systems of Systems engineering tools and techniques
• Integration of program protection activities into acquisition
oversight to address cyber threat
• Measuring results of our efforts
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
3
DoD SE Best Practice Continuum
Policy &
Guidance
Developing
the Workforce
& Advancing
SE Practice
Program Support
& Assessment
Teamwork &
Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
4
Overview of Acquisition Policy Changes*
•
•
•
•
Strategic
Guidance
Mandatory Materiel Development Decision (MDD)
Mandatory competing prototypes before MS B
Mandatory PDR and a report to the MDA (“the sliding PDR”)
[PDR Report to MDA if before MS B; formal PDR Assessment by MDA if after MS B]
Configuration Steering Boards at Component level to review all
MS A
MS B
MS C
requirements changes
Joint
Concepts
CBA
ICD
Materiel
Solution
MDD Analysis
JCIDS Process
•
Technology
Development CDD
Engineering and
Manufacturing
Development
Production and
Deployment
O&S
Full Rate Production
Decision Review
or
PDR
CPD
CDR
Renewed emphasis on manufacturing during system development:
– Re-titles SDD phase to EMD with two sub phases: Integrated System Design
and System Capability and Manufacturing Process Demonstration
– Establishes consideration of manufacturing maturity at key decision points
•
•
Mandatory system-level CDR with an initial product baseline and followed
by a Post-CDR Report to the MDA
Post-CDR Assessment by the MDA between EMD sub phases
* DoDI 5000.02, 8 December 2008
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
5
Acquisition Policy Opportunities for SE
• Early SE engagement with programs
• Program Support Reviews (PSRs) Pre-MS A/B/C
• Risk Reduction activities (e.g., Technical Risk
assessment in AoAs, Competitive Prototyping)
• SE Technical Reviews - Informed Trades for Feasible
Solutions
• Developmental Test and Evaluation
– Integrated DT/OT
– Updated T&E Strategy at MS A
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
6
New Systems Engineering Enclosure
• Codifies several previous SE policy memoranda
• Codifies a number of SE-related policies and statutes
since 2003:
–
–
–
–
–
–
Environmental Safety and Occupational Health
Modular Open Systems Approach
Data Management and Technical Data Rights
Item Unique Identification
Spectrum Supportability
Corrosion Prevention and Control
• Introduces new policy on Configuration Management
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
7
DoD Guidance
•
•
•
•
•
•
Defense Acquisition
Systems Engineering
Developmental Test and Evaluation (DTE)
Modeling and Simulation (M&S)
Safety
System Assurance
http://www.acq.osd.mil/sse/pg/guidance.html
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
8
DoD SE Best Practice Continuum
Policy &
Guidance
Developing
the Workforce
& Advancing
SE Practice
Program Support
& Assessment
Teamwork &
Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
9
OSD Systems Engineering Strategy
DSB DT&E Report &
231 Report
Recommendations
•
•
•
•
Integrated DT/OT
Reliability Improvement
Early SE
Access to Relevant Data for
Evaluation & Decision Making
DT&E
Revitalization
Milestones A/B Critical
Correct SE Staffing
Component Development Planning
Pre-MS A analysis
OSD Policy & Guidance
NRC Study
Recommendations
•
•
•
•
Reliability
Workforce
Development
Systemic Analysis
Recommendations*
Enhanced SE
Pre-MS B
• Achievable Acquisition Strategy
• Enhanced Gate Review Process
• Enhanced Staff Capabilities
*Based on 3,700 Program Assessment findings
from 40 Programs Support Reviews
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
10
Workforce Development
The Defense Acquisition Community
126,033 Government and Military Certified Professionals
500,000+ Defense Industry Personnel
SSE Functional Leader for SE, T&E, and PQM workforce
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
11
SE Human Capital Strategy
• FY08 NDAA Section 852: DoD Acquisition Workforce
Development Fund: >$300M per year across DoD
– SE, PQM and T&E initiatives to Recruit, Train, and Retain the
workforce
– DoD Human Capital Initiative - Published Annex for SPRDE,
PQM, and T&E Career Fields
• SE “core competency” assessment effort; completion Summer 2009
• “Program Systems Engineer” career path
• Partnership with INCOSE SE Certification Program
– “CSEP-Acq” aligned with Defense Acquisition Guidebook
• Equivalency granted for DAU Courses SYS101 and SYS202
– Expanding potential for Industry “certifications”
• what the future could hold
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
12
Human Capital Initiatives
(Defense Acquisition Workforce Development Fund 1)
Defense Acquisition
Workforce
Development Fund
Recruiting &
Hiring
Training &
Development
Retention &
Recognition
Intern Programs
Training Enhancement
& Capacity Expansion
Retention & Recognition
Recruiting Incentives
Acquisition Workforce
Management & Student
Information System
Career Broadening &
Academic Programs
Outreach Programs
Competency
Management &
Assessments
Journeyman Hiring
Workforce Planning
Highly Qualified
Experts
1
Based on NDAA Section 852, Defense Acquisition Workforce Development Act
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
13
Workforce Size
Notional DoD Systems Engineering
Population
25-35
35-45
45-55
Workforce Age
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
>55
14
Notional DoD Systems Engineering
Workforce Strategy
Workforce Size
Retain:
Rotational Training Assignments
Education with Industry
Qualification Cards
Incentives
Hazardous Duty Pay for PSEs??
Train:
Mentor
Recruit:
Highly
Qualified
Experts
Retain
Recruit:
Journeymen
25-35
35-45
45-55
Workforce Age
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
>55
15
NDIA Systems Engineering Division
E&T Committee
Mission:
Strengthen Systems
Engineering capabilities
through education, training and
experience across government,
industry, and academia.
Contact Govt Committee Chair:
Dr. Don Gelosh, CSEP-Acq
ODUSD (A&T) SSE / HC
703-695-0472
donald.gelosh@osd.mil
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
2009 Task:
To address the issue of
“self-proclaimed”
Systems Engineers, the
E&T Committee is
attempting to determine
the core essential set of
competencies that
distinguish systems
engineers from domain
engineers.
16
SE Competency Model
Life Cycle View
Systems Thinking
Unit of
Competence
Competence
Analytical
Essence of
Systems
Analytical
Problem Solving
Element(s)
Lvl
Factors that Affect
the SE Competency Profile
Behavior
Structure
Goal
Problem
Solving
Method
Unit of
Competence
Competence
Element(s)
Lvl
Material
Solution Analysis
Technology
Development
EMD
Production &
Deployment
Systems Engineer
Composite Competency Map
SE Technical
Competence
Analytical
Stakeholder
Requirements Analysis
Analytical
Requirements Analysis
Analytical
Analytical
Element(s)
Lvl
Minimum
Essential
SE Criteria
Architecture Design
Implementation
Comp. Level
Unit of
Competence
Professional / Interpersonal
Unit of
Competence
Competence
Technical
Management
Decision Management
Technical
Management
Risk Management
Technical
Management
Configuration
Management
Technical
Management
Element(s)
Competence
Element(s)
Lvl
Time
SE Technical Mgmt
Unit of
Competence
Operations &
Support
Domain
Knowledge
Lvl
Technical Planning
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
17
Research to Practice
Policy
Guidance
Education &
Training
Ongoing
Development
Acquisition
Community
(DoD and
Industry)
SERC Governance &
Collaborators
• Co-Sponsors
• Government PMO
• Tasking Activities
• Industry
• Associations
• Academia
SERC
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
18
Need for SE Research
• State-of-the-practice must keep up with application needs
of DoD acquisition
– Methods, processes, and tools to enable effective acquisition and
sustainment of systems
– Leveraging Modeling & Simulation for SE
• Size and complexity of modern systems drive need for
commensurate extensions of SE,
– To System of Systems (SoS), to complex systems, to
“architecting,” to net-centric sets of services
– Validation & Verification challenges
• SE “theory and practice” should be inclusive of, and
establish linkages to, challenged sub-specialty areas
– Software engineering, reliability engineering, system safety,
costing, etc.
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
19
Stevens-SERC Team
Stevens Institute of Technology
Lead University
University Partners
•
•
•
•
•
•
•
•
Auburn University
Air Force Institute of Technology
Carnegie Mellon University
Fraunhoffer Center at UMD
Massachusetts Institute of
Technology
Missouri University of Science and
Technology (S&T)
Pennsylvania State University
Southern Methodist University
•
•
•
•
•
•
•
•
•
Texas A&M University
Texas Tech University
University of Alabama in Huntsville
University of California at San Diego
University of Maryland
University of Massachusetts
University of Southern California
University of Virginia
Wayne State University
The DoD Systems Engineering Research Center will be responsible for systems engineering
research that supports the development, integration, testing and sustainability of complex
defense systems, enterprises and services. Stevens has MOUs to develop enhanced SE
courseware for competency development within DoD with AFIT, NPS and DAU. Further, SERC
members are located in 11 states, near many DoD facilities and all DAU campuses.
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
20
Initial SERC Research Tasks
• “Assessing Systems Engineering Effectiveness in Major Defense
Acquisition Programs (MDAPs)”
– USC task lead, with support from Stevens, Fraunhofer Center, MIT,
University of Alabama at Huntsville
– Characterize SE effectiveness within the context of DoD acquisition and
identify methods of measurement suitable for application to project
execution organizations (e.g., defense contractors), program
management organizations (e.g., project managers and program
executive offices) and oversight organizations (e.g., OUSD(AT&L))
(OSD)
• “Evaluation of Systems Engineering Methods, Processes, and Tools
(MPT) on Department of Defense and Intelligence Community
Programs”
– Stevens task lead, with support from USC, University of Alabama at
Huntsville, and Fraunhofer
– Examine and recommend areas for advancing current SE methods,
processes, and tools (MPTs) as they are applied across the DoD
acquisition life cycle focusing on three different development
environments: individual weapons systems, SoS, and network-centric
systems (NSA)
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
21
Ways Your Organizations Can Become
Involved…
• Identify SE research challenges
• Provide funding to sponsor research as a Tasking Activity
• Collaborate on DoD-sponsored research by identifying
pilot programs to participate, provide relevant acquisition
information, identify subject matter experts
• Make use of research findings to improve systems
engineering on acquisition programs
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
22
DoD SE Best Practice Continuum
Policy &
Guidance
Developing
the Workforce
& Advancing
SE Practice
Program Support
& Assessment
Teamwork &
Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
23
Our World of Stakeholders
• DoD
–
–
–
–
Components and Agencies
Product Centers
Program Executive Offices
Acquisition Programs (PMs
and CSEs)
– Practicing PMs and SEs
– Other DAWIA Career Fields
• Industry
– First tier contractors (e.g.,
Lockheed Martin, Boeing,
Northrop-Grumman,
Raytheon)
– Second tier contractors
– Practicing SEs
– SE tool vendors
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
• Education & Training
Institutions
– DoD organizations: DAU,
AFIT, NPS
– US universities with SE
programs (~60)
– SE short course providers
• Professional and Industrial
Associations
– DAU Alumni Association
– INCOSE
– NDIA SE Division, other assoc
Div
– IEEE Standards Committee;
Systems Council
– AIAA SE Committee
– TechAmerica
– ISO
24
DoD Challenge: “A”cquisition Process
• ICD, CDD, CPD
• Future Threat and Op
Tempo
• Planning, Programming,
and Budgeting Process
• FYDP
• POM Process
• Appropriation and
Authorization of Funds
Budgeting
Requirements
“Sweet Spot”
• Acquisition Policy,
Guidance, and Oversight.
• MDAP Decision Authority
Acquisition
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
25
DoD Goal: Increase the Overlap
Budgeting
Requirements
“Sweet Spot”
Acquisition
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
26
Systems & Software Engineering
Organization Overview
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
27
Opportunities for SSE Engagement
SSE
Statutory Direction
Policy & Guidance
• Systems Engineering
Congress
• DT&E
Program Support
• Program Support
Reviews
• OIPT, T&E and SE
WIPTs
Sec Def
Requirements
Developers
Service Acquisition
Executives
• AOTR, Post-CDR
Review & Assessment
PEOs
Workforce Planning
Program Offices
• Competency Models
• Certification Reqts
• Education & Training
Emerging Concepts
• Systems of Systems
• SE Research
AT&L Direction
ICD, CDD, CPD
DAB, ITAR, DSAB, OIPT
Prime Contractors
Second Tier Contractors
Education & Collaboration Infrastructure
Professional / Industry Associations
(NDIA, INCOSE, AIA, ITEA, TechAmerica, etc.)
DAU, Academic Institutions, SERC
PSR, SEP, TEMP,
Technical Reviews
CSEP-Acq, Research,
Industry-University SE
Workforce Initiatives,
Industry-Gov’t
Projects
Outreach
• SE Forum
• Engagement Strategy
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
28
DoD Human Systems Integration (HSI)
Problem
•
HSI is not being consistently
integrated within DoD acquisition
resulting in increased ownership
costs and lower system
effectiveness.
Approach
•
•
•
•
Assign DUSD(A&T) Executive
Authority to define policy, as
necessary, to implement changes.
Maximize the good work that has
been done (policies, plans,
methods, tools and standards).
Engage DoD stakeholders and
strengthen industry affiliations
including NDIA, TechAmerica and
INCOSE.
Identify gaps and apply necessary
resources to resolve gaps or
barriers to successful HSI
implementation.
Cross-DoD and Technical Community Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
29
Collaboration with INCOSE
•
International Panel: “What Defines a
Systems Engineer? Comparing and
Contrasting Global Perspectives on
Systems Engineering Competency”
–
–
–
–
–
–
–
July 19 – 23, 2009
Moderator: Dr. Don Gelosh, Human Capital
Strategy and Planning, OSD Directorate of
Systems and Software Engineering
Dr. Arthur Pyster, Distinguished Research
Professor, School of Systems and Enterprises,
Stevens Institute of Technology; Deputy
Executive Director, DoD Systems Engineering
Research Center; and member of the INCOSE
Board of Directors
Dr. John Snoderly, Program Director, Systems
Engineering, Defense Acquisition University
Samantha Brown, President-Elect of INCOSE and
Systems Engineering Innovation Centre (SEIC),
Loughborough, UK
Mark Kupeski, Director, Complex Systems
Integration, IBM Global Business Services
Dr. U. Dinesh Kumar, Professor in Quantitative
Methods and Information Systems, Indian
Institute of Management Bangalore
Professor Stephen Cook, Director, Defence and
Systems Institute, University of South Australia
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
30
INCOSE Multi-Level
Professional Certification
Senior Level
Foundation Level
Discipline
Extensions
Entry Level
ESEP
CSEP-Acq
CSEP
ASEP
Expert Systems Engineering Professional
CSEP w/ US DoD Acquisition
Certified Systems Engineering Professional
Associate Systems Engineering Professional
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
31
CSEP with US DoD Acquisition Extensions
• Targeted towards Systems Engineers who support or
work in a US Department of the Defense acquisition
environment
• Same core CSEPs experience, education, and
knowledge requirements
• Additional acquisition knowledge items tested
• Available since July 2008
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
32
DoD Best Practices Clearinghouse (BPCh)
The DoD Acquisition Best
Practices Clearinghouse (BPCh)
facilitates the selection and
implementation of systems
engineering and software
acquisition practices
appropriate to the needs of
individual acquisition programs.
https://bpch.dau.mil/
The BPCh uses an evidencebased approach, linking to
existing resources that describe
how to implement various best
practices.
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
33
We Need You!
•
•
•
•
•
•
Browse the Clearinghouse for applicable best practices
Provide feedback on practices
Submit additional evidence for an existing practice
Submit a lead or a possible best practice
Tell others about the Clearinghouse
Consider becoming an editor to help vet practices
submitted in your area of expertise
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
34
DoD SE Best Practice Continuum
Policy &
Guidance
Developing
the Workforce
& Advancing
SE Practice
Program Support
& Assessment
Teamwork &
Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
35
OSD’s Program Support Reviews…
“The Source of Systemic Root Cause Analysis Data”
Rigor in process …
Defense Acquisition Program
Support Methodology (DAPS) v1.1
DAPS
1.0 Mission Capabilities/Requirements Assessment Area
Sub-Area 1.1 – Mission Capabilities/Requirements
2.0 Resources Assessment Area
Plan
Conduct
Analyze
Tools & Materials
• Tailorable, Scalable
Methodology (DAPS v1.1)
• Templates
• Criteria
• Questions
• Training Materials
• Execution Guidance
• Subject Matter Expertise
Report
Sub-Area 2.1 – Technology Development Planning and
Allocation
Sub-Area 2.2 – Personnel
Sub-Area 2.3 – Facilities
Sub-Area 2.4 – Engineering Tools
3.0 Management Assessment Area
Sub-Area 3.1 – Technology Development Strategy
Sub-Area 3.2 – Technology Development Planning
Sub-Area 3.3 – Program and Project Management
Sub-Area 3.4 – Contracting and Subcontracting
Sub-Area 3.5 – Communication
4.0 Technical Process Assessment Area
Sub-Area 4.1 – Technology Selection Assessment
Sub-Area 4.2 – Requirements Development
Sub-Area 4.3 – Functional Analysis & Allocation
Sub-Area 4.4 – Design Synthesis
Sub-Area 4.5 – System Integration, Test and Verification
Sub-Area 4.6 – Transition to SDD
Sub-Area 4.7 – Process Improvement
5.0 Technical Product Assessment Area
…ensures integrity
Sub-Area 5.1 – System Description
Sub-Area 5.2 – System Performance
Sub-Area 5.3 – System Attributes
6.0 Environment Assessment Area
Sub-Area 6.1 – Statutory and Regulatory Environment
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
36
Program Support Review Data Set
(since March 2004)
PSRs/NARs completed: 57
AOTRs completed: 13
Nunn-McCurdy Certification: 13
Participation on Service-led IRTs: 3
Technical Assessments: 13
Reviews planned for CY08:
PSRs/NARs: 10
AOTRs: 1
NM: 1
Decision Support
Reviews
NunnOTRR
9%
Data
derived from
diverse & broad
program set
Agencies
9%
DAE Review
5%
Pre-MS C
19%
Other
25%
Pre-MS A
3%
Pre-MS B
26%
Service-Managed Acquisitions
Air Force
33%
McCurdy
13%
Programs by Domain Area
Business 2%
Missiles 9%
Fixed Wing
18%
Other 2%
Space 4%
Comms 3%
Marine
Corps
10%
Navy
21%
Army
27%
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
Land 13%
Rotary Wing
21%
Munitions 3%
C2-ISR 13%
Ships 8%
Unmanned 4%
37
OSD Systemic Analysis:
Data Model
Tactical, Program and Portfolio Management
Use collective set of
program findings…
Program
Review
Findings
Program
Effects &
Root Cause
Program
Unique
Solutions
… to identify systemic issues Strategic Management
at the root cause level…
Systemic
Issues
… and develop
recommendations that mitigate
problems at their source!
DoD Acquisition
Community
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
Systemic
Root
Causes
• Policy/Guidance
• Education & Training
• Best Practices
Corrective
Actions
• Other Processes (JCIDS, etc)
• Oversight (DAB/ITAB)
• Execution (staffing)
38
Deriving Systemic Issues…
“Negative”
Findings
PSR
findings
Analysis
44 Programs
Prelim.
Analysis
Results
Findings
Pareto &
Pivot
Tables)
Systemic
Issues
Hypothesis testing: Re-analyze, review…
• Use “Core” and “Systemic” Root Cause tagging to cull data into focused subsets
• Analyze and trend using Pareto charting and Pivot Tables
Number of Programs that Experienced the CRC
31
30
Occurred on
50% or more of
the programs
30
30
No.of Programs
26
25
21
20
20
17
16
13
15
6
6
Rotations /
continuity
10
10
Acq reform: Loss
of MS A
requirement
35
5
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
Acq reform:
Transferred too
much authority to
CTR
Pool of clearable
skilled people
Core Root Cause
Budget POM
process (PBBE)
Other
Acq reform: Loss
of govt capital
investment
JCIDS process
Enabling
Infrastructure
External
Influences
Culture
Business practices
0
39
SRCA Milestones & Results
Sep 07 – Feb 08
Mar 08 – Oct 08
Oct 08 – Dec 08
OUSD/SSE root cause analysis yields systemic issues and
preliminary recommendations
OUSD/SSE sponsored NDIA Task Group to develop executable
acquisition recommendations
NDIA Task Group concludes effort and proposes actions for both
government and industry
3
Recommendation
Areas
Early & sufficient
planning
Acquisition
Strategy &
Planning
(ASP)
Decision
Gate
Review
(DGR)
Clear, enforceable
execution criteria
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
Enhanced
Staff
Capability
(ESC)
People: the right skills
and numbers
40
New Opportunities
for Independent Reviews
What’s relevant:  Mandatory Milestone A for all “major weapon systems”
 MS B after system-level PDR* and a PDR Report to the MDA
 EMDD with Post-CDR* Report and MDA Assessment
 PSR and AOTR in policy
MS A
Strategic
Guidance
Joint
Concepts
CBA
ICD
MS B
Materiel
Solution Technology CDD
MDD Analysis Development
Potential Independent Technical
Reviews - PSRs
and AOTRs
Program Support Reviews (PSRs)
 All ACAT ID and IAM
 To inform the MDA on technical planning and
management processes thru risk identification
and mitigation recommendations
 To support OIPT program reviews and others
as requested by the MDA
* PDR – Preliminary Design Review
Engineering and
Manufacturing
Development and
Demonstration
CPD
Production and
Deployment
OTRR
O&S
Full Rate
Production
Decision
Review
Assessments of Operational Test
Readiness (AOTRs)
 All ACAT ID and special interest programs
 To inform the MDA, DOTE, & CAE of risk of
a system failing to meet operational suitability
and effectiveness goals
 To support CAE determination of materiel
readiness for IOT&E
* CDR – Critical Design Review
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
MS C
* OTRR – Operational Test Readiness Review
41
DoD SE Best Practice Continuum
Policy &
Guidance
Developing
the Workforce
& Advancing
SE Practice
Program Support
& Assessment
Teamwork &
Collaboration
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
42
Always Our Focus
The Mission:
Delivering Timely and Affordable Capabilities to the Warfighter
The Defense Acquisition Community
126,033 Government and Military Certified Professionals
500,000+ Defense Industry Personnel
For additional information: http://www.acq.osd.mil/sse
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
43
DoD Guidance (1/2)
•
Defense Acquisition
–
–
•
Defense Acquisition Guidebook (DAG)
Integrated Defense Acquisition, Technology, & Logistics Life Cycle Management Framework, Version 5.3.2:
December 8, 2008
Systems Engineering
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
DAG Chapter 4, Systems Engineering
Systems Engineering Plan (SEP) Preparation Guide, Version 2.01, April 2008
SEP Frequently Asked Questions (FAQs), February 8, 2008
SE WIPT Brief: An Overview of Technical Planning and Systems Engineering Plan (SEP) Development,
Version 1.0, March 20, 2008
SE Working Integrated Product Team (WIPT) Generic Charter Template
Systems Engineering Guide for Systems of Systems, Version 1.0, August 2008
Systems Engineering Assessment Methodology, Defense Acquisition Program Support (DAPS), Version 2.0,
October 21, 2008
Guide for Integrating Systems Engineering into DoD Acquisition Contracts, Version 1.0, December 11, 2006
Risk Management Guide for DoD Acquisition, 6th Edition, Version 1, August 2006
Integrated Master Plan / Integrated Master Schedule Preparation and Use Guide, Version 0.9, October 21,
2005
Program Manager's Guide: A Modular Open Systems Approach (MOSA) to Acquisition, Version 2.0,
September 2004
Modular Open Systems Approach (MOSA) Program Assessment and Rating Tool (PART), Version 1.02
DoD Guide for Achieving Reliability, Availability and Maintainability, August 1, 2005
Program Reliability and Maintainability Review Template, Version 1.0, August 15, 2008
Designing and Assessing Supportability in DOD Weapon Systems: A Guide to Increased Reliability and
Reduced Logistics Footprint, October 24, 2003
Technical Review Checklists
http://www.acq.osd.mil/sse/pg/guidance.html
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
44
DoD Guidance (2/2)
•
Developmental Test and Evaluation (DTE)
–
–
•
Modeling and Simulation (M&S)
–
–
–
•
DoD 5000.59-M, DoD Modeling and Simulation (M&S) Glossary, January 15, 1998
Recommended Practices Guide for Verification, Validation, and Accreditation (VV&A), Build
3.0, September 2006
Guide for Modeling and Simulation for the Acquisition Workforce, Version 1.01, October 2008
Safety
–
–
–
–
–
•
DAG Chapter 9, Integrated Test and Evaluation
Guide on Incorporating Test and Evaluation into DoD Acquisition Contracts, FINAL DRAFT
FOR FINAL APPROVAL VERSION, September 2008
System Safety — ESOH Management Evaluation Criteria for DoD Acquisition, Version 1.1,
January 2007
Joint Systems Safety Review Guide for USSOCOM Programs, Version 1.1, October 12, 2007
Unmanned System Safety Guide for DoD Acquisition, Version 0.92, June 27, 2007
Joint Weapons and Laser Safety Review Guide (Draft), Version 0.92, September 4, 2007
ESOH in Acquisition: Integrating ESOH into SE, Version 3.0, January 2008
System Assurance
–
–
–
–
DAG Chapter 7, Acquiring Information Technology and National Security Systems
DAG Chapter 8, Intelligence, Counterintelligence, and Security Support
DoD 5200.1-M, Acquisition Systems Protection Program, March 16, 1994
Engineering for System Assurance, Version 1.0, October 2008
http://www.acq.osd.mil/sse/pg/guidance.html
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
45
DoD Best Practices Clearinghouse
(BPCh) Program Goals
Useful Information
Active Knowledge Base
Help finding,
selecting, and
implementing
practices appropriate
to User’s situation;
fill the gap between
“what” and “how”
Not just another practice
list; experience data
updated, expanded
refined; encourages
organic growth
A Single Source
For answers about
practices, how to apply
them, when they are
good to use; lessons
learned; and risks to
avoid
Repository
Living Knowledge
Integration with
DoD communities
of practice (ACC)
and experts
SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T)
Validated practices
Consistent, verifiable
information
46