Systems Engineering Standard
Technical Reviews & Audits Standard
Presentation to NDIA SE Division
11 December 2013
Garry Roedler (WG Chair)
1
Agenda
 DoD Goals and Objectives
 Background – Current Situation and Approach
 Current Status
 Looking Forward
2
Goal of DoD Standardization Initiative
 “Provide DoD and DoD contractors with a structured,
uniform approach” in the areas identified as gaps
 Improve communication
 Ensure common expectations
 Add realism to bids
3
Standardization Objectives
 Mutual-benefit stipulations:
 Must meet both party’s needs and objectives
 Potential teaming partners must have existing experience with subject matter of
document and existing infrastructure for publishing standards
 Content of documents must be consistent with government needs
 Ensure right kind of standard is developed
 Clear statement of requirements that speaks “common language”
 Specific enough to support realistic bidding and resourcing
 Proper contractual language – SHALL and should
 Ensure right amount of the right standards are included
 Tailorable for different domains and contracting environments
 Consistent with DOD approach
 Conform to established, over-arching industry process standards
4
Current Situation
Standard Focus
Systems Engineering
Responsible
SDO
Approach
ISO/IEC/IEEE 15288 (System Life Cycle
Processes) will be used as basis for Systems
Engineering needs.
IEEE-CS
Project is expected to produce an addendum
to 15288 – “Standard for Application of
Systems Engineering on Defense Programs".
No acceptable industry standard to use as
base.
Technical Reviews and
Audits
IEEE-CS
Configuration
Management
SAE Intl
Manufacturing
Management
SAE Intl
Logistics Support
Reliability
SAE Intl
TBD
New full standard to be developed “Standard for Application of Technical
Reviews and Audits on Defense Programs".
EIA 649B will be used as basis for
Configuration Mgt needs.
Project is expected to produce an addendum
to EIA 649B.
Recent selection - team is forming - approach
to be announced.
Revision of MIL-HDBK-502A, which provides
application guidance to the TechAmerica
Standard 0017
Still under review
Expected
Publication Notes
Full draft available for review in
mid-2014
Increments under review now as
drafted.
End of 2014 IEEE 15288.1
Full draft available for review in
mid-2014
Increments under review now as
drafted.
End of 2014 IEEE 15288.2
Mid 2015
TBD
Adopted by
DoD in June
2013
TBD
Intended to document best
manufacturing practices aimed at
promoting the timely
development, production,
modification, fielding, and
sustainment of affordable products
Each standard is intended to be put on future system contracts
5
DOD SE/TRA Standards Process
Present effort
6
Membership of DoD Joint SE Working Group
Industry
Associations













 AIA
Aerospace Corp
BAE Systems
Ball Aerospace
Boeing
General Dynamics
Harris Corp
L-3 Com
Leidos
Lockheed Martin
Northrop Grumman
Raytheon
SAIC
United Technologies
Defense
•
 IEEE-CS/SA
•
 INCOSE
•
 ISO/IEC
•
 NDIA
•
 SAE Intl
•
•
Air Force (Multi-parts)
Army
Navy (Multi-parts)
OSD – DASD (SE)
DAU
DCMA
DSPO
Leadership Team





Chair, Garry Roedler, Lockheed Martin
Vice-chair, Dave Davis, USAF SMC
Secretary, Brian Shaw, The Aerospace Corp.
Technical Editor, Bill Bearden, INCOSE (SE)
Technical Editor, Mark Henley, L-3 Com (TR&A)
Membership of WG Has Broad Span
Will be adding members to cover academia
7
Review of IEEE Standards Process
IEEE Process Stresses Consensus
8
Reviewers Focus
 Need to ensure:
 Technical accuracy
 Usability in defense environment
 Requirements are relevant and reflect good practice
 Requirements for deliverables are compatible with current
methods, and do not drive a particular method/technique
 Requirements do not result in unintended consequences
9
Schedule Information
 Business rhythm/cadence
 TR&A and SE will have 2-week phase separation for
increments
Meeting
Date +0
+7
WG Provide Inputs
+14
+21
Leadership Team
Initial Review
Editor Prepares Draft
WG Review of Draft
Draft to WG
for Review
+25
+28
Editor
Consolidates
Comments
Submit
Review
Comments
to Editor
Review
Issues from
Comments
at Mtg
10
IEEE Joint SE WG Schedule/Status

Integrated schedule balances efforts of both projects
Draft development : on-going and on schedule
 Working draft: May 2014
1
2
 Publication: December 2014


Each project = 7 development increments plus reviews and balloting

15288.1 (SE) 2 increments completed; 3rd increment in review.

15288.2 (TR&A) 2 increments completed; 3rd increment about to be released.
SE Standard (addendum 15288.1)
SE working group meetings
SE Draft outline
8/1
TRA 15288.2 Develoment Schedule - Rev 2
TRA working group meetings
Draft Outline
TRA Inc #1: 3 reviews/audits - ITR, SFR, SRR
TRA Inc #2: 3 reviews/audits - SAR, SSR, PDR
TRA Inc #3: 3 reviews/audits - CDR, IRR, TRR
TRA Inc #4: 3 reviews/audits - FRR, FCA, SVR
TRA Inc #1: 3 reviews/audits - PCA, P/MRR, ISR
TRA Inc #6: front matter, annexes]
TRA Inc #7: any additional info needed
TRA Working Draft Complete
TRA Mandatory Editorial Coordination (MEC)
TRA Close Ballot Group Invitation (draft near completion)
9/12
10/10
I
D
11/7
12/5
1/9
2/6
3/6
4/3
5/1
5/29
6/26
7/24
8/21
9/18
10/16
11/13
12/10
W
I
D
W
I
D
1
W
I
D
Balloting Period
W
I
D
2
W
I
D
W
I
Draft Development
D
W
D
W
5/8
5/15
6/5
6/5
6/19
7/24
8/14
9/4
9/25
10/16
12/10
12/20
8/1
9/5
X
10/3
I
D
W
I
10/24
D
11/21
12/19
1/23
2/20
3/20
4/17
5/15
5/29
6/26
7/24
8/21
9/18
10/16
11/13
W
I
D
W
I
D
W
I
D
W
I
D
W
I
D
W
I
D
W
5/8
5/15
TRA Draft Complete – Ready for Ballot
TRA Establish Small Ballot Comment Resolution Group
TRA Ballot start
TRA Ballot close/results
TRA Recirculation updates/ballot (as needed)
TRA Submit to RevCom (Hard Date - deadline for meeting)
TRA RevCom Meeting
TRA Publish
6/5
6/5
6/19
7/24
8/14
9/4
9/25
10/16
12/10
12/20
Note: 2014 RevCom Meetings
Aug 19-21; Dec 8-10
11
Currently on track
12/25/2014
12/20/2014
12/10/2014
12/4/2014
11/27/2014
11/20/2014
11/13/2014
11/6/2014
10/30/2014
10/23/2014
10/16/2014
10/9/2014
10/2/2014
9/25/2014
9/18/2014
9/11/2014
9/4/2014
8/28/2014
8/21/2014
8/14/2014
8/7/2014
7/31/2014
7/24/2014
7/17/2014
7/10/2014
7/3/2014
6/26/2014
6/19/2014
6/12/2014
6/5/2014
5/29/2014
5/22/2014
5/15/2014
5/8/2014
5/1/2014
4/24/2014
4/17/2014
4/10/2014
4/3/2014
3/27/2014
3/20/2014
3/13/2014
3/6/2014
2/27/2014
2/20/2014
2/13/2014
2/6/2014
1/30/2014
1/23/2014
1/16/2014
1/9/2014
1/2/2014
12/26/2013
12/19/2013
12/12/2013
12/5/2013
11/28/2013
11/21/2013
11/14/2013
11/7/2013
10/31/2013
10/24/2013
10/17/2013
10/10/2013
10/3/2013
9/26/2013
9/19/2013
9/12/2013
9/5/2013
8/29/2013
8/22/2013
8/15/2013
8/15
X
SE Inc #1:Business or mission analysis, Stakeholder needs
and requirements definition process, System requirements
definition process, Architecture definition process, Design
definition process, System analysis process
SE Inc #2: Implementation process, Integration process,
Verification process, Transition process, Validation
process, Operation process
SE Inc #3: Maintenance process, Disposal process,
Acquisition process, Supply process, Project planning
process, Project assessment and control process
SE Inc #4: Decision management process, Risk
management process, Configuration management process,
Information management process, Measurement process,
Qualifty assurance process
SE Inc #5: Lifecycle model management process,
Infrastructure management proces, Portfolio management
process, Human resource management process, Quality
management process, Knowledge management process
SE Inc #6:Clauses 1 - 5, annexes
SE Inc #7: additional info needed
SE Working 15288.1 draft complete
SE Mandatory Editorial Coordination (MEC)
SE Establish Ballot Group (draft near completion)
SE Draft Complete – Ready for Ballot
SE Establish Small Ballot Comment Resolution Group
SE Ballot start
SE Ballot close/results
SE Recirculation updates/ballot (as needed)
SE Submit to RevCom (Hard Date - deadline for meeting)
SE RevCom Meeting
SE Publish
15288.2
8/8/2013
8/1/2013
15288.1
Progress:
Joint WG for DoD SE Standardization - Schedule
D: draft issued
I: inputs for draft due
W: working group meeting discussion (Review Comments
due 2 days prior to meeting)
Looking Forward
Alignment of
SE Resources
 2014
 Development and publication of aligned base documents
 ISO/IEC/IEEE 15288 (DIS 1Q14; FDIS 3Q14; Publish 4Q2014)
 IEEE 15288.1 (draft 2Q14; publish 4Q2014)
 IEEE 15288.2 (draft 2Q14; publish 4Q2014)
 Also, INCOSE SE Handbook (2Q2014)
 Communicate across other working groups to ensure
consistency of terms, concepts, and other information


Liaisons established with CM and Mfg Mgt Standards WGs
Communication across WGs to agree on resolution of identified issues
 Start preparation for adoption, governance, support, use
 2015
 Adoption of standards (early 2015)
 Complete preparation for use of the standards
12
13
Source Products
 Source Products for SE Standard
 ISO/IEC/IEEE 15288 (CD.2), 2013
 ISO/IEC/IEEE 15289, Content of life-cycle









information products (documentation),
2011
SMC-S-001, Systems Engineering
Requirements and Products, July 2013
DoDI 5000.02
NATO AAP-48, NATO System Life Cycle
Processes, July 2012 (Addendum
Standard to 15288; focused on NATO
Armament Systems)
Defense Acquisition Guide (DAG), Chapter
4, Systems Engineering, 2013
INCOSE SE Handbook V3.2 /V4.0 Draft
SEBoK
EIA 632, Engineering of a System
ISO/IEC/IEEE 24748-4
NAVSEA Instruction & Policy 5000-009 for
Systems Engineering Technical Reviews
• Source Products for TR&A Standard
– ISO/IEC/IEEE 15288 (CD.2), 2013
– DoDI 5000.02
– SMC-S-021, Technical Reviews and Audits
for Systems, Equipment and Computer
Software, Volume 1, September 2009
– NAVSEAINST 5009.9, Naval Systems
Engineering Technical Review Handbook,
July 2009
– NAVAIRINST 4355.19E, Systems
Engineering Technical Review Process
– NAVAIR Systems Engineering Technical
Review Process Handbook, initial release
– Defense Acquisition Guidebook, Chapter 4.
Source Products include a
combination of Industry and
DoD resources
14
Source Products Usage
 Approach for Use of Source Products
 Use as body of knowledge for defense programs
 Identify relevant information that should be considered



Identify differences or additional tasks that need to be considered
Identify key outputs and their attributes
Identify essential guidance needed to provide common understanding
 Assimilate key information at level that applies to most defense
programs

State in terms of “what”, not “how”
15
Review of PARs and Technical Approach
Standard for Application of SE on Defense Programs
 Summary of PAR for Systems Engineering
 Identifier of Standard – IEEE Std 15288.1
 Title: Standard for Application of Systems Engineering on Defense Programs
– Scope:
• System life cycle processes,
activities, and tasks of
ISO/IEC/IEEE 15288 for use on
any defense system across the
life cycle
– Purpose:
• This standard implements
ISO/IEC/IEEE 15288 for use by
United States Department of
Defense (DoD) organizations
and other defense agencies in
acquiring systems or systems
engineering support.
– Need:
• Provide the defense specific
language and terminology for the
standard to ensure the correct
application of acquirer-supplier
requirements for a defense prgm.
– Technical Approach:
• Addendum to ISO/IEC/IEEE 15288
and will:
– Not repeat processes and
information in 15288
– Include defense specific
language and terminology
– Include necessary tailoring or
changes to existing elements
– Include any additional
explanation or guidance
16
Review of PARs and Technical Approach
Standard for Application of Technical Reviews & Audits
 Summary of PAR for Technical Reviews & Audits
 Identifier of Standard – IEEE Std 15288.2
 Title: Standard for Application of Technical Reviews and Audits on Defense
Programs
– Scope:
• Establishes the requirements for technical
reviews and audits to be performed
throughout the acquisition lifecycle for the
U.S. Department of Defense (DoD) and other
defense agencies.
– Purpose:
• Amplify ISO/IEC/IEEE 15288 Clause 6.3.2.3.a
for selection, negotiation, agreement, and
performance of the necessary technical
reviews and audits, while allowing tailoring
flexibility for the variety of acquisition
situations/ environments when the technical
reviews or audits are conducted.
– Need:
• Provide the defense specific language and
terminology for the standard to ensure the
correct application of acquirer-supplier
requirements for a defense program.
– Technical Approach:
• Standard will be in the form of a full
standard that has links to ISO/IEC/IEEE
15288 and will:
– Elaborate on the activities and tasks related to
TR&A
– Include defense specific language and
terminology needed for the standard
– Include the criteria for reviews & audits
– Include the expected/required outcomes/
products of reviews & audits
– Include any additional explanation or
17
guidance