NDIA Systems Engineering Division
Annual Planning Meeting
2013 Summary and 2014 Plans
Action Plan Status
Committee Reports
December 11-12, 2013
With placeholders for inputs from OSD and Services - TBD
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
1
Introduction
Status of 2013 NDIA SE
Division activities
• Relative to 2013 Plan
• SED plan from 12/12 meeting,
briefed at 2/13 meeting
Committee Reports
• 2013 Accomplishments
• 2014 Plans
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
2
NDIA SED 2013 Tasks
- Derived from DoD/Services Initiatives (Voice of the Customer)
Summary of DASD(SE) Goals
•
•
•
•
•
Continued excellence in SE support to programs and
acquisition decisions
Improved, consistent PPP engagement with programs
resulting in successful protection strategies
ASA (ALT)
OCSE
DASD (SE)
•Dev Planning
•SE Workforce
•SE methods/ tools
•Reliability
•Security
protection
•ExportabiIity
•Soldier/Small Unit
Challenge
•ERS
– New processes for Software Assurance, Trusted Microelectronics,
Exportability
•Anti-Counterfeit
Advocacy and impact in ensuring SE workforce capacity
•Risk Mgmt
and capability
Provide depth to acquisition policy and processes with
SE
•Schedule
risk
guidance, practices, and continuous learning opportunities
•Supplier
risks
Make progress on areas of DoD interest:
– Quantitative Risk Management
•Security design
– SE Tradespace Analysis
•SE models (MBx)
– Supply Chain Risk Management and Damage Assessment
– Anti-Counterfeit
•SoSE
– Advanced Modeling for Engineering Resiliency
DoD Sys Eng
2012/12/12 | Page-12
Distribution Statement A – Approved for public release by OSR, SR Case # 12-S-3121 applies.
Questions NDIA can help answer
YSTEMS
NGINEERING
OLICY
1.
2.
3.
4.
SAF AQRE
Identify the new skills required of an engineer to
succeed on an Agile development team
Recognizing that we can’t always create the ideal team, what
processes can be developed to assist engineers on an Agile
development team
The System of Systems mission level requirements analysis
process has been identifying requirements that satisfy a
mission and allocating those requirements to systems. The
relationship between those allocated requirements and
the application of new R&D products to meet those
requirements is not robust.
How can the Navy work with Industry to improve the
assessment of the compendium of R&D products* for
application to system requirements, to increase success of
the Technology Demonstration acquisition phase?
DASN RDT&E SE&P 2012
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
DASN (RDT&E)
SEP
•Agile in
acquisition
•SoS mission
reqts
•R&D assessment
5
Verbal
Inputs
Provided
•Policy (levels)
•Process
•Tradespace
•Architecture
•Cost Assessment
•Specialty Engrg
•People /
Workforce
(specialists vs.
generalists)
TOC
3
NDIA SED 2013 Action Plan Status
#
NDIA SED 2013 Tasks
Committee
Plan
1
Quantitative risk management guidance.
[DASD(SE)]
SEEC
(Brown, Rassa)
Ongoing action. Feb workshop planned.
Risk workshop planned, deferred.
No action in 2012 or 2013 – drop it?
2
Support Engineered Resilient Systems
(ERS) initiatives. Link ERS with M&S?
[DASD(SE)]
SED (Rassa)
M&S
IEEE/NDIA ERS workshop in planning.
Other actions pending. Division
organizational emphasis?
No action yet, still pending.
3
Anti-counterfeit – SE role, industry
feedback for supply chain risk
management, augment body of
knowledge [DASD(SE)]
SED
(Henry)
Issue SED request for 1:1 company
inputs provided direct to DASD(SE).
Potential follow-up workshop. Could be
company proprietary? Coordinate with
other NDIA, AIA, etc.
No request from OSD; no action
4
Design Patterns for Security – share
SERC research findings (UVA)
[DASD(SE)]
SED
(Henry)
Schedule Dr. Horowitz (UVA) for NDIA
SED briefing.
Done, Feb 2013 SED meeting.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
4
NDIA SED 2013 Action Plan Status
#
NDIA SED 2013 Tasks
Committee
Plan
5
Advanced Modeling: Model Based for “X”
(MBx) – how could acquisition strategies
be enhanced to use models (RFPs, etc.)
[DASD(SE)]
M&S
(Coolahan)
M&S committee study or report. Collect
industry input, examples.
See M&S plan.
6
SoSE: how to demand articulation and
insertion of SoS context during life cycle
acquisition? [DASD(SE)]
SoS
(Dahmann)
SoS committee study or report and
recommendations. International issues?

7
Soldier / Small Unit Challenge
[ASA(ALT)]
SED WG
(Serna)
WG to consider soldier and small unit
architecture.
Discontinued; lack of Army support
8
Design for Reliability – need RFP
language. [ASA(ALT)]
LCS
(Cothran)
Consider input on RFP language.
LCS committee is inactive; no chair
9
Security / Protection for coalition
operations [ASA(ALT)]
System
Assurance
(Croll)
SA committee to consider security
issues for coalition – provide
recommendations to SED for action
SA committee rechartered to SSE
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
5
NDIA SED 2013 Action Plan
Status
#
NDIA SED 2013 Tasks
10
Design for Exportability – BBP 2.0, foreign
military sales (FMS), direct commercial
sales (DCS) [ASA(ALT)]
11
Reconcile Agile development with DoD
5000.2 acquisition practices,
requirements, IPT structure, cost, skills
[DASN (RDT&E)]
SW
(Draper)
Conduct workshop. Collect best
practices (cross-discipline, SEI, DHS,
AFEI, INCOSE, …
Partner with SEI, ADAPT. See Agile
guidance. Kickoff ‘Agile and SE’ at NDIA
SE Conf – WG formation in 2014.
12
Consider how to move tradespace for
architecture early in the life cycle (preMDD)
Architecture
(Sheeley)
MA (Lohse)
Start initially with white paper vision on
use of architecture to bound/expand the
trade space (solution agnostic)
See Arch and MA status briefings
13
Collect industry best practices for growing
and retaining specialists vs. generalists
E&T (Gelosh)
How does industry keep its specialists?
See E&T briefing for status
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Committee
Plan
No SED action
TOC
6
NDIA SED 2014 Actions
#
NDIA SED 2013 Tasks
Committee
Plan / Status
1
Brief the existing M&S life cycle cost model report
at the NDIA SED February meeting
M&S
(Bergenthal,
Coolahan)
Plan for Feb 2014 NDIA SED meeting.
2
Review the Top 5 SE issues report on technical
authority and consider if any NDIA actions are
appropriate. Consider best practices (industry or
government) to implement technical authority.
NDIA SED
(Henry, Rassa,
Draper, Serna)
3
Request DoD briefing on how program startups
work (Dave Ahern). Provide industry best
practices on program startup/assist teams.
NDIA SED
(Henry, Rassa)
Schedule for NDIA SED meeting briefings.
4
Support MORS affordability handbook
development and community of practice initiatives.
Affordability WG /
NDIA SED
Serna
5
NDIA to provide recommendations on deployment
of SE 15288.1 and 15288.2 standards (e.g., RFP
language, transition, governance, compliance).
NDIA SED
(Draper / Rassa /
Roedler / Henry)
Coordinate with joint IEEE/DoD/industry WG.
6
Assess new DoDI 5000.02 and provide
recommendations on any NDIA implementation
actions needed (impact analysis).
SEEC (Brown) or
short term WG
Focus on industry clarification, adoption and
impact analysis.
7
Collect industry best practices for quantitative risk
management.
NDIA SED
(Rassa, Forbes)
Establish risk management WG, with meetings
held following NDIA SED meetings.
8
Support Engineered Resilient Systems (ERS)
initiatives.
NDIA SED / IEEE
(Rassa)
Follow on from open 2013 actions. IEEE activity
with NDIA support.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
7
NDIA SED 2014 Actions
#
NDIA SED 2013 Tasks
Committee
Plan / Status
9
Schedule committees for special topics and
detailed briefs at NDIA SED meetings.
NDIA SED (Henry)
Feb: SoS – Aug?
April: SSE (PPP) – June?
June: M&S
August: E&T
October (SE Conf plenary?): DT&E
10
Consider ways to increase NDIA SED
collaboration internally (committees, other NDIA
divisions) as well as externally (other
organizations, company representatives)
NDIA SED
Consider a consolidated and managed list of
NDIA external stakeholders at the SED level.
Ensure integration without duplicating effort.
11
Several committees could use more industry and
government participation – consider ways to build
more involvement, given current constraints.
NDIA SED
Collect all needs and descriptions in a
consolidated request to Eng VPs?
12
Document and publicize the procedures for
releasing reports through NDIA. Identify which
committee products are in draft form that need to
be formalized.
NDIA SED
(Rassa, Britt)
Reports need to go through NDIA (Britt) prior to
sending to the SED steering group for review.
For web publishing send to Britt Bommelje and
Rebecca Danahy.
13
Britt to provide a copy of the NDIA SED charter.
NDIA
Distributed via email.
14
Look at M&S use assessment and identify areas
for further tasks to support MBSE.
M&S
This could be useful to drive other research.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
8
NDIA SED 2014 Actions
#
NDIA SED 2013 Tasks
Committee
15
Some NDIA SED web pages are badly out of date
– SED leadership and committee to update web
pages under their areas of control.
NDIA SED /
Committee Leads
16
ODASD(SE) to provide guidance on next steps for
SE Effectiveness study (e.g., briefing SAEs)
NDIA SED /
ODASD(SE)
Dona to provide feedback.
17
Consider a follow-on to the root cause analysis
study of PSR findings.
NDIA SED /
ODASD(SE)
Potential POCs: Bob Rassa, Jim Thompson,
Pete Nolte, Gary Bliss, …
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Plan / Status
TOC
9
Committee Inputs
- 2013 Task Progress / 2014 Task Plans
Architecture (Arch)
Automatic Test (ATC)
Development Test and Evaluation (DT&E)
Education and Training (E&T)
Enterprise Health Mgmt (EHM)
Environment, Safety, and Occupational Health (ESOH)
Human-System Integration (HSI) – no slides
Joint Committee on Systems Engrg in Manufacturing (JCSEM) – no slides
Life Cycle Support (LCS) – no slides
Modeling and Simulation (M&S)
Mission Analysis (MA)
Systems Engineering Effectiveness Committee (SEEC)
Software Committee
System of Systems (SoS)
Systems Security Engineering (SSE)
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
10
Architecture Committee – 2013 Task Summary
•
Meetings:
• 7 Virtual meetings
• 1 Face-to-face meeting
•
SE Conference:
• 15 papers
•
Accomplishments:
• Completed the DoDAF Recommendations Report
•
•
Delivered it to Steve Henry
Posted on the website
• Presented Architecture Metrics results (worked in 2012) at the INCOSE Symposium
• Participated in SEI SoS Architecture workshop
• Received architecture informational briefings / demos from C.Gedo, Architecture
Stnds Group (DISA & GTRI), and on J6 Architecture portal from B.Piazza (JS J6)
•
In-work:
• Reviewing whitepaper “Role of Architecture to Influence the Development Planning
Tradespace” with the Mission Analysis subcommittee
• Open system architecture discussions with Dr. Flietz (AFRL)
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
11
Architecture Committee - 2014 Task Plan
Proposed 2014 Tasks:
Deliverables/Products
• Complete joint Architecture/Mission Analysis
white paper “Role of Architecture to Influence
the Development Planning Tradespace”
• Investigate status and make proposals
around subject of Open System Architectures
• Initiate joint discussions with Test Committee
• “Role of Architecture to Influence the
Development Planning Tradespace”
whitepaper
Schedule / Resources
Issues / Concerns:
• Release whitepaper 1Q
• Initiate Open System Architecture discussions
1Q
• Initiate joint Architecture – Test discussions 1Q
• Availability of committee members
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
12
Automatic Test Committee – 2013 Status
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
13
Automatic Test Committee - 2014 Task Plan
Projects Working Group
Proposed 2014 Tasks:
• Government/Industry Liaison Reports
• Detail, Staff, Schedule & Initiate new ATC
tasks:
• Industry Survey for Smart Automatic Test
System Opportunities
• Software Architecture for Synthetic
Instrumentation
Schedule / Resources
Status/Review/Present at regular ATC meetings:
• March/April (adjacent DoD ATS Management
Board (AMB) Meeting – Gov’t participating
committee) at NDIA HQ
• September in St Louis (AUTOTESTCON)
• December with SED (TBD)
No resource issues or support anticipated
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Deliverables/Products
• Semi-annual Reports posted on ATC Web-site:
•
•
•
•
•
•
US Army ATS Liaison Report
USAF ATS Liaison Report
US Navy/USMC-Air ATS Liaison Report
USMC-Ground ATS Liaison Report
DoD AMB Liaison Report
Commercial ATS Liaison Report
• Deliverables from new tasks: TBD
Issues / Concerns:
• Only one concern specific to ATC:
• Reduced defense budgets and fallout from
the Las Vegas GSA meeting has had a
significant impact on Government (AMB)
participation in many events involving travel
– not just NDIA - including holding virtual
AMB meetings.
• Response:
• ATC is holding its meeting at NDIA HQ
adjacent AMB scheduled meetings, and will
remain flexible in scheduling our meetings to
maximize attendance, participation, etc.
TOC
14
Developmental Test and Evaluation
Committee
Status and Plans December 2013
Beth Wilson, Raytheon
Steve Scukanec, Northrop Grumman
Industry Co-Chairs
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
15
DT&E Committee
2013 Status
Activity
Plans for 2013
Status/Plans
Scientific Test and
Analysis Techniques
STAT for T&E
Effective use of statistical
approaches for test optimization
(including DOE) with examples
White paper complete
Preparing journal publications
Presented results at SE conference
Modeling & Simulation
Evaluate test as a stakeholder
Joint meetings with SED meetings
Propose for 2014
NDIA T&E Conf
DT&E panel, STAT report
Conference cancelled
NDIA SE Conf
DT&E tracks
Joint tracks, metrics/STAT reports
Added Task Dec 2012:
Metrics
Provide more information on
leading indicators for T&E
Status: draft report
Plans: results at SE conference
Added Task Dec 2012:
Reliability
Develop a better test strategy
for reliability (collaborate with
IEEE Reliability Society)
Discussed at June 2013 meeting
Leads identified Nov 2013
Added Task Jun 2013:
Cyber Testing
Develop recommendations for
Cyber Guidelines 2.0 in 2014
Lead and team identified
Kicked off Nov 2013 (joint with ITEA)
Added Task Jun 2013:
Chief Developmental
Tester
Develop recommendations for
industry interaction with Chief
Developmental Tester role
Lead and team identified
Architecture
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Complete
In Process
TOC
Cancelled
16
DT&E Committee - 2014 Task Plan
Projects Working Group
Proposed 2014 Tasks:
Deliverables/Products
•
•
•
•
• Cyber Testing Recommendations white paper
• Chief Developmental Tester white paper
• Candidate KSABs for role (E&T)
• Industry interaction recommendation
• Tutorial on role for SE conference
• Inputs to reliability standards effort
• Joint meeting notes
Cyber Testing Project
Chief Developmental Tester Project
Collaboration with IEEE Reliability
Joint meetings with SED Committees
• Modeling and Simulation
• Architecture
• Systems Security Engineering
• Education and Training
Schedule / Resources
Issues / Concerns:
• Cyber Testing Draft Recommendations Feb 2014
• Weekly teleconferences started 12/3/2013
• Full white paper later in 2014
• Chief Developmental Tester
• Joint meeting with E&T with SED meeting
• Tutorial at SE conference
• Reliability standards input for Jan/Aug meetings
• Test as a Stakeholder joint meetings with SED
• Modeling and Simulation
• Architecture
• Systems Security Engineering
• Industry engagement
• NDIA T&E conference may be cancelled (Seattle
in March)
• ITEA conference (Denver in September) may be
postponed and moved
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
17
Summary of DT&E Committee
2013 Results/2014 Plans
SE
T&E
Topic
Activity
Test Optimization
2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing
2013-2014: Industry recommendations for cyber testing
Chief Developmental Tester
2013-2014: Industry interaction with new role
Performance Measurement
2013: Requirements Verification leading indicators report
Reliability
2013-2014: Recommendations for better reliability testing
Modeling and Simulation
2014: T&E Perspective for Modeling/Simulation products
Architecture
2014: T&E Perspective for Architecture views
Education and Training
2014: Chief Developmental Tester role
Systems Security
Engineering
2014: Cyber testing guidelines connections to
Program Protection Planning
Completed
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Current
Proposed
TOC
18
DT&E Committee 2012-2013
Statistical Test Optimization
Activity
Scientific Test and
Analysis Techniques
STAT for T&E
Plans for 2013
Examples of effective use of
statistical approaches for test
optimization (including DOE) for
an implementation framework
Status/Plans
White paper complete
SE Conference track: 2012 and 2013
ITEA journal publication
MORS conference Nov 2013
2012 SE Conference summit/workshop thread
-Tutorials on Monday 10/22
- Presentations on Wednesday 10/24
- Synthesis Panel on Wednesday 10/24
Follow-on Results:
- DT&E Committee white paper
- ITEA Journal article
- SE Conference track Oct 2013
2012 - 2013
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
19
DT&E Committee 2012-2013
Leading Indicator Metrics
Activity
Metrics
(with Performance
Measurement WG)
Plans for 2013
Provide more information on
leading indicators for T&E
Status/Plans
Draft report
Presented results at SE conference
2012 SE Conference workshop
- Follow-on workshop with System Performance Measurement WG
- Focused on Requirements Verification not addressed in first report
Potential Measures Identified:
- System Maturity Level
- Verification Requirement Maturity
- Technical Measures and Stakeholder Need
2012 - 2013
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
20
DT&E Committee 2012-2013
Reliability Testing
Activity
Reliability Testing
(with IEEE)
Plans for 2013
Develop a better test strategy
for reliability (collaborate with
IEEE Reliability Society)
Status/Plans
Solicit feedback on existing standards
Identify recommendations to improve
reliability testing
Reliability Testing Focus
- Software reliability standard
- Cyber testing initiatives
- Data for reliability growth measures
Input/Feedback Opportunities:
- IEEE meetings January and August
- Periodic revision periods
Project Leads:
Lou Gullo, Taz Daughtrey
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
21
DT&E Committee 2013-2014
Cyber Testing
Activity
Cyber Testing
Plans for 2013 - 2014
Review what industry does for
cyber testing, expand focus to
include interoperability
Status/Plans
Recommendations for Cyber Testing
guidelines
Project Lead: Dave Desjardins
Joint Project with ITEA
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
22
DT&E Committee 2013-2014
Chief Developmental Tester
Activity
Chief Developmental
Tester
Plans for 2013-2014
Status/Plans
Review service policies for new
role and industry
implementations
Propose model for industry interaction
White paper
T&E/SE conference presentations
The Secretary of Defense shall require that each major defense
acquisition program be supported by—
‘‘(A) a chief developmental tester; and
‘‘(B) a governmental test agency, serving as lead developmental
test and evaluation organization for the program.
Coordinating DT&E activities
Insight into Contractor activities
Oversee T&E activities
Inform government PM about contractor DT&E results
Project Lead:
Joe Manas
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
23
Backup
DT&E Committee Results 2008-2013
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
24
Summary of DT&E Committee
Test and Evaluation Efforts
DT&E Committee
Topic
Activity
DoD T&E Policy Study
2006 – 2008: Workshops and Study Report
Improving T&E in the DoD Acquisition Process
Integrated Testing
2008 – 2010: Integrated Test Study
NDIA presentations and tutorial
ITEA journal article
RFP Language for T&E
2010 – 2011: Comments for Update to OSD Guide
Incorporating T&E into DoD Acquisition Contracts
Test Optimization
2012: Statistical Optimization Conference Thread
2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing
Started 2013: Industry recommendations for cyber testing
Chief Developmental Tester
Started 2013: Industry interaction with new role
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
25
Summary of DT&E Committee
Systems Engineering Efforts
SE Collaboration
SED Committee
Activity
Software
2009: SW T&E Summit Recommendations
System of Systems
2010 – 2011: T&E for SoS Workshop and Initiatives
2012: Final Report on Best Practices Model
Performance
Measurement
2012: Leading Indicators for T&E workshop
Modeling and
Simulation
2011: Effective Use of Modeling and Simulation for T&E
2012: Distributed Model-Based Testing
Reliability
Started 2013: Recommendations for better reliability testing
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
2013: Requirements Verification leading indicators report
TOC
26
DT&E Committee Current Structure
Since 2010
NDIA
Systems Engineering
Division
NDIA
Test and Evaluation
Division
SED
Committee
Collaborative
Projects
T&E
Themes
For
Conferences
NDIA Industrial Committee
on Test and Evaluation
(ICOTE)
NDIA
DT&E Committee
SE Themes & Issues
Project Results
DT&E Themes
NDIA Systems Engineering Conference (Q4)
Collaborative Projects
T&E Themes & Issues
NDIA Test and Evaluation Annual Conference (Q1)
Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
27
DoD T&E Policy Study
August 2006: DT&E Committee Kickoff
Policy Study:
“Improving T&E in the DoD Acquisition Process”
Industry T&E policy recommendations
Workshops:
August 2007
January 2008
Focus Areas:
1. Earlier contractor and tester involvement
2. Integrated DT/OT and DT operational relevance
3. Suitability
April 2008: Report Summarized Results:
10 Findings
15 Recommendations
2006-2008
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
28
Integrated Testing (CT/DT/OT)
Implementation Framework
2008-2010
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
29
Software Test and Evaluation
Software Summit
RFP Language
Training & Competency Model
Policy, Guidance, and Standards
2009
Tools, Automation, Methodologies, Process
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
How Much
Testing
is
Enough?
Lifecycle
and
End-to-End
Software T&E
TOC
Changing
Paradigms
30
RFP Language
Master #
Industry #
Reviewer
Line
Comment and Rationale Recommended Input A/R/P
#
Industry Comments for Update:
Incorporating Test and Evaluation
Into Department of Defense
Acquisition Contracts
Recommendations
from SW Summit
2010-2011
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
31
Test and Evaluation for
Systems of Systems
2009: “Sleepless Nights”
List of Issues
2010: “Sominex”
Resulting Initiatives
2010: Workshop
2011: Best Practices
Wave Model
External Environment
Initiate
SoS
Conduct
SoS Analysis
Evolve
SoS
Arch
Develop
SoS
Arch
Implement
SoS
Update
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Continue
SoS Analysis
Evolve
SoS
Arch
Plan
SoS
Update
Plan
SoS
Update
2009-2012
Continue
SoS Analysis
Continue
SoS Analysis
Plan
SoS
Update
Implement
SoS
Update
Implement
SoS
Update
2012: Final Report
TOC
32
Effective Use of Modeling and
Simulation for Test and Evaluation
Joint Meeting in August 2011
• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test
and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data
Management Working Group
• LVC Architecture Roadmap Implementation
(LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group
• Raytheon Presentation on M&S for T&E
• Potential Topics for November AMSWG Meeting
2011
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
33
Modeling & Simulation for
Distributed Testing
Benefits:
•Find integration issues earlier
•Test to learn in ‘safe’ environment
•Protect proprietary information
•Facilitate DT to OT transition
•Increase performance testing range in operating environments
•Support end to end studies throughout the program
Barriers:
•Security
•Lack of persistent network
•Early consideration of technical issues
•Perceived value
•Disconnect between the communities (M&S and T&E)
Recommendations:
•Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)
•Create a framework for reusing and repurposing M&S through the product model
•Establish M&S as part of statistical test design
•Determine what tests are conducted to acquire data for model validation.
•Fewer test events with better models.
•Recommend the use of M&S to do I&T
•Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint Meeting
August 2012
Joint Track
SE Conference October 2012
2012
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
34
Education and Training Committee – 2013 Task
Summary
•
Established a Joint Competency Working Group (JCWG)
• Includes members from E&T Committee and INCOSE Competency WG
• Main objective is to evolve to a globally accepted* and marketed
standard competency framework, based on systems engineering
effectiveness, that can be used to produce competency models tailored
to the needs of the customer organizations.
•
Developed white paper and proposal to evolve current INCOSE UK
Competencies Framework along four paths:
• Evolve to an SE role-based competency framework that is extensible,
scalable, and tailorable by the customer organization.
• Evolve by adding the concept of classes. This is where the competencies
achieve their interdependence with each other.
• Evolve by including a Professional Class that covers leadership and soft
skill competencies.
• Evolve by adding a new level of proficiency called the Senior Practitioner
to the existing four levels in the INCOSE UK Competencies Framework.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC product.
* Approved for release as an INCOSE
35
Education and Training Committee - 2014 Task Plan
Proposed 2014 Tasks:
• Develop crisp definitions of Knowledge, Skills,
Abilities, Behaviors and Experience.
• Develop better definitions of the six classes.
• Refine definition of the Senior Practitioner level.
• Develop better definitions for each version (0.5,
0.75, and 1.0).
• Explore how system complexity affects SE Roles.
• Develop use cases.
• Ensure consistency with INCOSE Certification
program and SE Handbook.
Deliverables/Products
•
•
•
•
•
Competencies Framework V 0.5
Competencies Framework V 0.75
Competencies Framework V 1.0
Conference Papers and Presentations
Final Report
Schedule / Resources:
Issues / Concerns:
• Two weeks before INCOSE IW 2014, send
draft Version 0.5 to JCWG members.
• Conduct working session at IW 2014 to
finalize Version 0.5.
• Conduct working session at IS 2014 to
finalize Version 0.75
• Ultimate Goal: Produce Version 1.0 by
IW 2015.
• Few outside UK use framework – Need to seek
broader buy-in early and throughout
development process and show how we can
use evolved competencies framework to clarify
what a SE is, what they do, and why they are
important.
• Project may face analysis paralysis – Need to
set reasonable POA&M, stick with schedule,
discuss, move forward, exhibit strong
leadership along the way.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
36
EHM Committee – 2013 Task Summary
2013 Tasks Planned
Status
Accomplishments (deliverables, etc.)
- Comments
• Integrated Diagnostics
Handbook update
In-work
Accomplishments: OSD/ATL has been working on rewrite of MilHdbk-1814. EHM participation from May 2013. Effort to
continue through mid 2014
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
37
(EHM Committee - 2014 Task Plan
Proposed 2014 Tasks:
Deliverables/Products
• working with OSD and rewrite of Integrated
Diagnostic handbook
• Review/Comments of Integrated Diagnostics
Handbook as it progresses
Schedule / Resources
Issues / Concerns:
Status/Review/Present at regular EHM meetings:
• March/April EHM Meeting
• September EHM Meeting
• December with SED
• No Risks, Barriers or Concerns specific to
EHM; However, reduced defense budgets
has had a significant impact on
Government, and defense contractor
participation in many events
No resource issues or support anticipated
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
38
ESOH Committee – 2013 Task Summary
Completed 2013 Tasks:
• Supplemented the new Defense Acquisition Guidebook Chapter 4, Systems Engineering
• Teaming with DoD Acquisition ESOH IPT
• “Handbook for Human Systems Integration (HSI) & Environment, Safety, and Occupational Health
(ESOH) in Pre- Milestone A Analysis of Alternatives (AoA) and Joint Capabilities Integration and
Development System (JCIDS) Activities,” published October 2013
• “Guide to ESOH in the Systems Engineering Plan (SEP); Programmatic Environment, Safety, and
Occupational Health Evaluation (PESHE); and National Environmental Policy Act (NEPA)/Executive
Order (EO) 12114 Compliance Schedule,” published October 2013
• Coordinated with Aerospace Industrial Association (AIA) on rewrite of National Aerospace
Standard 411 (NAS411)
• Teaming with DoD Acquisition ESOH IPT
• Goal of rewrite: provide detailed implementation instructions to contractors on implementation of the
MIL-STD-882E, DoD Standard Practice for System Safety, Task 108, "Hazardous Materials
Management Plan"
• NAS 411, “Hazardous Materials Management Program,” published 30 September 2013
• NAS 411-1, “Hazardous Materials Target List,” published 30 September 2013
• Key element: List of hazardous materials categorized per Task 108 as Prohibited or Restricted
• Planned the NDIA SE Conference ESOH Track
• Combined HSI and ESOH Track and coordinated abstracts with the HSI Committee
• Six abstracts to address MRAP Lessons Learned for Rapid and Urgent Program ESOH
management, streamlining ESOH documentation initiatives, and updates on DoD Policy
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
39
ESOH Committee - 2014 Task Plan
Proposed 2014 Tasks:
• Focus on improving implementation of current
DoD Acquisition SE ESOH policy
• Continue work with AIA on NAS 411-1 list of
Tracked materials
• Work with DSOC/SOH Council on memo
regarding 15 Nov 2013 Army CAE memo on
ESOH risk management and reporting
• MIL-STD-882E implementation work
• Develop mechanism and collect Industry and
Government feedback on policy and guidance
• Quarterly webinars
Deliverables / Products
• List of Tracked Hazardous Materials
• Memo co-signed by P&R and DUSD (I&E)
• MIL-STD-882E corrections and DIDs
• Comments on improving DoD ESOH policy and
guidance, and MIL-STD-882E
• Webinars
Schedule / Resources
Issues / Concerns:
• NAS 411-1 list of Tracked materials – ECD Dec
2014/funding by AIA and DoD participants
• Memo regarding 15 Nov 2013 Army CAE memo –
ECD Jun 2014/funding by DoD participants
• MIL-STD-882E implementation work – TBD
• Develop mechanism and collect Industry and
Government feedback on policy and guidance –
ECD Dec 2014/funding by participants
• Quarterly webinars – ECD June 2014/DoD funded
• Agreement between DoD and AIA on Tracked list
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
• Time for staffing and advocacy for proposed P&R
and DUSD (I&E) memo
• Funding for MIL-STD-882E update development
• Time and funding for questionnaire development
• OSD support
TOC
40
Modeling and Simulation Committee – 2013
Task Summary
• Final report on M&S use assessment
- Briefing on product given at October 2013 Systems Engineering
Conference
- Key spreadsheet completed and shared with ODASD/SE/SA
liaison
- Final report draft in progress – to be completed in mid-December
• Committee meetings
- Meetings held in February, April, June, and August 2013
- Presentations posted on committee web site
• Report on essential elements of a System Model
-
Subcommittee formed in August 2013
Expect final report in July 2014
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
41
Modeling and Simulation Committee - 2014 Task Plan
Proposed 2014 Tasks:
Deliverables/Products
• Complete study to determine essential
elements of a “System Model” that evolves
over the life cycle (started in 2013)
• Provide continuing situational awareness for
M&S Committee members (ongoing)
• Motivate submittal of abstracts to populate
full-length M&S track at 2014 SE Conference
• Final report on essential elements of a
System Model
• Committee meeting presentations posted
on committee web site
Schedule / Resources
Issues / Concerns:
• Final report on essential elements of a
System Model – July 2014
• Have some volunteers; need to recruit
more
• Committee meetings in February, April,
June, and August 2014
• Need organizations to volunteer to host
• Finding volunteers to work on unfunded
study efforts is getting much more difficult in
current fiscal environment.
• Continuing Government restrictions on
travel and fees for professional society
activities is creating difficulties in
maintaining perception of value of society
activities to industry participants.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
42
Mission Analysis Committee – 2013 Task
Summary
•
•
•
•
Collaborated with ASD R&E/Research on the integration of
S&T/IR&D
 Industry involvement in the Multidisciplinary University
Research Initiative (MURI) process
 Increase Industry awareness and use of Defense Innovation
Marketplace
Developed draft whitepaper on “The Role of Architecture in
the Development Planning Trade Space” with the
Architecture Committee
Collaborated with the Government DPWG on Development
Planning Activities Models
 Best Practices, Techniques, Methodologies, and Tools
Completed all actions from June 2012 NDIA DPWG
Workshop, except:
 Intellectual Property as a barrier to successful DP
 OCI as a barrier to successful DP
 Final DPWG Workshop Report not submitted
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
43
Mission Analysis Committee - 2014 Task Plan
Proposed 2014 Tasks:
• Support the Government DPWG and ASD
R&E/Research on an as needed basis
• Organizational Conflict of Interest (OCI) Task
 Collaborate with the Government DPWG to
address OCI
 Develop a clear and concise understanding of
the FAR / DFARS language
 Communicate findings across Gov’t/ Industry
• Complete the “Role of Architecture in Development
Planning” Whitepaper with the Architecture Committee
Deliverables/Products
• “Role of Architecture in Development
Planning” Whitepaper w/ the Architecture
Committee
• Final Report on OCI Task findings and
recommendations
Schedule / Resources
Issues / Concerns:
• “Role of Architecture in Development
Planning” Whitepaper – April 2014
• OCI Findings and Recommendations – June,
2014
• OCI Final Report – August, 2014
• Diminishing number of contributing committee
members
• Developing Planning instantiation across
Government and Industry has been largely
successful. It is questionable as to the need to
continue the Mission Analysis Committee.
Resources may be better utilized elsewhere.
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
44
SE Effectiveness Committee
– 2013 Task Summary
•
Widely disseminated the results of the SE Effectiveness Study
through numerous channels
• Government presentations
o DASD(SE)
o USD(AT&L)
o DoD SE Forum
o Army SE Forum
o DoD DPWG
• Conference presentations
o IEEE Syscon
o INCOSE IS2013
o NDIA GVSETS
o NDIA SE Conference (panel)
• Presentations for INCOSE chapters
o Finger Lakes chapter
o Space Coast chapter
o Chicagoland chapter
• Corporate presentations
o Westinghouse Nuclear
o Northrop Grumman Corp.
o Harris Corp.
o Boeing
• Other presentations
o MITRE
o INCOSE webinar
o Podcast for SEI
• Released detailed survey results report
• Discussed follow-on work ideas with DASD(SE)
• SE self-assessment tool for programs
• “Lightweight” Pre-PSR program assessment
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
45
Systems Engineering Effectiveness Committee –
2014 Task Plan
Proposed 2014 Tasks:
Task 1: Develop a means of measuring risk management
process value
- Enable the evaluation of risk management in programs
- Enable the comparison of alternate risk management
paradigms
Deliverables/Products
• Task 1: Presentation at Annual SE Conference
• Task 2: Report out at year-end Planning Meeting
• Task 3: TBD
- Enable the improvement of risk management
processes
• Task 2: “Get this* out to the SAEs.”, (awaiting further
instructions from DASD (SE) – some possibilities listed
below)
-
Periodic self-assessment of programs
Pre- Program Support Reviews (PSR) assessment of
programs
- Task 3: Establish collaboration effort with INCOSE SE
Effectiveness Working Group
* The SE Effectiveness Study results (per The Honorary Frank Kendall, USD
(AT&L))
Schedule / Resources
Issues / Concerns:
•
•
•
•
• Working agreement with INCOSE SE Effectiveness
Working Group
Task 1: Oct 2014
Task 2: Dec 2014
Task 3: TBD
Task 1 & 2: Assigned liaison from DASD (SE)
office
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
46
SoS SE Committee
2013 Task Summary
•
Committee inputs to TTCP Product addressing SoS at key points in
the systems development process
•
•
•
•
February 2013: Initial presentation of TTCP draft
March 2013: Inputs from committee members on TTCP draft
November 2013: Shared Updated Product, with request for added inputs
December 2013: Inputs coming in now
• SoS Webinar Series: SoS Engineering Collaborators Information
Exchange (SoSECIE)
• 14 Webinars, each with between 40-100 participants
• Presentations are posted on SoS SE Website
http://www.acq.osd.mil/se/outreach/sosecollab.html
• Committee participation SEI workshop on SoS SW Reuse Patterns
• Committee Meetings focused on Issues of Interest
•
•
•
•
February - TTCP Product Review
April – T-Area-SoS – EU SoS Research Agenda Development Projects
July – Army Victory Architecture (with Architecture Committee)
August – M&S and SoS (with M&S Committee)
• Six SoS tracks at SE Conference, most joint with other committees
• Drafted 15288 annex for systems of systems accepted for revision
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
47
SoS Committee
2014 Task Plan
Proposed 2014 Tasks
Deliverables/Products
• Input and Feedback from TTCP
• Regular committee exchanges
 Focus on new and emerging
Service SoS Initiatives
 Joint meeting with SSE on
SoS/SSE
• SoS Webinar Series: SoS Engineering
Collaborators Information Exchange
(SoSECIE)
• Feedback on inputs to TTCP work
product
• Presentations on SoS SE Committee
Area on NDIA Website
• Webinars slides posted on DASD SE
site
Schedule / Resources
• Bi-Monthly committee exchanges
• Bi-Weekly Webinars
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Issues / Concerns
• None
TOC
48
Systems Security Engineering
Committee
Status and Plans December 2013
Holly Coulter Dunlap, Raytheon
Beth Wilson, Raytheon
Industry Co-Chairs
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
49
SSE Committee
2013 Status: NEW
Activity
Added Task Apr 2013:
New SSE Committee
Plans for 2013
Continue work of
Systems Assurance
Committee to follow up
on May 2012 Program
Protection Planning
(PPP) workshop
Status/Plans
Kickoff held June 18, 2013
Committee renamed Systems Security
Engineering (SSE) Committee
NDIA SE Conference:
SSE track, joint SSE/SoS track
Progress on 5 priorities identified May 2012
Follow-on Workshop planned for May 2013
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Complete
In Process
TOC
Cancelled
50
SSE Committee - 2014 Task Plan
Projects Working Group
Proposed 2014 Tasks:
Deliverables/Products
• PPP Implementation Workshop
• Joint meetings with SED Committees
• Developmental Test and Evaluation:
Connections between PPP and cyber
testing guidelines
• Systems of Systems: PPP leverage
points in the SoS Wave Model
• Workshop Recommendations
• NDIA SE Conference Progress Briefings
• Comments on PPP related guidance
Schedule / Resources
Issues / Concerns:
• PPP Workshop Apr/May
• Joint meetings with SED Committees Jun/Aug
• Developmental Test and Evaluation
• Systems of Systems
• Industry and government engagement
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
51
Summary of SSE Committee
2014 Plans
SED
SSE
Topic
Activity
PPP Workshop
2014: Follow-on to 2012 Workshop
Focus on Taxonomy and Metrics
May 20-22: MITRE facility in McLean, VA
Industry Inputs
Comments on guideline documents
Inputs into PPP implementation
Systems of Systems
2014: PPP leverage points in the SoS Wave Model
Developmental Test
and Evaluation
2014: Cyber testing guidelines connections to Program
Protection Planning
Completed
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
Current
Proposed
TOC
52
2013
Systems Security Engineering
• Restart Former Systems Assurance Committee
• New Systems Security Engineering Committee
• Kick-off June 18th
• Track at SE Symposium
• Planning follow-on workshop in 2014 on Program Protection Plan
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
53
2013
NDIA SE Conference
Issue Short Title
NDIA SE Conference Paper
1
Taxonomy
16290 – Critical Program Information Test Vector (Geoff “Ninja”
Donatelli, Raytheon)
2
Metrics
16185 – Software Assurance and NDAA 2013: Software Code
Quality Checking (John Keane DoD VA IPO, Vik Chauhan Deloitte
Consulting)
3
Contracts and
Acquisition Strategy
16223 – System Security Engineering and Comprehensive Program
Protection (Melinda Reed, OSASD SE)
4
Threat and
Attack Vectors
16051 – Engineering Your Software for Attacks (Bob Martin, Mitre)
16077 – Security Engineering in a Systems of Systems Environment
(George Rebovich, Mitre)
16001 – Strategic Cybersecurity Threat Analysis Framework: Know
Your Enemy to Defeat Your Enemy (Michele Myauo, Microsoft)
16111 – A Supply Chain Attack Framework to Support DoD Supply
Chain Security Risk Management (Dr. John Miller, Mitre)
5
Education
16153 – A Practical Educational Approach to Program Protection
Planning (Dr. Don Gelosh, Worcester Polytechnic Institute)
NDIA SE Division – Annual Planning Meeting
December 11-12, 2013
TOC
54