Evaluate SE Methods, Processes and Tools
Technical Task Plan
USC Workshop
Los Angeles, CA
29 January 2009
Agenda
• Overview and changes in the MPT task
• Near term activities
– Sponsor environment
• Revised schedule
• Workshop activities
SOW Language
Look at current SE methods, processes, and tools (MPTs) as they are applied across the
DoD acquisition life cycle focusing on three different development environments:
individual weapons systems, SoS, and network-centric systems. Research will be targeted
at improving current/identifying new SE MPTs that will better support the practice of SE
in these three environments. Specifically, this task will:
1. Define critical attributes of current SE MPTs across the weapons system, SoS, and
network-centric services environments;
2. Identify strengths and weaknesses for these current MPTs and any shortcomings in
their application across DoD;
3. Recommend, in priority order, MPTs for further study to innovate or create improved or
new MPTs to eliminate identified shortcomings;
4. Upon selection by the government of MPTs recommended in sub-task 3 for further
study, perform research to innovate or create improved or new MPTs to eliminate
identified shortcomings, thereby advancing the state of practice of SE within the
community; and
5. For the improvements delivered in sub-task 4 above, propose a methodology for
validating the programs.
MPT Task Overview
3.3.1
Establish Criteria and
Validate with Sponsor
Apply Selection
Criteria
Identified Raw MPTs
3.3.2
Select MPTs for
Evaluation
HH
Complete Detailed
Attributes
HH
M M
M
HH M
LL
MPT
Sources
DoD Guideboooks
DoD Programs/Reviews
Service Repositories
Defense Industry
Commercial industry
Queue of selected and prioritized MPTs
3.3.3
Describe MPTs
Fully described
MPTs to evaluate
Apply Evaluation
Criteria
Evaluate MPTs
eWorkshop
Evaluated MPTs
Cumulative MPT
Coverage
BPCH
3.3.4
3.3.5
MPT Analysis
Repeat
3.3.6
To Users
Reports and
Recommendations
Recommended MPTs
Improvements Needed
Overall Gap Analysis
Research Areas
Systems Engineering and
Level-of-Effort Contracts
Dennis Barnabe, SERC PM
21 Nov 2008
SERC Kickoff Meeting
Slippery Slope Logic
• Mission-card
• Agility
• Prototype/Discovery
• LOE Contract
Relationships to Agility
Low
High
L
Local
L
H
L
L
L
Requirements Detail
Mission Satisfaction
Maintainability
Redundancy Risk
Scalability
Complexity/Size
Integration/Interoperability
H
Broad
H
L
H
H
H
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
Testing
Prototype or Discovery
Life Cycle
Planning
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
QRC
Testing
Life Cycle
Planning
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
Testing
Life Cycle
Planning
Development with eye toward sustained Ops
(Usual) LOE SE Implications
• Requirements lacking
• Limited (if any) ‘formal’ Reviews
– No coordination/insight among related efforts
– Interface and duplication risks
– No ability to assess technical health
• Standards application, etc
• No formal ‘transition’ planning
– What if it works?
• Build to Cost
– No actual cost estimate of satisfying mission need
– If successful, Operations cuts into Development
• Deemed ‘tech transfer issue’
•
Schedule lacking
– Inability to coordinate among other efforts
• “Success” defaults to ‘what is delivered’
MPT Task Overview
Apply Selection
Criteria
Identified Raw MPTs
3.3.2
Select MPTs for
Evaluation
HH
Complete Detailed
Attributes
Changes Required
3.3.1
Establish Criteria and
Validate with Sponsor
HH
M M
M
HH M
LL
MPT
Sources
DoD Guideboooks
DoD Programs/Reviews
Service Repositories
Defense Industry
Commercial industry
Queue of selected and prioritized MPTs
3.3.3
Describe MPTs
Fully described
MPTs to evaluate
Apply Evaluation
Criteria
Evaluate MPTs
eWorkshop
Evaluated MPTs
Cumulative MPT
Coverage
BPCH
3.3.4
3.3.5
MPT Analysis
Repeat
3.3.6
To Users
Reports and
Recommendations
Recommended MPTs
Improvements Needed
Overall Gap Analysis
Research Areas
Changes to identification process1
• Guidance:
– Focus on IC environment (context) changes strategy to initially
leverage BPCh Content Provider Network (CPN)
– Requires different candidate MPT collection strategy based on IC
context and requirements
• New Strategy:
– Extend context attributes of current MPT description to support
definition of IC environment
– Define and validate IC environment and requirements using a
revised MPT description template with extended context
attributes
– Compare to other environments (contexts) and where
similarities are found, mine environment for MPTs
Changes to identification process2
• Tactics:
– Develop initial set of context attributes and values that
characterize NSA environment based on current understanding
– Revise current MPT template to include extended attribute list,
MPT requirements, information summary, selection
recommendation and support rationale
– Validate attributes and practice criteria in requirements
interviews with NSA personnel (critical)
– Use template for 3-pronged MPT identification efforts
1.
2.
3.
Review sources provided in SOW and BPCh CPN
Review open literature and web-based sources
Capture current applicable NSA MPTs
1 and 2 can begin when initial template is available; 3 depends on
sponsor participation and ability to coordinate schedules/access
– Adapt selection criteria and process to using the new template
MPT Task Changes
Establish Preliminary
Sponsor Environment and
Needs
Develop Extended
MPT Template
Extended
environmental
attributes
Validate Environment and
Needs with Sponsor
Identify and Mine Comparative
Environments for MPTs
Review
sources
provided
in SoW
Review
literature
and web
Conduct interviews
Refine Templates
and Select MPTs
for Evaluation
H
H
H
M
M
L
Initial MPT Candidate
Templates
Queue of selected and prioritized MPTs
Access
experts
through
team
MPT Identification/selection Activities1
• Establish Preliminary Sponsor Environment and Needs
– Develop initial MPT evaluation/characterization template
– Revise and extend proposed attribute set
• Extend context attributes
• One-page template for candidate identification
• Validate Environment and Needs
– Interview sponsor personnel
• Describe the type of people to interview
• Develop interview structure based on the template
• Revise preliminary attributes, values and needs and capture new
ideas
– Revise/extend template as needed
– Revise evaluation criteria based on needs assessment
Italics indicate tasks of the workshop
MPT Identification/selection Activities2
• Gather MPT candidates from broader community based on
environmental description
– Identify best approach for this
– First target is INCOSE Workshop next week
• Identify comparable environments
– Through literature, web and expert inputs, identify
development/acquisition/deployment environments that share
attribute values with validated sponsor environment
• Mine comparable environments for candidate MPTs
– Review SOW-specified sources
– Review comparable environments as they are identified for
candidate MPTs
MPT Identification/selection Activities3
• Select MPTs for evaluation
– Review candidate templates
– Refine and extend template descriptions for promising
candidates
– Select evaluation candidates
Initial environment description
• The NSA environment can be described as
– A short development cycle to meet quick response needs with
lowered quality requirements at initial deployment
– Evolutionary deployment strategy that may begin with limited
deployment at relatively low-quality and evolve into broader
deployment at higher quality
– High level of interdependency with existing products
• “Mashing” and expanding of results from other projects to create
new results
• Providing new results for further processing by others
• Modifying existing capabilities to meet rapidly changing
constraints and/or availability of different data
• High level of glueware
Original MPT Attributes
MPT Attributes – What changes are needed?
Proposed Revised Schedule
MPT Activities during workshop
• First Session
– Clean up environment description (Rich, Ken)
• Less geek language – more general description
• Is there a taxonomy to help with completeness?
• Hopefully discuss with customer
– Determine and define attribute changes (including values) (Forrest)
– Develop the MPT mining template (Paul?)
• Second Session
– Brainstorm MPT mining activities (Paul)
• Opportunities, “helper” groups (INCOSE, Redstone SE group, etc.)
• Methodology
– Build necessary instruments for INCOSE (Forrest)
• Possible extra session after the SERC reception tonight?
– Possibly at Radisson or near airport (depending on majority)
Backup
Systems Engineering and
Level-of-Effort Contracts
Dennis Barnabe, SERC PM
21 Nov 2008
SERC Kickoff Meeting
Slippery Slope Logic
• Mission-card
• Agility
• Prototype/Discovery
• LOE Contract
Relationships to Agility
Low
High
L
Local
L
H
L
L
L
Requirements Detail
Mission Satisfaction
Maintainability
Redundancy Risk
Scalability
Complexity/Size
Integration/Interoperability
H
Broad
H
L
H
H
H
(Usual) LOE SE Implications
• Requirements lacking
• Limited (if any) ‘formal’ Reviews
– No coordination/insight among related efforts
– Interface and duplication risks
– No ability to assess technical health
• Standards application, etc
• No formal ‘transition’ planning
– What if it works?
• Build to Cost
– No actual cost estimate of satisfying mission need
– If successful, Operations cuts into Development
• Deemed ‘tech transfer issue’
•
Schedule lacking
– Inability to coordinate among other efforts
• “Success” defaults to ‘what is delivered’
Tailoring vice Avoidance
Iterative Acquisition
Development
Discovery
Deploy
?
90
Days
90
Days
QRC
Deployments
Operational Baseline
Right Tool for the Job
• LOE has its niche
• SE (& Acquisition) approach must evolve as Objective changes
–
–
–
–
–
Prototype/Discovery
QRC
Limited Ops
Full Ops
Production
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
Testing
Prototype or Discovery
Life Cycle
Planning
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
QRC
Testing
Life Cycle
Planning
SE “Equalizer”
Requirements
Configuration
Management
Technical
Reviews
Technical
Documentation
Testing
Life Cycle
Planning
Development with eye toward sustained Ops
Possible LOE SE Leverage Points
• Ensure Standard Inclusions
– On contract
– Adherence
• ‘Formal’ Gates for phase transitions
– Prototype/PofC QRC Limited Ops Sustained Ops
• Evolve SE Processes appropriately for given Phase
– TTOs must be written to support
Possible new contextual attributes
• Brainstormed attribute list with values where available – to be refined!
– Criticality for meeting requirements (QRC-high, QRC-low, high, medium,
low)
– Volatility/evolution of requirements (High (>1%/month), Normal(.011%/month), low (<.01%/month)
– Level of quality required at deployment (functional, reliable, critical)
– Level of security required at deployment (SCI, Classified, Unclassified)
– Dependence on other systems for critical data and functionality (Very
high, high, medium, low, none)
• Need to coordinate among other efforts
• Assessability of technical health (health of data sources required?)
– Length/stability of life cycle
• Stability of life cycle definition (phases)
• Evolution/stability of required ceremony in response to system life cycle needs
– how do I prepare enough ceremony up front to be able to make adjustments
easily when system maturity/deployment change – nondeterministic?
• Breadth of applicability
• Uniqueness of application (are 3 people already doing this and you don’t know
it)
• Scalability – in function and number of copies deployed
• Level of transition planning required