Integration and Test
in a CMMI Level 5
Environment
AIAA Space 2009 Conference
14–17 September 2009
Cecilia Harrison
Warren Scheinin
Northrop Grumman Corporation
AIAA Space 2009, Harrison and Scheinin
The Steps to Success:
An Agenda in Reverse Order
Success
The Result: Managed Integration
The Details: How it is Done
The Solution: Engineering for Integration
The Problem: Some Assembly Required
The Environment: CMMI Level 5
The Set Up
2
AIAA Space 2009, Harrison and Scheinin
The Set Up
• Some products were not being delivered in as timely a manner or
within budget as planned
• Defects injected early in the product life cycle were not being
detected until late in the life cycle
• Components did not always plug together as easily as hoped
• Unanticipated and unwanted outputs needed to be eliminated
• Project personnel often experienced stress as milestones drew near
• Smooth running processes often hit bumps when external
stakeholders provided inputs
We knew we could be more efficient
3
AIAA Space 2009, Harrison and Scheinin
The Environment: CMMI Level 5
• Institutionalized processes, tools and stakeholder involvement are
focused on success factors
• Selected processes are optimized using results based, quantitative
methods
• Early life cycle processes are not allowed to crowd out later life
cycle processes at project start-up and early execution
• Staged representation lifts all process areas in parallel
• Continually improving performance through incremental and
innovative changes
4
AIAA Space 2009, Harrison and Scheinin
Climbing the CMMI Level 5 Ladder
• Each CMMI Level is a step to Higher Project Maturity
–
–
–
–
Starts with the foundations for a maintainable system
Gets your head above water
Clears the fog of fighting fires
Engages the supercharger
Level 3
Defined
Level 2
Managed
Level 1
Initial
5
The Ad Hoc Sink Hole
AIAA Space 2009, Harrison and Scheinin
Level 5
Optimizing
Level 4
Quantitatively
Managed
Level 4: Understand Your Processes and
Subprocesses
• Targets defined processes for quantitative control
• Take advantage of organizational knowledge
– Identify the implementations of processes which perform best
– Know that processes are performing within natural bounds that are
consistent across teams
– Surgical precision of Six Sigma methods
Defect Detection Profile
– Co-ordinate with other projects
• Better understanding and use of
Level 3 metrics
160.00
140.00
120.00
Defects/KSLOC
– Goal setting becomes more
important for Levels 4 and 5
– Need better definitions of the
measures and measurement
processes
– Lower level metrics of
subprocesses
Process Capability Metric
180.00
100.00
All Projec
New Proc
80.00
60.00
40.00
20.00
0.00
Req'mts
Design
Code
Unit Test
6
AIAA Space 2009, Harrison and Scheinin
Integrate
Life Cycle Phase
Phase
Sys Test
Del 90 Days
Level 5: Adding the Edge
• Get ahead of the curve by catching problems before they attack the
project
• Reduce the variation
– Initiate late cycle processes early
– Create procedures/checklists
– Strengthen process audits
• Increase the effectiveness (increase
the mean)
–
–
–
–
Train people
Create checklists
Reduce waste and re-work
Replicate best practices
Revolutionary Process Capability
Improvement
7
AIAA Space 2009, Harrison and Scheinin
The Problem: Some Assembly Required
• Things do not always work as intended
• Needed two-pronged attack on the classical weakness of integration:
surprise and delay
• Many projects disempowered integration and test until it is too late
• Major flaws are generally “baked into the cake” before integration
efforts even start
• Integration and Test teams are often brought onto the project far too
late to influence and/or contribute to the system design
• I&T Team can have negative affect on the dynamics of the project
team if not on board from “Day One”
• Risks need to be identified before they become problems
• Let software do it is not a good answer
8
AIAA Space 2009, Harrison and Scheinin
Worse Case Scenario
• Early Defects Cause Late Term Problems
• Need eyes of Integration and Test
Managers and leads early in
product life cycle
Requirements
Acquisition
Defect
s
Process Development
Design
Product Development
Test and Evaluation
Operations
Program
Failure
9
Missed defect in early product realization
processes provide opportunities for potential
failure modes to be created and propagated,
causing Mission failure
AIAA Space 2009, Harrison and Scheinin
Experience at NGIS
• Lessons Learned show that as the product development life proceeds,
the more positive the impact of early integration and test management
Integration
Management
Impact on Success
10
9
Risk Management
8
7
6
Tech Perform
Management
5
4
3
2
Schedule
Management
1
0
Scope Management
Subcontract
Management
Pre-Proposal
Program
Initiation
Program
Execution
Business Acquisition Phase
High leverage points
10
AIAA Space 2009, Harrison and Scheinin
Cost Management
Communications
Management
Quality Management
Issue Management
The Solution: Engineering for Integration
• Focuses on managing the integration and test efforts
• Mechanism to assess and improve system reliability starting from
“Day One”
• Empowers the Integration and Test team by involving them at the
earliest part of the project lifecycle
• Supports performance analysis to optimize throughput, capacity, and
product quality
• Proactively make demands of the system design to suit Managed
Integration and Performance Analysis needs
• Implemented in parallel to the System Development effort
Byproducts are satisfied customers and users
11
AIAA Space 2009, Harrison and Scheinin
The Details: Managed Integration
• Process of integrating hardware and software products in frequent,
small, additive steps into a full system environment
•
Typically characterized by:
– Deliberately conceived architectures that enable parallel, component-level
integration
– Test tools sufficient to analyze weaknesses in submitted products, so as to
enable their rapid correction/improvement
– Emphasis on evaluation of both internal and external interfaces
– Concentration on a meaningful set of basic functionality (Core Function and
Long Mission Threads)
– Early capacity for full operational stress conditions
– Potential for rapid turnaround of products and tasks in integration, such as
in “continuous integration” approaches
12
AIAA Space 2009, Harrison and Scheinin
10 General Elements of Managed Integration
• Early Infrastructure Selection
• System Architecture Skeleton Strategy
• Layered Service Provisioning
• Long Mission Threads Emphasis
• Strategic Component Selection
• Minimize Interdependencies
• Specify, Organize, and Control System States
• Automate Regression Testing
• Design for Performance Tuning
• Control Configurations and Versions
13
AIAA Space 2009, Harrison and Scheinin
The Details: Performance Analysis
• Targets details of implementation and obscure characteristics of
complex systems that can have extraordinary impact on overall
performance
• Solve issues that could become major factors impacting delivery
timelines
• Ensures that flow down of general performance requirements into
specific requirements are realistic and testable
• Uses system architecture skeleton to analytically extrapolates overall
understand of components on the system collectively to mitigate
overall performance risk
• Allows focusing of limited resources to components of the system
which will experience the greatest performance stress
14
AIAA Space 2009, Harrison and Scheinin
4 General Elements of Program for Performance
Analysis
“Program for Performance Analysis” is a well-planned effort to examine
performance and apply a variety of strategies to gain appropriate,
timely understanding of the system.
• Component Benchmarking
• Communication Channel Characterization
• Long Mission Thread Exercise
• Interface performance Description
15
AIAA Space 2009, Harrison and Scheinin
The Result: Managed Integration From Day One
16
•
Integration and Test managers and leads working side by side with
System Engineers and Architects to define scenarios that will drive a
design that meets the required performance
•
Early establishment of systems engineering and architecture
elements which most directly pertain to and predict integration
success
•
Proactively exploiting the emerging system in integration to achieve
higher quality, higher confidence results
AIAA Space 2009, Harrison and Scheinin
Indicators of Success
• More up front thinking means less work later
• More value-added products produced with reduced effort and time
– Instead of overrunning budgets and schedules, products are delivered
early and on budget
– Extra deliveries are possible without strain
• Less “help” from senior
management but more
resources available
• Lots of new work pours in
• Communications with
other groups is easier
Systems
• Safety
• Reliability
• Availability
• Maintainability
Program
Quality
• Quality
Assurance
• Software
Assurance
• Hardware
Assurance
Mission
Assurance
Supplier
AIAA Space 2009, Harrison and Scheinin
Reviews and
Assessments
• Risk
Management
• Qualification
• Selection
• Assessment
17
• Independent
A Level 5 Organization is Never Satisfied….
More
stakeholder
involvemen
t
A family
of
integratio
n
strategies
Add more
subprocesse
s
18
Better
metrics
More
efficient
statusing
AIAA Space 2009, Harrison and Scheinin
19
AIAA Space 2009, Harrison and Scheinin