Process Improvement for
Software Projects
CMMI and Lean Six Sigma
USC CSC1 510
16 Nov 2015
Rick Hefner, Ph.D.
California Institute of Technology
rhefner@Caltech.edu
Agenda
• Current Challenges Facing the Software Industry
• Capability Maturity Model Integrated
• Lean Six Sigma
• Corporate Support
2
What are some of the challenges
facing software companies?
3
Software Projects Have Historically Suffered from Mistakes
People-Related Mistakes
1. Undermined motivation
2. Weak personnel
3. Uncontrolled problem
employees
4. Heroics
5. Adding people to a late
project
6. Noisy, crowded offices
7. Friction between developers
and customers
8. Unrealistic expectations
9. Lack of effective project
sponsorship
10. Lack of stakeholder buy-in
11. Lack of user input
12. Politics placed over
substance
13. Wishful thinking
4
Process-Related Mistakes
14. Overly optimistic schedules
15. Insufficient Risk
Management
16. Contractor failure
Insufficient
planning
17. Abandonment of planning
under pressure
18. Wasted time during the
fuzzy front end
19. Shortchanged upstream
activities
20. Inadequate design
21. Shortchanged quality
assurance
22. Insufficient management
controls
23. Premature or too frequent
convergence
25. Omitting necessary tasks
from estimates
26. Planning to catch up later
27. Code-like-hell programming
Reference: Steve McConnell, Rapid Development
Product-Related Mistakes
28. Requirements gold-plating
29. Feature creep
30. Developer gold-plating
31. Push me, pull me
negotiation
32. Research-oriented
development
Technology-Related Mistakes
33. Silver-bullet syndrome
34. Overestimated savings
from
new tools or methods
35. Switching tools in the
middle
of a project
36. Lack of automated
source-code control
Standish Group survey of
13,000 projects (2003)
• 34% successes
• 15% failures
• 51% overruns
Top Software Engineering Issues in Defense Industry
1. The impact of requirements upon software is not consistently quantified
and managed in development or sustainment
2. Fundamental system engineering decisions are made without full
participation of software engineering
3. Software life-cycle planning and management by acquirers and
suppliers is ineffective
4. The quantity and quality of software engineering expertise is insufficient
to meet the demands of government and the defense industry
5. Traditional software verification techniques are costly and ineffective for
dealing with the scale and complexity of modern systems
6. There is a failure to assure correct, predictable, safe, secure execution
of complex software in distributed environments
7. Inadequate attention is given to total lifecycle issues for COTS/NDI
impacts on lifecycle cost and risk
“Top Software Engineering Issues In Defense Industry”,
NDIA Systems Engineering Division and Software Committee, Sep 2006
5
Many Approaches to Solving the Problem
• Which weaknesses are causing my problems?
• Which strengths may mitigate my problems?
• Which improvement investments offer the best return?
People
Business
Environment
Tools
Process
6
Management
Structure
Product
Methods
Technology
Heritage of Standards for Systems Engineering
2002
2002
ISO/IEC
15504
ISO/IEC
19760
(FDIS)
(PDTR)
1998
1994
EIA / IS
632
1994
1974
1969
Mil-Std499
Mil-Std499A
Mil-Std499B
(Not Released)
EIA
632
(Full Std)
1994
(Trial Use)
ISO/IEC
15288
(FDIS)
1998
(Interim Standard)
IEEE
1220
2002
1998
IEEE
1220
(Full Std)
EIA/IS
731
SE CM
(Interim Standard)
2002
Legend
Supersedes
Source for
7
CMMISE/SW/IPPD
Standards for Systems Engineering, Jerry Lake, 2002
The Frameworks Quagmire
Sarah A. Sheard, Software Productivity Consortium
8
Evolution of the Frameworks Quagmire, Software Magazine
Two Complimentary Approaches
to Process Improvement
Model-Driven (e.g., CMMI)
• Determine the industry best
practice
– Benchmarking, models
• Clarify what your customer wants
(Voice of Customer)
– Critical to Quality (CTQs)
• Compare your current practices to
the model
– Appraisal, education
• Determine what your processes
can do (Voice of Process)
– Statistical Process Control
• Identify and prioritize improvement
opportunities
– Implementation
– Institutionalization
• Identify and prioritize improvement
opportunities
– Causal analysis of data
• Look for ways to optimize the
processes
9
Data-Driven (e.g., Lean Six Sigma)
• Anticipate your customers/
competitors (Voice of Business)
– Design for Six Sigma
Capability Maturity Model Integrated
10
How would you ensure best practices
are implemented consistently in your
software company?
What is the Capability Maturity Model Integrated?
• The CMMI is a collection of industry best-practices for
engineering, services, acquisition, project management,
support, and process management
– Developed under the sponsorship of DoD
– Consistent with DoD and commercial standards
Three Constellations sharing common components and structure
• CMMI for Development - used by engineering organizations
• CMMI for Acquisition - used by buyers (e.g., govt. agencies)
• CMMI for Services - used by service providers (e.g., help desk)
12
Basic Building Blocks – 22 Process Areas
Implemented by
the organization
Implemented by
each project
Project Management
• Project Planning
• Project
Monitoring and
Control
• Supplier
Agreement
Management
• Integrated
Project
Management)
• Risk
Management
• Quantitative
Project
Management
13
Engineering
• Requirements
Development
• Requirements
Management
• Technical
Solution
• Product
Integration
• Verification
• Validation
Support
• Configuration
Management
• Process and
Product Quality
Assurance
• Measurement
and Analysis
• Decision
Analysis and
Resolution
• Causal Analysis
and Resolution
Process
Management
• Organizational
Process Focus
• Organizational
Process
Definition
• Organizational
Training
• Organizational
Process
Performance
• Organizational
Performance
Management
Expected Practices Provide Guidance
for Implementation and Institutionalization
Project Planning – Implementation
SG 1 Establish Estimates
SP 1.1 Estimate the Scope of the Project
SP 1.2 Establish Estimates of Work Product
and Task Attributes
SP 1.3 Define Project Lifecycle Phases
SP 1.4 Estimate Effort and Cost
SG 2 Develop a Project Plan
SP 2.1 Establish the Budget and Schedule
SP 2.2 Identify Project Risks
SP 2.3 Plan Data Management
SP 2.4 Plan the Project’s Resources
SP 2.5 Plan Needed Knowledge and Skills
SP 2.6 Plan Stakeholder Involvement
SP 2.7 Establish the Project Plan
SG 3 Obtain Commitment to the Plan
SP 3.1 Review Plans That Affect the Project
SP 3.2 Reconcile Work and Resource Levels
SP 3.3 Obtain Plan Commitment
14
Project Planning - Institutionalization
GG 2 Institutionalize a Managed Process
GP 2.1 Establish an Organizational Policy
GP 2.2 Plan the Process
GP 2.3 Provide Resources
GP 2.4 Assign Responsibility
GP 2.5 Train People
GP 2.6 Manage Configurations
GP 2.7 Identify and Involve Relevant
Stakeholders
GP 2.8 Monitor and Control the Process
GP 2.9 Objectively Evaluate Adherence
GP 2.10 Review Status with Higher Level
Management
GG 3 Institutionalize a Defined Process
GP 3.1 Establish a Defined Process
GP 3.2 Collect Improvement Information
Practice Ratings for the Organization/Projects
15
How is the CMMI Used for Process
Improvement?
IDEAL
Model
www.sei.cmu.edu/ideal/
16
Typical CMMI Benefits Cited in Literature
• Reduced costs
– 33% decrease in the average
cost to fix a defect (Boeing)
– 20% reduction in unit
software costs (Lockheed
Martin)
• Faster Schedules
– 50% reduction in release
turnaround time (Boeing)
– 60% reduction in re-work
following test (Boeing)
• Greater Productivity
– 25-30% increase in
productivity within 3 years
(Lockheed Martin, Harris,
Siemens)
• Higher Quality
– 50% reduction of software
defects (Lockheed Martin)
• Customer Satisfaction
– 55% increase in award fees
(Lockheed Martin)
17
Lean Six Sigma
18
What is Lean Six Sigma (LSS)?
• Lean Six Sigma is a powerful
approach to improving business
• LSS improvement projects are
performed by teams
• Teams use a set of tools and
techniques to understand
problems and find solutions
• Lean Six Sigma integrates tools
and techniques from two proven
process improvement methods
+
19
DMAIC: Define-Measure-Analyze-Improve-Control
A proven, structured problem-solving approach
for improving existing processes
• Define: Identify the issue causing the
perceived problem or improvement opportunity
• Measure: Collect data from the process
• Analysis: Study the data for cause and effect
• Improve: Eliminate cause(s) and determine resulting effects
• Control: Sustain the gains by monitoring the system
20
DMAIC Toolkit
Define
• Charter
Measure
• Kano Model
• Voice of the
Process
• Voice of the
Customer
• Data Collection
Methods
• Voice of the
Business
• Measurement
System
Evaluation
• Quality
Function
Deployment
Analyze
• Cause & Effect
Diagrams/
Matrix
• Design of
Experiments
• Failure Modes
& Effects
Analysis
• ANOVA
• Statistical
Inference
• Robust Design
• Reliability
Analysis
• Root Cause
Analysis,
including 5
Whys
• Hypothesis Test
21
Improve
• Modeling
• Tolerancing
• Systems
Thinking
• Decision & Risk
Analysis
Control
Statistical Controls
• Control
Charts
• Time Series
methods
Non-Statistical
Controls
• Procedural
adherence
• Performance
Management
• Preventive
measures
Lean - Definition
• Lean is the
philosophy of
improving
performance by
reducing waste and
eliminating
bottlenecks
• Used together, the
two methodologies
become a powerful
set of tools for
improving business
processes
22
"The 8 Wastes - DOWNTIME" by Source
(WP:NFCC#4). Licensed under Fair use
via Wikipedia.
Organizational Adoption: Roles & Responsibilities
• Champions – Facilitate the leadership,
implementation, and deployment
• Sponsors – Provide resources
• Process Owners – Responsible for
the processes being improved
• Master Black Belts – Serve as mentors for Black Belts
• Black Belts – Lead major Six Sigma projects
– Typically requires 4 weeks of training
• Green Belts – Lead minor Six Sigma teams, or serve on
improvement teams under a Black Belt
– Typically requires 2 weeks of training
23
A Typical Lean Six Sigma Project in Aerospace
The organization notes that systems integration has been problematic
on past projects (budget/schedule overruns)
A Six Sigma team is formed to scope the problem, collect data from
past projects, and determine the root cause(s)
The team’s analysis of the historical data indicates that ineffective peer
reviews are leaving significant errors to be found in test
Procedures and criteria for better peer reviews are written, using best
practices from past projects
A pilot project uses the new peer review procedures and criteria, and
collects data to verify they solve the problem
The organization’s standard process and training is modified to
incorporate the procedures and criteria, to prevent similar problems on
future projects
24
Corporate Support
25
Organizational Infrastructure Required
Process Group
Training Program
Measurement Repositories
Predictive Modeling
Best-Practice Libraries
Audits & Appraisals
Defects per component
Policies, Processes,
Templates & Tools
Process Improvement
Communications
25
20
15
UCL
10
_
X
5
0
1
11
21
31
41
51
61
71
Component #
Developing and maintaining mature processes requires
significant time and investment in infrastructure
26
Northrop Grumman Approach:
Institutionalizing Our Improvements
We systematically analyze quality and
process data and trends to determine
how to improve our processes
We improve our process assets
based on internal and external best
practices
Disposition
Analysis
ISO/AS9100
Findings
CMMI Appraisal
Findings
Customer
Comments
Independent
Audits
• Systems/
Software
Engineering
Process Group
• Working Groups
• Program
Management
Advisory Board
Lessons Learned
& Metrics
Tools
27
Increasing
program
efficiency
Industry
Standards
Internal
Best Practices
Configuration
Control Board
Information
Deployed
to
programs
Policy
Process
Six Sigma
Projects
Procedures
External
Best Practices
Checklists and
Guides
Templates and
Examples
msCAS
eToolkit
StartIt!
My MS Portal
PAL
Workbench
PCDB
Lessons Learned
• Multiple improvement initiatives helps encourage a change in behavior as
opposed to “achieving a goal”
– Reinforces that change (improvement) is a way of life
• Benefits result from institutionalizing local improvements across the wider
organization
– CMMI establishes the needed mechanisms
• Training over half the staff as Lean Six Sigma Green Belts resulted in a
change of language and culture
– Voice of Customer, data-driven decisions, causal analysis, etc.
– Better to understand/use tools in everyday work than to adopt the “religion”
• CMMI and Lean Six Sigma compliment each other
– CMMI can yield behaviors without benefits
– Lean Six Sigma improvements based solely on data may miss innovative
improvements (assumes a local optimum)
28