Software Engineering
Management
Course # CISH-6050
Lecture 2:
Software Process Maturity
Convener:
Houman Younessi
05/14/2012
AGENDA
• Software Process Maturity
- Need for Process Maturity?
- Maturity Concepts and Origins
Software Maturity Models
- Frameworks Quagmire
- Standards vs. Capability Maturity
Models
2
CISH-6050 - Software Engineering Management
AGENDA …
• Standards
-
Software
Systems Engineering
Quality
Measurements
 Six Sigma
SPICE
In Class Team Assignment
3
CISH-6050 - Software Engineering Management
Software Process Maturity
What is it?
Why do it?
4
CISH-6050 - Software Engineering Management
Software Process Maturity …
• A need for Process Maturity, from
SW-CMM V1.1
- Two decades of unfulfilled promises
about productivity & quality gains from
applying new software methodologies &
technologies
- Industry & government organizations
realize fundamental problem is inability
to manage the software process
5
CISH-6050 - Software Engineering Management
Software Process Maturity …
- Even in undisciplined organizations, some
individual software projects produce
excellent results
- When such projects succeed, generally
through the heroic efforts of a dedicated
team vs. through repeating the proven
methods of an organization with a mature
software process
6
CISH-6050 - Software Engineering Management
Software Process Maturity …
- Success relying on specific individuals
provides no basis for long-term
productivity & quality improvement
throughout an organization
- Continuous improvement occurs
through focused & sustained effort in
building a process infrastructure of
effective software engineering and
management practices
7
CISH-6050 - Software Engineering Management
Software Process Maturity …
• Characteristics of an Immature
Software Organization
-
Software process improvised
Software process not always followed
Reactionary vs. planned
Fire fighting mode
Quality & functionality compromised to make
schedule
- Difficult to predict product quality
- Schedules and budgets routinely exceeded
8
CISH-6050 - Software Engineering Management
Software Process Maturity …
• Characteristics of a Mature Software
Organization
- Organization-wide processes communicated
to all staff
- Work done and tracked according to
planned process
- Process improvement where necessary
- Project quality managed & monitored
- Schedules & budgets are realistic and
usually achieved
9
CISH-6050 - Software Engineering Management
Process Maturity Concepts
• Software Process Capability
- Range of expected results that can be
achieved by following a software process
- Software process capability of an
organization provides one means of
predicting the most likely outcome to be
expected from the next software project the
organization undertakes
10
CISH-6050 - Software Engineering Management
Process Maturity Concepts …
• Software Process Performance
- Represents actual results achieved by
following a software process
- Based on the attributes of a specific project
and the context within which it is conducted,
the actual performance of the project may
not reflect the full process capability of the
organization
11
CISH-6050 - Software Engineering Management
Process Maturity Concepts …
• Software Process Maturity
- Extent to which a specific process is
explicitly defined, managed, measured,
controlled, and effective
- Implies potential for growth in capability
- Indicates the richness of an organization's
software process & the consistency with
which it is applied in projects throughout
the organization
12
CISH-6050 - Software Engineering Management
Process Maturity Concepts …
• Institutionalization
- Building infrastructure & corporate culture
that supports the methods, practices, and
procedures of the business so they
endure after those who originally defined
them have gone
13
CISH-6050 - Software Engineering Management
Process Maturity
How Process Maturity Started
14
CISH-6050 - Software Engineering Management
Process Maturity
• Start of Process Maturity
- Software process management objectives
include producing products according to a
plan, while simultaneously improving the
organization to produce better products
- These are also the basic principles of
Statistical Process Control (SPC)
- A process is under statistical process
control if its future performance is
predictable within established statistical
limits
15
CISH-6050 - Software Engineering Management
Process Maturity …
• W. E. Deming
- Introduce the concept of SPC
- Worked with Japanese industry after
World War 2 to improve quality
- Japanese industry committed to
continuous process improvement for
many years
- SPC contributed largely to the quality of
Japanese manufactured goods
16
CISH-6050 - Software Engineering Management
Process Maturity …
• SPC Methodology
- Basis - measuring product defects and
relating these defects to the process
- Process is improved with goal of
reducing the number of defects
- Process is improved until it is repeatable
- Process is then standardized and further
improvement cycles begin
17
CISH-6050 - Software Engineering Management
Process Maturity …
• SPC Application
- With manufacturing, process and product
relationship is very obvious
 Improving process to remove defects
in the manufacturing process will
produce a better product
- Relationship less obvious with intangible
products like software
 Dependent on an intellectual process
which can not be automated
18
CISH-6050 - Software Engineering Management
Process Maturity …
Watts Humphrey on SPC:
"While there are important differences,
these [SPC] concepts are just as
applicable to software as they are to
producing consumer goods like cameras,
television sets, or automobiles."
19
CISH-6050 - Software Engineering Management
Intellectual Product Quality
With Intellectual products like software,
where quality is principally dependent on
design, four factors affect product quality
Development Technology
Process Quality
Product Quality
People Quality
Cost, Time, Schedule
20
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
• Example: Very large systems made
up of subsystems, which are
developed by different teams
- Factor determining product quality will
probably be the software process
- Why Process Quality factor rather than
the other 3 factors?
21
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
• Example: For smaller teams, with
only few team members
- The quality of the development team (i.e.
people quality factor) is more important
than the development process used.
- Why People Quality factor rather than the
other 3 factors?
22
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
• Product Quality Factors affecting
productivity
- Smaller teams
 Development Technology
 Developers spend good portion of
time in development activities
 Good development tools affect
productivity
23
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
- Larger teams
 Software Process
 Members spend less time developing
& more time communicating and
understanding other parts of the
system
 Basic level of development
technology essential for information
management
24
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
• For all projects, regardless of size,
Cost, Time, and Schedule are key
quality factors
-
Under budgeted
Unrealistic schedules
Insufficient resource
Inadequate resource
25
CISH-6050 - Software Engineering Management
Intellectual Product Quality …
• Process improvement to become a
mature development organization
-
Assess the organization's current process
Develop goal of desired process capability
Determine where to improve
Produce a plan to accomplish the actions
Commit resources and funding for the plan
Make improvements
Repeat the improvement process
26
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
Thoughts on the
Frameworks Quagmire?
27
CISH-6050 - Software Engineering Management
Source: S. A. Sheard, "Evolution of the Frameworks
Quagmire", IEEE Computer, July 2001, p. 97.
28
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
• Elements of the Frameworks
Quagmire
-
Software Capability Maturity Models
Systems Engineering Maturity Models
Software Standards
Systems Engineering Standards
Measurements Standards
Quality Standards
29
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
• Three ways to create a standard:
1. De facto – free interplay of market forces;
example: MS Windows
2. Government body – example: DoD
standards for the US
3. Voluntary consensus – established by
standard setting organization; example:
ISO, ANSI, IEEE
30
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
• Voluntary consensus organizations:
- ISO - International Standards Organization:
 Non gov’t org established in 1947
 Established to improve international
communication and trade
 160 Technical committees
- ANSI - American National Standards
Institute:
 Represents US in ISO
 Carries out standards activities
31
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
• Voluntary consensus organizations:
- IEEE – Institute of Electrical and
Electronics Engineers:
 Professional society
 Active with Telecom and Computer
standards activities
 300,000 members world wide
 530+ currently active standards
32
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
• Standards vs Capability Models
- Both describe good systems/software
engineering, but differently:
 Standards must go through a
defined industry approval process
to meet a nation's guidelines, such
as those set by ANSI
 Capability models can be created
by anyone with resources
33
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
- What not how:
 Both focus on processes and related
activities (what), not on methods or
tools (how)
- Purpose:
 Capability Models provide a way to
evaluate systems or SE capability
 US Military standards originally
supported contracts to aid the
government in utilization of consistent
processes by contractors
34
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
- Lifecycle:
 Capability models don’t prescribe a
life cycle, but rather applies to any
system life-cycle
 Standards may prescribe a life-cycle
- Number of process elements
 Capability models have 18 or 19
 Most standards have less than 10
35
CISH-6050 - Software Engineering Management
The Frameworks Quagmire …
• Standards and Models distinction
can blur:
- EIA/IS 731 is SE Capability Model
submitted as standard
- Model heavily tied to existing standards
EIA 632 and IEEE 1220
- Two consistent SE standards
 EIA 632 – What to do in SE systems
 EIA/IS 731 – How to measure and
improve the SE systems capability
36
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
Software Standards
37
CISH-6050 - Software Engineering Management
Quagmire: Software Standards
• MIL-STD-498 Created in 1994 by US
Defense Department
to integrate:
1. DOD-STD-2167A Software
development
2. DOD-STD-2168 Software quality
3. DOD-STD-7935A Documentation
requirements
38
CISH-6050 - Software Engineering Management
Quagmire: Software Standards …
• J-STD-016 Demilitarized version
of MIL-STD-498:
1. Released by joint
IEEE and (EIA)
committee
2. Based on MILSTD-498
3. Limited changes
39
CISH-6050 - Software Engineering Management
Quagmire: Software Standards …
• ISO/IEC 12207 Jointly released in
1995 by ISO and IEC
1. Standard for
Information
Technology
software life-cycle
processes
40
CISH-6050 - Software Engineering Management
Quagmire: Software Standards …
• IEEE/IEA 12207 Jointly created by
IEEE and EIA work
group
1. Based on ISO/IEC
12207
2. Supersedes
MIL-STD-498 and
J-STD-016
41
CISH-6050 - Software Engineering Management
Quagmire: Software Standards …
• RTCA DO -178B Software
Considerations in
Airborne Systems and
Equipment Cert.
1. Addresses aviation
software systems
safety
2. Developed
independently of
other frameworks
42
CISH-6050 - Software Engineering Management
Quagmire: Software Standards …
• Summary – Four
Software Standards in
Quagmire:
1. IEEE/EIA 12207
2. ISO/IEC 12207
3. RTCA DO-178B
4. SPICE (To be
discussed)
43
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
System Engineering
Standards
44
CISH-6050 - Software Engineering Management
Quagmire: SE Standards
• MIL-STD-499B - Submitted in May, 1992 by
the Department Of Defense:
1. Total systems approach for developing
defense systems
2. Industry never accepted these standards
3. DOD Cancelled in 1993
45
CISH-6050 - Software Engineering Management
Quagmire: SE Standards …
• EIA/IS 632 - Approved as an Interim
Standard (IS) in December, 1994 :
1. Total systems approach for developing
defense systems
2. EIA sponsored a small group to make
minor wording changes to MIL-STD-499B
46
CISH-6050 - Software Engineering Management
Quagmire: SE Standards …
• IEEE 1220 - Released as trial-use standard
in 1994 & re-released as full standard in 1998 :
1. Created at the same time as EIA/IS 632
2. Contains a more commercial life-cycle and
less military terminology
47
CISH-6050 - Software Engineering Management
Quagmire: SE Standards …
• EIA 632 - Released in January, 1999:
1. Bears little resemblance to the EIA/IS 632
2. Defines 13 processes and 34
requirements for engineering a system
3. Can be applied to any enterprise based
life cycle phase to engineer system
48
CISH-6050 - Software Engineering Management
Quagmire: SE Standards …
• ISO/IEC 15288 –
1. Addresses both systems engineering and
management (business) processes
2. Focuses on “system” vs. “component”
49
CISH-6050 - Software Engineering Management
Quagmire: SE Standards …
• Summary – 5 Systems Engineering
standards in quagmire:
- MIL-STD 499B, EIA/IS 632, IEEE 1220,
EIA 632, ISO/IEC 15288
50
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
Quality Standards
51
CISH-6050 - Software Engineering Management
Quagmire: Quality Standards
• Malcom Baldridge
National Quality
Award:
- Established in
1987
- Quality standard
and way of
comparing org’s
quality practices
- Practices in seven
categories
52
CISH-6050 - Software Engineering Management
Quagmire: Quality Standards …
• ISO 9000 Series –
Set of standards and
guidelines:
- Released in 1994
- Originally
developed for
manufacturing
environment
- Not written for a
specific industry
or business
53
CISH-6050 - Software Engineering Management
Quagmire: Quality Standards …
• ISO 9000 Series:
- Q9000 –
American Society
for Quality’s
version of ISO
9000
- TL9000 –
Telecom’s
adaptation of ISO
9000
54
CISH-6050 - Software Engineering Management
Quagmire: Quality Standards …
• ISO 9000 Benefits:
-
Product Registration
Establishes foundation of quality
Increased productivity
Consistency
Improved competitiveness
Customer demand
Corporate image
55
CISH-6050 - Software Engineering Management
Quagmire: Quality Standards …
• ISO 9000 for Software Development:
- ISO 9000 – Quality systems
- ISO 9001 – QA in design/dev, production,
installation, servicing
- ISO 9002 – QA in production & installation
- ISO 9003 – QA in final inspection & test
- ISO 9004 – Quality management and
systems elements – provides guidance on
ISO 9001 - 9003
56
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
Measurement Standards
57
CISH-6050 - Software Engineering Management
Quagmire: Measurement Standards
• Practical Systems
& Software
Measurements
(PSM)
• ISO/IEC 15939
• Six Sigma
58
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma
• Example Six Sigma Benefits:
- Raytheon Aircraft saved $500K from a
single project
- Textron saved $5M in six months
- Fidelity Wide Processing expects to deliver
$6M - $8M in cost reductions in 2003
- In 1997, Six Sigma delivered more than
$300M to GE operating income
- In 1998, financial benefits of Six Sigma at
GE doubled to over $600M
59
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• What is Six Sigma?
- Started in manufacturing in Motorola in the
1980’s
- Defect reduction methodology
- Focuses on quality of the customer
experience
- Improves profitability
- Each sigma is a deviation from the ideal
level of operation
- Fewer defects as sigma level increases
60
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
Six Sigma and cost of quality:
Sigma
%
Accurate
Defects Per
Million Output
Cost of
Quality
1
31
690,000
N/A
2
69.1
308,537
N/A
3
93.32
66,807
4
99.379
6,210
25% - 40% of
Sales
15% - 25% of
Sales
(Industry Avg)
5
99.9767
233
6
99.99966
3.4
61
5% - 15% of
Sales
<1% of Sales
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• Is 99% Accurate close enough?
- 117,000 pieces of first class mail lost per
hour
- 800,000 mishandled personal checks each
day
- 23,087 lemons among new computers
shipped each month
- 7.2 hours per month without electricity
62
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• Applying Six Sigma Breakthrough
Strategy – 8 stages:
-
Recognize
Define
Measure
Analyze
Improve
Control
Standardize
Integrate
63
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• The 8 stages are applied across the
organization on three levels:
1. Business Level
2. Operations Level
3. Process Level
• Example: Recognize …
- Business – true state of business
- Operations – operational issues that link to
business systems
- Process – functional problems that link to
operational issues
64
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• Six Sigma Players:
- Champions – Exec. Leader champions Six
Sigma across orgs & ensures all connected
- Master Black Belts – In house experts for
disseminating Breakthrough Strategy
knowledge throughout organization
- Black Belts – Apply Six Sigma Breakthrough
Strategy tools and knowledge to projects
- Green Belts – Employees who execute Six
Sigma as part of their overall job
65
CISH-6050 - Software Engineering Management
Quagmire: Msmt Stds – Six Sigma …
• Is Six Sigma for everyone?
- Metrics/Measurements are key and must be
properly utilized by the organization
- Smaller companies may take longer to realize
benefits of Six Sigma, because it might take
them longer to produce 1M products
- If organization has had extensive IT cuts,
might be difficult to fully embrace Six Sigma
and effectively use it
- Need executive championship, top down
66
CISH-6050 - Software Engineering Management
The Frameworks Quagmire
Software Standard:
SPICE
67
CISH-6050 - Software Engineering Management
Quagmire: SPICE
• ISO/IEC TR 15504
SPICE:
- Software Process
Improvement
Capability
dEtermination
(SPICE)
- Standard for
Software Process
Assessment
68
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• First goal achieved in June, 1995 with
release of version 1:
- Draft International Standard for software
process assessment
- Worked to revise technical report to be
ultimate publication as full standard
- Evolved to include process evaluations
- SPICE has reference model that can be
replaced by models like CMM
69
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• SPICE Trials:
- Empirical trials designed to validate the
developing standard – coverage, usability,
applicability
- Provide evidence of benefits of standard
- Trials conducted in 3 phases
 Design and usability testing of core
 Product integration & repeatability testing
 Validation of overall SPICE goals
70
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• Benefits for Participants in the SPICE
Trials:
-
Influence the emerging standard
Customized benchmarking
Early access to SPICE trial results
Support in conducting assessments
71
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• SPICE Trial Phase 1:
- Goal: Validate initial working drafts in
SPICE project
- Tested key components of SPICE
document set
- 35 trials conducted in Europe, Canada,
and Pacific Rim
- Recommendations to product set prior to
starting Phase 2 of trials
72
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• SPICE Trial Phase 2:
- Commenced in September, 1996
- Provided further experience feedback on
using SPICE
- Several hundred assessments conducted
- Focus on evaluating usability, repeatability,
portability of framework
- Results finalized in October, 1998
- Recommendations made
73
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
SPICE Process Assessment
74
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
SPICE Documentation Suite
75
CISH-6050 - Software Engineering Management
Quagmire: SPICE …
• SPICE Reference Model:
- Two dimensional reference model defined
by SPICE:
1. Process dimension (base practice)
which includes essential activities of a
specific process
2. Process Capability dimension
(generic practice) which is
characterized by attributes applicable
to any process; needed to improve
capability
76
CISH-6050 - Software Engineering Management
SPICE Reference Model
• 5 Base Practices:
1. Customer-Supplier (CUS) - processes
that directly impact the customer
2. Engineering (ENG) - processes that
directly affect software product
3. Project (PRO) - processes which
establish the project
4. Support (SUP) - processes which enable
& support perf. of other processes
5. Organization (ORG) - processes which
establish business goals of the org
77
CISH-6050 - Software Engineering Management
SPICE Reference Model …
• 5 Process Categories contain 35
Process Areas and 201 Base
Practices:
1. CUS – 8 process areas; 39 base practices
2. ENG - 7 process areas; 32 base practices
3. PRO – 8 process areas; 50 base practices
4. SUP – 5 process areas; 32 base practices
5. ORG – 7 process areas; 48 base practices
78
CISH-6050 - Software Engineering Management
SPICE Reference Model …
• Process Category Nomenclature:
- PC.PR.PT
 PC = 3 character Process Category
 PR = Process Number within Process
Category
 PT = Base Practice within Process
Category
• Example: ENG.3.1 = Engineering, process 3
(develop software design), base practice 1
(develop architecture design)
79
CISH-6050 - Software Engineering Management
SPICE Reference Model …
• Six Capabilities in SPICE Model:
- Level 0 – Not Performed
 No identifiable work products
- Level 1 – Performed Informally
 Performance not rigorously planned &
tracked
 Performance depends on individual
 Identifiable work products
80
CISH-6050 - Software Engineering Management
SPICE Reference Model …
- Level 2 – Planned and Tracked
 Performance according to specified
procedure is verified
 Work products conform to specified
standards and requirements
- Level 3 – Well Defined
 Base practices performed according to
well defined process using approved,
tailored version of standard
documented process
81
CISH-6050 - Software Engineering Management
SPICE Reference Model …
- Level 4 – Quantitatively controlled
 Detailed measures of performance are
collected and analyzed
 Quantitatively understood & managed
- Level 5 – Continuously Improving
 Quantitative process effectiveness and
performance efficiency goals are
established
 Based on organization business goals
82
CISH-6050 - Software Engineering Management
SPICE Reference Model …
• Capability levels have 13 Common
Features (CF) and 25 Generic
Practices (GP) applicable to all
process categories:
-
Level 1 – 1 CF
Level 2 – 4 CF; 12 GP
Level 3 – 2 CF; 5 GP
Level 4 – 4 CF; 3 GP
Level 5 – 2 CF; 5 GP
83
CISH-6050 - Software Engineering Management
SPICE Reference Model …
• Capability Level Nomenclature:
- CL.CF.PT
 CL = Capability Level Number
 CF = Common Feature Number within
the capability level
 PT = Generic Practice within Common
Feature
• Example: 2.3.1 = Capability Level 2 (Planned
and Tracked), Common Feature 3 (Verify
Performance), Generic Practice 1 (Verify
Process Compliance)
84
CISH-6050 - Software Engineering Management
SPICE Reference Model …
SPICE Model Architecture – Process & Capability
85
CISH-6050 - Software Engineering Management
SPICE Reference Model …
Combining Process & Capability Architectures
86
CISH-6050 - Software Engineering Management
Lecture 2 Recap: Process Maturity
• Software Process Maturity
• Software Maturity Models
- Frameworks Quagmire
• Standards
- Software, Systems Engineering,
Quality, Measurements
- Six Sigma
• SPICE
• In Class Team Assignment
87
CISH-6050 - Software Engineering Management
References
• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability
Maturity Model for Software", Version 1.1, Software
Engineering Institute, Carnegie Mellon University, February,
1993. Available at
http://www.sei.cmu.edu/publications/documents/93.reports/93.
tr.024.html
• ISO/IEC TR 15504:1998 - Software Process Assessment
(SPICE) documentation suite. Available at
http://www.sqi.gu.edu.au/spice/suite/download.html
• S. A. Sheard, "Evolution of the Frameworks Quagmire", IEEE
Computer, July 2001, pp. 96-98. Available at
http://www.software.org/pub/externalpapers/Evolution_Frame
works_Quagmire.pdf
88
CISH-6050 - Software Engineering Management
References
• S. A. Sheard, J. G. Lake, "Systems Engineering Standards
and Models Compared", Proceedings of the International
Council on Systems Engineering, Vancouver, British,
Columbia, 1998. Available at
http://www.software.org/pub/externalpapers/9804-2.html
• W. S. Humphrey, Managing the Software Process, AddisonWesley, Reading, MA, 1989
• I. Sommerville, Software Engineering, 5th ed., AddisonWesley, Reading, MA, 1995
89
CISH-6050 - Software Engineering Management
References
• D. B. Simmons, N. C. Ellis, H. Fujihara, W. Kuo, Software
Measurement: A Visualization Toolkit for Project Control and
Process Improvement, Prentice Hall PTR, Upper Saddle
River, NJ, 07458, 1998
• M. Harry, R. Schroeder, Six Sigma, Doubleday, New York,
NY, 10036, 2000
• T. Mayor, “Targeting Perfection”, CIO Magazine, Vol. 17, No.
5, December 1, 2003, pp. 62-70. Also available at
http://www.cio.com/archive/120103/sigma.html
90
CISH-6050 - Software Engineering Management