Standards & Assessments
CMMI, ISO 9000, TL9000
Sources: ASQ CSQE Primer
Introduction to CMMI
CMMI Distilled
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
1
August 4 Class
CMMI Introduction &
Configuration Management Appraisal
ISO 9000/TL-9000
Due today (31-July):
Cycle 2 Design & Code, hand off to System Tester
System Test Plan Inspected & Baselined
Project notebook updates including inspection records, meeting
minutes, etc.
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
2
Topics
Audits & Assessments
CMM / CMMI & SCAMPI
ISO 9000: ISO 9001:2000, ISO 9000-3:1997, TickIT
Q9000, TL9000
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
3
Capability Maturity Model (CMM)
Created in 1987 by Software Engineering Institute (SEI)
5 level model based on proficiency in Key Process Areas
(KPAs)
Migrating to Capability Maturity Model Integration (CMMI)
Three source models:
– CMM for Software
– Systems Engineering Capability model
– Integrated Product Development CMM
CMMI v1.1
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
4
What is it?
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
5
Why Would I want one?
Required
– Contractual
– Senior Management Decree (e.g. ROI of 7 to 1)
Sales Tool
Want to improve
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
6
Schedule Example
Drop Page Fields Here
Drop Page Fields Here
Organization 1
Project Schedule Performance
Organization 2
Project Schedule Performance
Count of Months Late
Count of Months Late
4
5
4
Drop Series Fields Here
2
# Projects
# Projects
3
3
Drop Series Fields Here
2
1
1
0
0
1
2
3
4
5
7
4
Months Late
7/31/2007
5
6
Months Late
SE 652 2007_7_31_CMMI_Software_Quality.ppt
7
Process Capability
Ability of a process to produce planned results
• Predictable
• Measureable
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
8
Process Models
CMMI is model based
Model = structured collection of elements that describes characteristics of
effective processes
Process Area = cluster of related practices that when performed collectively,
satisfy a set of goals considered important for making significant
improvement in that area
Processes selected are those proven by experience to be effective (i.e. best
practices, practical knowledge from previous endeavors)
Notes:
A process area is not a process
A model is not a process
models show what to do, not how to do it!
Philosophy
“All models are wrong, some are useful” – George Box
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
9
CMMI Models
Model Options:
Software Engineering (SW)
Systems Engineering + Software Engineering (SE/SW)
Systems Engineering + Software Engineering + Integrated Process & Product
Development (SE/SW/IPPD)
… + Supplier Sourcing (SE/SW/IPPD/SS)
Representation Options:
Staged (Maturity Levels)
Migration from CMM to CMMI
Continuous (Capability Levels)
Migration from EIA/IS-731 to CMMI
Recommended order for process improvements, but not prescribed …
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
10
Levels
Zero – Ad Hoc
One – Doing it (in Continuous, Ad Hoc in Staged)
Two – Process performed for individual projects
Three – Process focus at organizational level
Four – Projects and processes are quantitatively managed
Five – Projects and processes being optimized based on performance data
& results
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
11
Representations Revisited
Continuous Model
– 25 Process Areas each assessed at level 0-5
Configuration Mgmt = capability level 3
Risk Mgmt = not done (capability level 0)
Requirements Mgmt = capability level 2
– Result can be presented as a Kiviat chart
Staged Model
– 25 Process Areas assigned to each of 4 Maturity Levels (see next slide)
– Result is a grade (1-5)
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
12
Staged Representation
Maturity Levels (MLx)
5
Optimizing
Focus on process improvement
4
Quantitatively Managed
Process measured & controlled
3
Defined
Process characterized by organization is proactive
2
Managed
Process characterized for project & often reactive
1
Initial
Process unpredictable, poorly controlled & reactive
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
13
Staged Representation
Process Area Mapping to Maturity Levels
5. Optimizing
Continuous Process
Improvement
Organizational Innovation & Deployment
Causal Analysis & Resolution
4. Quantitatively
Managed
Quantitative
Management
Organizational Process Performance
Quantitative Project Management
3. Defined
Process
Standardization
2. Managed
Basic Project
Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Organizational Process Focus
Organizational Process Definition
Organizational Training
Risk Management
Decision Analysis & Resolution
Requirements Management
Project Planning
Project Monitoring & Control
Supplier Agreement Management
Measurement & Analysis
Process & Product Quality Assurance
Configuration Management
1. Initial
7/31/2007
None
SE 652 2007_7_31_CMMI_Software_Quality.ppt
15
Continuous Representation
Process Areas
Engineering
Process Management
–
–
–
–
–
Organizational Process Focus (OPF-3)
Organizational Process Definition (OPD-3)
Organizational Training (OT-3)
Organizational Process Performance (OPP-4)
Organizational Innovation & Deployment
(OID-5)
Project Planning (PP-2)
Project Monitoring & Control (PMC-2)
Supplier Agreement Management (SAM-2)
Integrated Project Management (IPM-3)
Risk Management (RSKM-3)
Integrated Teaming (IT-3)
Integrated Supplier Management (ISM-3)
Quantitative Project Management (QPM-4)
7/31/2007
Requirements Management (REQM-2)
Requirements Development (RD-3)
Technical Solution (TS-3)
Product Integration (PI-3)
Verification (VER-3)
Validation (VAL-3)
Support
Project Management
–
–
–
–
–
–
–
–
–
–
–
–
–
–
– Configuration Management (CM-2)
– Process & Product Quality Assurance
(PPQA – 2)
– Measurement and Analysis (MA-2)
– Decision Analysis and Resolution (DAR-3)
– Organizational Environment for Integration
(OEI-3)
– Causal Analysis and Resolution (CAR-5)
SE 652 2007_7_31_CMMI_Software_Quality.ppt
16
CMMI Assessment Cheat Sheet
Institutionalization – Ingrained Way of Doing Business that an organization follows routinely
as part of its corporate culture
Specific Goals – Required model component that describes the unique characteristics that must
be present to satisfy the process area
Specific Practice – Expected model component that is considered important to achieving the
associated specific goal. The specific practices describe the activities expected to result in
achievement of the specific goals of a process area. (In continuous representation – every
specific practice (SP) is associated with a CL, in staged – all SPs are treated equally)
Generic Goal – Required model component that describes the characteristics that must be
present to satisfy the institutionalization of the processes that implement a process area
Generic Practice – Expected model component that is considered important in achieving the
associated generic goal. The generic practices describe the activities that are expected to
result in achievement of the generic goal and contribute to the institutionalization of the
processes associated with a process area.
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
17
CMMI Assessment Cheat Sheet (continued)
Managed Process:
–
–
–
–
–
–
–
Performed process planned & executed in accordance with policy
Employs skilled people
Adequate resources
Produces controlled outputs
Involves relevant stake holders
Monitored, controlled & reviewed
Evaluated for adherence to process description
Defined Process:
–
–
–
Managed process tailored from the organizational standard processes
Maintained process description
Contributes work products, measures & other process info to organizational process assets
Performed Process
– Accomplishes needed work to produce work products
– Specific goals of the process area are satisfied
Establish & Maintain
– Includes documentation & usage:
• Planned
• Documented &
• Used
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
18
Configuration Management (CM)
Assessment
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
19
DeMarco & Lister on Process
Organizations driving to be SEI Level 5 (at least level N+1)
Standards are good, but …
Most success centered around standard interfaces
Mandating a “best practice” is a bad practice
Process improvement is good, but process improvement programs aren’t
Competent people improve processes all the time (pride, growth, etc.)
Formal process improvement moves responsibility from the individual to the organization
Process improvement programs focus on process rather than product
(making a poor product efficiently is often worse than making a good product poorly)
Focus on process “level” tends to make organizations risk averse
“The projects most worth doing are the ones that will move you down one full level on
your process scale!”
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
20
Break 
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
21
Quality Standard Rationale
Customers want & need assessments of supplier quality
Means:
Individually audit (i.e. qualify) vendor:
Specific products
Processes (e.g. manufacturing, design & development, support)
Alternative:
Common Quality Assurance standards & audits
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
22
Major Audit Types
First Party Audit
Within own company (aka internal audit)
Used to measure own performance, strengths & weaknesses against internally
established procedures & systems
Second Party Audit
Performed by customer on their supplier (aka external audit)
Third Party Audit
Outside, independent auditor contracted to audit on behalf of company or a
supplier (e.g. ISO 9000 registration audit)
Assessments (e.g. SCAMPI)
Similar to first party audit, but typically performed by external assessors
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
23
Other Audit Types
System Audit – examination of bigger picture of organization &/or project
Typical cross organizational, cross process & cross product
Process Audit – verify inputs, actions & outputs in accordance with
defined requirements (e.g. software inspections)
Product Audit – final product or service for “fitness for use”
Customer oriented
Compliance Audit
Regulatory – audit to government regulations
Management – audit to organizational rules, effectiveness & conformance
Quality – systematic & independent of quality activities vs. established
procedures
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
24
ISO 9001:2000
Objective
Provide confidence that vendor can produce quality products
Assumptions: good practices will produce good products
Standard for assessing organization’s Quality Management System (QMS)
– Processes
– Activities
– Behaviors
– Training
But, ISO focuses on Quality Assurance not Quality Control
ISO-9001 certification does not guarantee quality products!
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
25
Tenants of ISO 9001
1) Say what you do
2) Do what you say
3) Prove it!
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
26
ISO 9000 Audits
Customers write requirements for current ISO-9001
certification into purchasing contracts
Organizations apply for 3rd party audit,
end result is ISO-9001 certification
ISO International Accreditation Forum (IAF) board
Audits national accreditation boards (i.e. one board each nation)
Who register individual registrars (e.g. Lloyd’s, DNV)
Who audit organization internal auditors (e.g. Lucent Optical
Networking) & spot check
Who audit design, development, manufacturing & support
teams within the organization
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
27
ISO Alphabet Soup
ISO 9000:2000
Overall framework, fundamentals of quality management systems &
terminology
ISO 9001:2000
Requirements for quality management systems (qms) & what is required to
demonstrate compliance
ISO 90003 2004 (previously 9000-3)
Guidelines for the application of ISO 9001:2000 to computer software
ISO 19011
Guidelines for auditing quality and environmental management systems
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
28
What is wrong with ISO 9001?
Vendors ISO-9001 certified, but quality still elusive!
No visibility into supplier quality levels
Not getting quality levels they wanted
Solution:
TL9000 (Quest forum, telecommunications)
QS9000 (automotive)
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
29
TL9000
ISO on steroids
Wholly subsumes ISO 9001-2000
Requires vendors prove they are actually improving
Metrics focused on cost drivers of service providers:
Know vendor is measuring
Visibility into quality improvement results
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
30
TL9000 Top Management Requirements
Monitor & improve customer satisfaction
Set long & short term objectives for organization effectiveness
Set targets for TL9000 product performance metrics
Use an explicit life-cycle model
Establish a quality improvement program
Periodic management review of quality system
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
31
TL9000 Metrics
Cross-discipline metrics
–
–
–
–
# of problem reports
Problem report fix response time
Overdue problem report fix responsiveness
On-time delivery
Hardware & Software measurements
– System Outages
Hardware measurements
– Return rates
Software measurements
–
–
–
–
7/31/2007
Software installation & release application aborts
Corrective patch quality
Feature patch quality
Software update quality
SE 652 2007_7_31_CMMI_Software_Quality.ppt
32
TL9000 Common Audit Questions
•
Do you know how to find your Quality Policy, QMS and the processes you
should be using for your work?
•
Do you know your organization’s product delivery & improvement goals and
what you must do to support them?
•
Do you know what skills you should have?
•
Do you know what you have to do to approve/baseline/finalize your
documents, designs & code?
•
Do you know how to store & find records of reviews, inspections, key
decisions, etc.?
•
Do you know what to do if a problem is found with the product or process?
•
Do you know your organization’s performance with respect to customer
satisfaction, quality of delivered products & process execution?
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
33
TL9000 Sample Requirements
Planning
– Must have methods for estimating & tracking
– Determine where you will do reviews & tests
– Risk management plans, customer, user & supplier involvement in reviews &
evaluation
Software Outputs
– Requires architecture, detailed designs, code & user documentation
– Each design thread must be reviewed at some point prior to integration or system
test
Software Testing
– All testing must have test plans; test process must be documented
– Plans must include test cases with inputs, output & test success criteria
– Plans must include types of testing, requirements traceability, coverage definition
& measurement, test environment, defect handling, et.al.
– Integration testing specifically required
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
34
Team Project Postmortem
Tracking process improvements during project
Process Improvement Proposals (PIP)
Port-Mortem
Areas to consider
Better personal practices
Improved tools
Process changes
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
35
Postmortem process
Team discussion of project data
Review & critique of roles
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
36
Postmortem process
Review Process Data
Review of cycle data including SUMP & SUMQ forms
Examine data on team & team member activities & accomplishments
Identify where process worked & where it didn’t
Quality Review
Analysis of team’s defect data
Actual performance vs. plan
Lessons learned
Opportunities for improvement
Problems to be corrected in future
PIP forms for all improvement suggestions
Role Evaluations
What worked?
Problems?
Improvement areas?
Improvement goals for next cycle / project?
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
37
Cycle Report
Table of contents
Summary
Role Reports
Leadership – leadership perspective
Motivational & commitment issues, meeting facilitation, req’d instructor support
Development
Effectiveness of development strategy, design & implementation issues
Planning
Team’s performance vs. plan, improvements to planning process
Quality / Process
Process discipline, adherence, documentation, PIPs & analysis, inspections
Cross-team system testing planning & execution
Support
Facilities, CM & Change Control, change activity data & change handling, ITL
Engineer Reports – individual assessments
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
38
Role Evaluations & Peer Forms
Consider & fill out PEER forms
Ratings (1-5) on work, team & project performance, roles & team members
Additional role evaluations suggestions
Constructive feedback
Discuss behaviors or product, not person
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
39
Project Notebook
Updated Requirements & Design documents
Conceptual Design, SRS, SDS, System Test Plan, User Documentation*
Updated Process descriptions
Baseline processes, continuous process improvement, CM
Tracking forms
ITL, LOGD, Inspection forms, LOGTEST
Planning & actual performance
Team Task, Schedule, SUMP, SUMQ, SUMS, SUMTASK, CCR*
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
40
August 4 Class
CMMI Introduction &
Configuration Management Appraisal
ISO 9000/TL-9000
Due July 31:
Cycle 2 Design & Code, hand off to System Tester
System Test Plan Inspected & Baselined
Project notebook updates including inspection records, meeting minutes, etc.
Deliverables for August 7
Project Postmortem (cycle report)
Cycle 2 presentations
Peer Feedback forms
Completed project notebooks
Cycle Exit
Completed project (source, documents & all quality records)
7/31/2007
SE 652 2007_7_31_CMMI_Software_Quality.ppt
41