Software Engineering
Management
Course # CISH-6050
Lecture 7:
Software Process Assessment
Convener:
Houman Younessi
1
07/09/2012
AGENDA …
• Process Assessment
-
SEI SW-CMM Improvement Approach
Do organizational assessments pay off?
Whistleblowers
Reference: SPICE Assessment
Documents
2
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment
• SEI SW-CMM Improvement
Approach: IDEAL
-
Initiating
Diagnosing
Establishing
Acting
Leveraging
• Diagnose process, establish
priorities, & act on them
3
CISH-6050 - Software Engineering Management
SW-CMM: IDEAL
• Initiating:
- Stimulus for improvement
- Set context & sponsorship
- Establish improvement infrastructure
• Diagnosing:
- Appraise & characterize current practice
- Develop recommendations
- Document phase results
4
CISH-6050 - Software Engineering Management
SW-CMM: IDEAL …
• Establishing:
- Set strategy & priorities
- Establish process action items
- Plan actions
• Acting:
- Define processes and measures
- Plan & execute pilots
- Plan, execute, & track installation
5
CISH-6050 - Software Engineering Management
SW-CMM: IDEAL …
• Leveraging:
- Document & analyze lessons
- Revise organizational approach
6
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment
• General Classes of SW-CMM
Appraisal
1. Software Process Assessment
 Determine state of organization’s
software process
2. Software Capability Evaluations
 Identify contractors qualified to
perform software work
7
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Software Process Assessment:
- Identify improvement priorities within
organization
- Assessment team uses CMM to guide
identifying & prioritizing findings
- Findings & KPA guidance used to plan
improvement strategy for organization
8
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Software Capability Evaluations:
- Identify risks associated with a project
or contract to build high quality on
schedule & budget
- During acquisition process, capability
evaluation may be performed on
bidders
- Findings of an evaluation may be used
to identify risk with using a contractor
- Performed on existing contracts to
monitor process performance
9
CISH-6050 - Software Engineering Management
Software Process Assessment &
Capability Evaluation Steps:
10
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Common Steps:
- Team Selection
 Select team trained in CMM
 Knowledgeable in SE & mgmt
- Maturity Questionnaire
 Site reps complete questionnaire
- Response Analysis
 Analyze results of questionnaire
 Investigation areas = KPAs
11
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Common Steps …
- On-site Visit
 Using results analysis, conduct onsite visit to view process areas
 Using KPAs as guide, question,
listen, review & synthesize info
 Apply professional judgment
 Document rationale for situations
where KPAs not met
12
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Common Steps …
- Findings
 At end of on-site period, team
produces list of findings
 Identifies strengths & weaknesses
of org’s software processes
 Software Process Assessment ->
Basis for PI recommendations
 Software Capability Evaluation ->
Findings part of risk analysis
13
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Common Steps …
- KPA Profile
 Team prepares KPA profile,
showing where KPAs satisfied / not
satisfied by organization
 KPA can be satisfied and still have
associated findings, as long as
findings don’t identify major
problems achieving goals of KPA
14
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Differences: Process Assessments
& Capability Evaluation
- Results of process assessment or
capability evaluation may differ
- Assessment / evaluation scope may vary:
 Different definitions of ‘Organization’
 Org may be based on senior
management, geo location, common
app, profit/loss center, etc.
 Sample of selected projects
15
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Differences …
- Motivation, objective, outcome & results
ownership differ
 These factors lead to differences in
dynamics of interviews, scope of
inquiry, info collected, & results
 Assessment & evaluation methods
are different
 Assessment training doesn’t prepare
team to do evaluation, vice versa
16
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Differences …
- Process Assessment – performed in
open, collaborative environment
 Commitment from mgmt & staff to do
process improvement
 Objective: surface problems & help
improve organization
 Emphasis on interviews as tool for
understanding organization’s
software process
17
CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …
• Differences …
- Capability Evaluation – performed in
audit-oriented environment
 Objective tied to monetary
considerations
 Emphasis on documented audit
trail that reveals software process
actually implemented by
organization
18
CISH-6050 - Software Engineering Management
AGENDA …
• Process Assessment
- SEI SW-CMM Improvement Approach
- Do organizational assessments pay
off?
- Whistleblowers
- Reference: SPICE Assessment
Documents
19
CISH-6050 - Software Engineering Management
Software Process Assessments
• Why do process assessment?
- “Why Do Organizations Have
Assessments? Do They Pay Off?”,
CMU SEI Technical Report, July, 1999
- Panel discussion from SEPG ’99 in
Atlanta, GA on March 10, 1999
- Each panelist is one of most active lead
assessor for SEI for CMM-Based
Appraisals for Internal Process
Improvement (CBA IPI)
20
CISH-6050 - Software Engineering Management
Software Process Assessments …
• Electronic Data Systems
- Why do assessments?
 Driven by customer: higher
maturity to keep customers; extra
bidding points for higher maturity;
DoD requirements
- Do assessments pay off?
 Yes!
 Need exec sponsorship, funding
 Everyone learns, helps org culture
21
CISH-6050 - Software Engineering Management
Software Process Assessments …
• Reuters
- Why do assessments?
 Framework for continuous process
improvement; maturity levels build on
previous level
 Raises awareness in group
- Do assessments pay off?
 Clear view of cost, effort, resource
 Measure key performance items
 Method for introducing global change
22
CISH-6050 - Software Engineering Management
Software Process Assessments …
• U. S. Navy
- Why do assessments?
 Early SEI affiliate
 Beta test group for Maturity
Questionnaire
 Continued with SEI assessments
- Do assessments pay off?
 Invaluable to organization
 In 1997 determined 4.1 to 1 ROI
 Education for new team members
23
CISH-6050 - Software Engineering Management
Software Process Assessments …
• Panel Q&A
-
Why pay cost of doing CBA IPI?
How often do you do full CBA IPI?
How small of organization to do CBA IPI?
How to reduce impact/stress on org when
doing CBA IPI?
24
CISH-6050 - Software Engineering Management
AGENDA …
• Process Assessment
-
SEI SW-CMM Improvement Approach
Do organizational assessments pay off?
Whistleblowers
Reference: SPICE Assessment
Documents
25
CISH-6050 - Software Engineering Management
Software Process Assessments …
• Blowing the Whistle on Troubled
Software Projects
- Article by Mark Keil & Daniel Robey
- Controversy behind going forward to
management when auditors find
problems with projects
- Case study of IS Auditors
 1999 Study
 75 Auditors
26
CISH-6050 - Software Engineering Management
Software Process Assessments …
• Key Points
-
Auditor obligation vs. risk
Reluctance to blow the whistle
Reluctance to hear the whistle
Conditions that encourage or influence
whistle blowing
27
CISH-6050 - Software Engineering Management
AGENDA …
• Process Assessment
-
SEI SW-CMM Improvement Approach
Do organizational assessments pay off?
Whistleblowers
Reference: SPICE Assessment
Documents
28
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference
• SPICE Software Process
Assessment Information
- SPICE documentation available free of
charge from Software Quality Institute
web site:
http://www.sqi.gu.edu.au/spice/suite/do
wnload.html
29
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …
• SPICE Part 3 – Rating Process
- Section 4.2: Defining Assessment Input
- Section 4.4: Assessing and Rating
Process
- Section 4.5: Recording the Assessment
Output
30
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …
• SPICE Part 4 – Guide to Conducting
Assessment
-
Section 4.2: Process Rating Scheme
Section 4.3.1.1: Team-based Assessment
Section 4.4: Assessment Stages (8)
Section 4.5: Success Factors for Process
Assessment
- Section 5.1: Reviewing Assessment
Inputs
31
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …
• SPICE Part 4 …
- Section 5.2: Selecting Process Instances
- Section 5.3: Preparing for Team-based
Assessment
- Section 5.4: Collecting and Verifying
Information
- Section 5.5: Determine Actual Ratings for
Process Instances
- Section 5.6: Determining Derived Ratings
- Section 5.7: Validating the Ratings
- Section 5.8: Presenting Assessment Output
32
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …
• SPICE Part 5 – Construction,
Selection, & Use of Assessment Tools
- Section 4.1: Form and Purpose of an
Assessment Instrument
- Section 4.2 (and Annexes A, B, C, D):
Implementation of Standard Indicators
- Section 4.3: Tailoring of Indicators
Contained in Assessment Instrument
- Section 4.4: Modular Assessment
Instruments
33
CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …
• SPICE Part 5 …
- Section 4.5: Capturing and Processing
Assessment Data
- Section 4.6: Using an Assessment
Instrument
• SPICE Part 6 – Qualifications &
Training of Assessors
- Section 4.1: Role of the Assessor
- Section 5: Assessor Competence
34
CISH-6050 - Software Engineering Management
References
• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity
Model for Software", Version 1.1, Software Engineering Institute,
Carnegie Mellon University, February, 1993. Available at
http://www.sei.cmu.edu/publications/documents/93.reports/93.tr.024.html
• W. S. Humphrey, Managing the Software Process, Addison-Wesley,
Reading, MA, 1989
• M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity
Model: Guidelines for Improving the Software Process, Addison-Wesley,
Boston, MA, 1995
• “Process Maturity Profile of the Software Community 2002 Mid-Year
Update”, Software Engineering Institute, Carnegie Mellon University,
August, 2002. Available at http://www.sei.cmu.edu/sema/pdf/SWCMM/2002aug.pdf
35
CISH-6050 - Software Engineering Management
References …
• D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G.
Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”,
Software Engineering Institute, Carnegie Mellon University, July, 1999.
Available at
http://www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr0
12abstract.html
• M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”,
Communications of the ACM, Vol. 44, No. 4, April 2001, pp. 87-93. PDF
file (ACMp87-Keil.pdf) available online via ACM Digital Library when
accessed from the RPI at Hartford Library
• ISO/IEC TR 15504:1998 - Software Process Assessment (SPICE)
documentation suite. Available at
http://www.sqi.gu.edu.au/spice/suite/download.html
36
CISH-6050 - Software Engineering Management