Transdyne Corporation
http://transdynecorp.com
SCAMPI Class A, B, C
ID No. 0100145-01
Contributing Factors Determining the Sufficiency of COCOMO-Based
Estimation Processes in SCAMPISM Appraisals
23rd International Forum on COCOMO and Systems/Software Cost Modeling and ICM Workshop 3
Dr. Mary Anne Herndon
Certified SCAMPI A, B and C Appraiser
Transdyne Corporation, San Diego, CA
October 27 2008
SM CMM Integration, SCAMPI, SCAMPI Lead Assessor, SCAMPI Lead Appraiser, and
SEI are service marks of Carnegie Mellon University.
 CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie
Mellon University.
Contributing Factors Determining the
Sufficiency of COCOMO-Based
Estimation Processes in SCAMPISM Appraisals
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Agenda
• Overview of the SCAMPI Family
• Three Phases of a SCAMPI Appraisal
• The Rules for Verification of Appraisal Evidence
• Expected SCAMPI Lead Appraiser Competencies and Behaviors
• Overview of CMMI-DEV v1.2 Model and COCOMO-based Estimation Processes
• Examples of Contributing Factors in Process Areas
• Benefits and Summary
2
SEI SCAMPI Reference Documents
SCAMPI Method
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Reference Documents
•
Appraisal Requirements for
CMMI (ARC) v1.2
(CMU/SEI-2006-TR-011), 2006
•
SCAMPI Lead Appraiser SM
Body of Knowledge (SLA BOK)
(CMU/SEI-2007-TR-019), 2007
Class A
•
Standard CMMI Appraisal
Method for Process
Improvement
Method Definition Document
(MDD) v1.2
(CMU/SEI-2006-HB-002), 2006
Class B and C
•
Handbook for Conducting
Standard CMMI Appraisal
Method for Process
Improvement (SCAMPI) B and C
Appraisals
(CMU/SEI-2005-HB-005), 2005
Class A, B, and C
3
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Three Classes of SCAMPI Appraisal
Methods
Overview of SCAMPI Methods
Institutionalization
A
SCAMPI B provides options in model and
organizational scope, but characterization
of practices is fixed to one scale and is
performed on implemented practices.
Depth of investigation
SCAMPI A is the most rigorous method
and is the only method that can result in
ratings (benchmark).
B
Deployment
SCAMPI C provides a wide
range of options, including
characterization of planned approaches
to process implementation according
to a scale defined by the user.
C
Plans
Breadth of tailoring
4
SCAMPI Method Appraisal Phases
SCAMPI Phase Activities
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Relevant When Implementing
COCOMO-Based Estimation
Processes
There are three phases of a
SCAMPI appraisal
Phase 1 - Plan and Prepare
for the Appraisal
Yes
Phase 2 - Conduct Appraisal
Yes
Phase 3 - Report Results
Reporting results obtained in
Phases 1 and 2
5
Phase 1 Requirements – Plan and
Prepare for the Appraisal
Requirement
Types of objective
evidence gathered*
SCAMPI A
Documents AND
interviews
SCAMPI B
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
SCAMPI C
Documents
AND
interviews
Documents OR interviews
Class A
Ratings generated*
Goal ratings
required
No ratings
allowed
No ratings allowed
Organizational Unit
coverage*
Required
Not required
Not required
Minimum team size*
4
2
1
Appraisal Team Leader
requirements
SCAMPI A Lead
Appraiser
SCAMPI B and
C Team Leader
SCAMPI B and C Team Leader
Appraisal Input
Statement and Plan
signed by Sponsor
Yes
Yes
Yes
Class B
Class C
* Requirement Relevant When Implementing COCOMO-Based Estimation Processes
6
Phase 1 Requirements – Plan and Prepare
for the Appraisal (Continued)
Requirement
SCAMPI A
SCAMPI B
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
SCAMPI C
Team members
complete Intro to
CMMI and some form
of team training
Yes
Yes
Yes – if a team is used
Some form of
Readiness Review *
Yes
Yes
No
Class A
Objective evidence
collection plan*
Required
Criteria for team *
experience in
engineering and
management
Yes
Required
Required
Class B
Yes
Yes
(may
not be met but
Class C
must be documented)
* Requirement Relevant When Implementing COCOMO-Based Estimation Processes
7
Phase 2 Requirements - Conduct Appraisal
Requirement
SCAMPI A
Direct Artifacts
reviewed*
Yes
Some form of
Interviews conducted*
Yes
Sources of objective
evidence (interviews,
documentation,
instruments)*
2
Interviews conducted
by at least 2 team
members*
Yes
Observations
corroborated*
Yes
SCAMPI B
Yes
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
SCAMPI C
No *
* One type of objective evidence
collection is required
Yes
No*
* One type of objective evidence
collection is required
2
1
Class A
Yes
No
Class B
Yes
No
Class C
Validation of
preliminary findings*
Yes
Yes
No
* Requirement Relevant When Implementing COCOMO-Based Estimation Processes
8
SCAMPI Phase 2 “Conduct Appraisal” Processes
Process
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Description
• Examine objective evidence
• Document objective evidence (data sufficiency)
• Verify objective evidence ( practice strengths &
weakness)
• Practice Characterization
Class A – Fully Implemented (FI)
Largely Implemented (LI)
Partially Implemented (PI)
Not Implemented (NI)
Not Yet (NY)
Appraisal Team
Class B – Green, Yellow, Red
Class C – High, Medium, Low (optional)
• Validate preliminary appraisal outputs
• Generate appraisal results
Class A - Goal Rating, Maturity and Capability
Levels, final findings, Appraisal Disclosure Statement (ADS)
Class B and C – characterizations (optional in C),
final findings, ADS
9
Phase 3 Requirements – Report Results
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Activities
Common requirements for all SCAMPI
Classes:
•
Appraisal Requirements for CMMI
(ARC) v1.2
(CMU/SEI-2006-TR-011), 2006
• Designated appraisal results shall be
provided to the stakeholders identified
in the appraisal plan
• The appraisal record shall be
delivered to the appraisal sponsor
• The appraisal data package shall be
submitted to the CMMI Steward
The following items must be submitted to the
CMMI steward:
• Appraisal plan, including the
appraisal input
• Appraisal findings – strength and/or
weakness statements
• Appraisal Disclosure Statement
(ADS)
• Record of Entry
SAS
10
The Types of Evidence used in SCAMPI Appraisals
Process
Implementation
Indicators
(Evidence)
Direct Artifact
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Description and Examples
The tangible outputs associated with implementation of a CMMI
practice.
Example: COCOMO output data provided by a project participating in
the appraisal
Indirect Artifact
Evidence that is a consequence of implementing a practice, but not
the primary purpose of the practice.
Example: Minutes of project planning meeting discussing the use of
COCOMO.
Affirmations
Oral/written evidence confirming/supporting implementation of a
practice (or lack of implementation) typically obtained by interviews or
presentations and provided usually by the person performing the work.
Example: During a presentation, the project manager states that
“he/she was responsible for implementing COCOMO on the project”.
11
Rules for Types of Evidence used in SCAMPI Appraisals
Process
Implementation
Indicators (Evidence)
Class A
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Data Coverage and Sufficiency
Each practice must have at least one direct artifact and
indirect/affirmation.
For each goal, at least 50% of the cells in the participating projects
must have at least one affirmation.
Class B
Each practice must have at least direct artifact and there must be at
least one affirmation from the set of projects participating in the
appraisal for each practice.
Class C
Each practice must have at least one direct or indirect or affirmation
as evidence.
References
•
Appraisal Requirements for CMMI (ARC) v1.2
(CMU/SEI-2006-TR-011), 2006
•
Standard CMMI Appraisal Method for Process Improvement
Method Definition Document (MDD) v1.2
(CMU/SEI-2006-HB-002), 2006
•
Handbook for Conducting Standard CMMI Appraisal Method
for Process Improvement (SCAMPI) B and C Appraisals
(CMU/SEI-2005-HB-005), 2005
12
Rules for Corroboration of Evidence
SCAMPI Class
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Corroboration Requirement
Class A and B
The objective evidence is obtained from at least two different
sources.
At least one of the two sources must reflect work actually being
done (e.g. process area implementation)
Class C
Not required
References
•
Appraisal Requirements for CMMI (ARC) v1.2
(CMU/SEI-2006-TR-011), 2006
•
Standard CMMI Appraisal Method for Process Improvement
Method Definition Document (MDD) v1.2
(CMU/SEI-2006-HB-002), 2006
•
Handbook for Conducting Standard CMMI Appraisal Method
for Process Improvement (SCAMPI) B and C Appraisals
(CMU/SEI-2005-HB-005), 2005
13
SCAMPI Lead Appraiser SM Body of Knowledge (SLA BOK)
Competency Cluster - MDD Knowledge Area (KA) Mapping
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Competency
Cluster
COCOMO Relevant MDD
Knowledge Area (KA)
Competency
Cluster
COCOMO Relevant MDD
Knowledge Area (KA)
1. Achieving &
Managing
Agreements
KA 3 Managing Objective Evidence
6. Understanding &
Adapting to
Organizational
Context*
KA 1 Appraisal Planning
KA 4 Judging Processes
2. Decision Making
and Problem Solving
KA 2 Preparing Team & Participants
KA 3 Managing Objective Evidence
7. Model
Interpretation*
KA 1
KA 2
KA 3
KA 4
KA 5
3. Project Planning
& Management*
KA 2 Preparing Team & Participants
KA 3 Managing Objective Evidence
8. SCAMPI Method
Tailoring,
Adaptation*
KA 3 Managing Objective Evidence
KA 4 Judging Processes
4. Interpersonal
Communication &
facilitation
KA 3 Managing Objective Evidence
KA 4 Judging Processes
9. Professionalism
KA 4 Judging Processes
Lead appraiser
5. Integration,
Articulation, and
Expression of
Information*
KA 3 Managing Objective Evidence
KA 4 Judging Processes
* Higher Maturity
addressed in these
Competency
Clusters
Appraisal Planning
Preparing Team & Participants
Managing Objective Evidence
Judging Processes
Reporting Outcomes
Appraisal team 14
CMMI v1.2-DEV: Continuous and Staged Representations
Category
Process Areas Including IPPD
Organizational Process Focus
Process
Management Organizational Process Definition + IPPD
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Level
Process Areas
5
Optimizing
Organizational Innovation and
Deployment
Causal Analysis and Resolution
4
Managed
Organizational Process Performance
Quantitative Project Management
3
Defined
Integrated Project Management
+ IPPD
Risk Management
Decision Analysis and Resolution
Organizational Process Focus
Organizational Process Definition
+ IPPD
Organizational Training
Requirements Development
Technical Solution
Product Integration
Verification
Validation
2
Managed
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Configuration Management
Process and Product Quality
Assurance
Measurement and Analysis
Requirements Management
Project Planning
Project
Management Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management + IPPD
Risk Management
Quantitative Project Management
Engineering Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Support
Configuration Management
Process and Product Quality
Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
1 Initial
15
15
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Overview: CMMI-DEV v1.2 Process Areas (PAs)
and COCOMO-Based Estimation Processes
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Support PAs
Organization
planning improvement
of estimation process
Process
Management
PAs
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Generic Goals
Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
(Institutionalization)
COCOMO
Model
Estimating
Process
Engineering PAs
Project
Management
PAs
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management
Risk Management
Quantitative Project Management
Appraisal team
verifying
COCOMO-based
estimation
evidence
16
Examples of Contributing Factors in SCAMPI
Appraisal Phases 1 and 2 for Process Management
Process Areas (Capability/Maturity Levels 1 - 3)
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Process Areas
Phase 1: Appraisal Planning
Phase 2: Conduct Appraisal (Evidence)
OPD, OPF, OT
Sufficient time and resources for
institutionalization of organizational
and defined project estimation
processes, including tailoring criteria
and any staff COCOMO training
Improvement estimates for estimation
Verbal
Affirmations
(interviews &
presentations)
Project
managers,
process
engineers,
higher level
managers,
COCOMO
experienced
staff and
trainers
Historical sufficiency of COCOMObased estimation measurements for
relevant projects
Identification of business goals for
improving project estimation
A “thread” of planned defined estimation
process and implementation, including
estimation data, completed project plans,
COCOMO outputs, thresholds for tracking
project completion, analysis of “significant
variances”
Improvement information of estimation
process, such as lessons learned,
measurements, action plans,.etc.
Peer reviews of estimation process
QA audits of estimation process
Relevant appraisal findings
Training plan, record of staff/project
managers responsible for COCOMO;
measures of effectiveness
17
Examples of Contributing Factors in SCAMPI
Appraisal Phases 1 and 2 for Process, Project
Management and Support Process Areas
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
(Higher Capability/Maturity Levels)
Process Areas
Phase 1: Appraisal Planning
Phase 2: Conduct Appraisal (Evidence)
CAR, OID,
OPP, QPM
Business goals linked to
estimation process performance
model
A “thread” of projects using estimation
process and implementation of process
model, defined process/subprocess including
ranges for tracking project completion,
analysis of special and common cause
variations
Verbal
Affirmations
(interviews &
presentations):
Project
managers,
process
engineers,
higher level
managers,
COCOMO
experienced
staff
Historical sufficiency of
COCOMO-based estimation
measurements
Sufficient time for statistical
stabilization of estimation
process
Improvement information of estimation
process, such as lessons learned,
measurements, action plans, etc.
Peer reviews of process performance model
Candidate realistic (process and
innovative) improvements in
estimation process linked to
documented business goals
Adequate resources to develop
process performance models
to improve project estimation
Causal analysis reports of
defects/improvements in current COCOMObased estimation process
Action and deployment plans and results of
improvements
18
Examples of Contributing Factors in SCAMPI
Appraisal Phases 1 and 2 for Project Management
Process Areas (Capability/Maturity Levels 1 - 3)
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
Process Areas
Phase 1: Appraisal Planning
Phase 2: Conduct Appraisal (Evidence)
IPM, PP, PMC,
RSKM, SAM
Sufficient time and resources for
institutionalization of organizational
and defined project estimation
processes, including tailoring plan
and any staff COCOMO training
A “thread” of planned defined estimation
process and implementation, including
estimation data, completed project plans,
COCOMO outputs, thresholds for tracking
project completion, analysis of “significant
variances”
Verbal
Affirmations
(interviews &
presentations)
Project
managers,
process
engineers,
higher level
managers,
COCOMO
experienced
staff
Historical sufficiency of COCOMObased estimation measurements
Appropriate risk management
strategy
Tracking and completion of project action
items, as appropriate
Appropriate risk identification
and management of projects
Improvement information of estimation
process, such as lessons learned,
measurements, completed action plans
Peer reviews & QA audits of estimation
process
Training plan and record of staff/project
managers responsible for COCOMO
19
Examples of Contributing Factors for SCAMPI
Appraisal Phases 1 and 2 for Engineering and
Support Process Areas
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
(Capability/Maturity Levels 1 - 3)
Process Areas
Phase 1: Appraisal Planning
Phase 2: Conduct Appraisal (Evidence)
RD, REQM PI,
TS, VAL, VER
Sufficient time and resources
for institutionalization of
organizational and defined
project estimation processes,
including any staff COCOMO
training
A “thread” of planned defined estimation process and
implementation, including estimation data, completed
project plans, complexity measures, COCOMO
outputs, thresholds for tracking project completion,
analysis of “significant variances”
CM, DAR, MA,
PPQA
Verbal
Affirmations
(interviews &
presentations)
Project
managers,
process
engineers,
higher level
managers,
COCOMO
experienced
staff
Historical sufficiency of
COCOMO-based estimation
measurements
Implementation of life cycle
models
Selection guidance for project
complexity values
Tracking and completion of project action items, as
appropriate,in product life cycle phases
Relevant project planning and tracking measurements
for life cycle phases
Relevant implementations of decision analysis
process, if appropriate
Peer reviews of estimation process
QA audits of estimation process
Revision histories of COCOMO outputs
20
Contributing Factors for SCAMPI Appraisal Phases
1 and 2 for Institutionalization
(Generic Goals (GG) 1 – 5)
Generic Goal
Phases 1 and 2: Plan and Conduct Appraisal
GG 1: Achieve
Specific Goals
COCOMO-related evidence for each PA in model scope
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
And
GG 2: Institutionalize
a Managed Process
COCOMO-related evidence for PP, PMC,CM, MA, PPQA, OT*
IPM*
And
GG 3: Institutionalize
a Defined Process
COCOMO-related evidence for IPM, OPD, OPF PAs
And
GG 4: Institutionalize
a Quantitatively
Managed Process
COCOMO-related evidence for OPP, QPM PAs
And
GG 5: Institutionalize
an Optimizing
Process
COCOMO-related evidence for CAR, OID, PAs
*If PA is in the model scope of appraisal
21
SUMMARY
Benefits
SCAMPI
Appraisals
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
COCOMO is helpful for organizations to improve process consistency in
estimating project resources. COCOMO automates resource estimation
using calibrated algorithms and data sets maintained by a large
community of practitioners and developers.
In mature organizations, the use of COCOMO establishes historical data
that is a perquisite for the journey to higher maturity levels.
Maturity Level 4 organizations use the project estimation database to
develop the process performance models used to forecast project
resources for different domains and customers.
Maturity Level 5 organizations are challenged by changing business
environments to reduce product costs and schedules. The resource
estimates provided by COCOMO algorithms provides an aid to
quantitatively understand the random variance in development processes
to model candidate improvements.
Key planning factors to increase the efficiency of data collection and
verification activities in SCAMPI appraisals include providing:
1.
2.
3.
4.
“Evidence threads” to each CMMI practice “touched” by COCOMO
estimated resources.
Staff adequately trained on COCOMO for the project(s)
Project managers actually using the estimates to plan and manage
the project(s).
Higher level managers that are aware of resource estimation as an
effective management tool to achieve business goals.
22
The End
Transdyne Corporation
http://transdynecorp.com
COCOMO 2008 Forum
You have just seen a
summary of SCAMPI A, B,
C and CMMI-DEV v1.2
requirements relevant to
organizations implementing
COCOMO-based estimation
processes from the “30,000
feet” level.
Questions, Comments and
Concerns ?
23