Sampling the SCAMPI Sampling Rules
or
Trying to Explain the Unexplainable
Pat O’Toole, PACT
www.PACTCMMI.com
May, 2012
1
SCAMPI Sampling Rules
SCAMPI Class A v1.3
SCAMPI
–Standard CMMI Appraisal Method for Process Improvement
SCAMPI MDD
– SCAMPI Method Definition Document
 250+ page process document for SCAMPI A appraisals
Specific source of angst
–SCAMPI MDD v1.3 Section 1.1.4
–It’s only 7 pages, but do NOT drive or operate heavy
equipment after reading it!
3
SCAMPI Sampling Rules
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
4
SCAMPI Sampling Rules
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
5
SCAMPI Sampling Rules
Basic Unit <of Work>
For CMMI-DEV, “Basic Unit” is roughly equivalent
to “Project”
–However, “project” doesn’t work as well for the
Services or Acquisition constellations
–“Basic Unit” is much more generic
–Therefore, for purposes of this presentation, I will use
“PROJECT!”
6
SCAMPI Sampling Rules
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
7
SCAMPI Sampling Rules
Sampling Factors – Why?
 Project work may be performed in different ways
depending on various characteristics
– Small projects may be conducted differently than large ones
– Project work conducted by your USA-based team may be
done differently than your offshore development group
– Scrum projects are run differently than Waterfall projects
Sampling factors ensure that the organizational scope
provides a robust (and complete) view of the various
ways work is performed by the organization.
8
SCAMPI Sampling Rules
Mandatory Candidate Sampling Factors
 The Sampling Factors that MUST be considered:
–Location (e.g., Minneapolis vs. Las Vegas)
–Customer (e.g., DoD vs. commercial industry)
–Size (e.g., small vs. medium vs. large vs. 4XL)
–Organizational Structure (e.g., different divisions)
–Type of Work (e.g., development vs. maintenance)
For each candidate, the question to be answered is:
–Do different settings for this factor affect the way that
work is performed in the organization to be appraised?
 Yes – Relevant Sampling Factor
 No – Irrelevant Sampling Factor.
9
SCAMPI Sampling Rules
Other Potential Sampling Factors
Other Relevant Sampling Factors may exist and, if
so, must also be considered
–Life Cycle (e.g., Scrum projects vs. Waterfall projects)
–Funding Source (e.g., customer vs. internal funding)
–Criticality (e.g., nice-to-have vs. life critical)
–Etc.
Your organization may have unique characteristics
that significantly affect how work is performed
–These would be included as Relevant Sampling Factors
in a SCAMPI A appraisal.
10
SCAMPI Sampling Rules
Example
Project Location Customer
Size
Type of Work
1
MSP
Internal
Small Maintenance
2
MSP
Internal Medium Development
3
MSP
External Medium Development
4
MSP
External Medium Maintenance
5
MSP
External
Small Maintenance
6
MSP
External
Large Development
7
MSP
External Medium Development
8
MSP
External Medium Development
9
MSP
External Medium Development
10
MSP
External
Large Development
11
MSP
External
Small Maintenance
12
MSP
Internal Medium Development
13
MSP
External Medium Maintenance
14
MSP
External
Small Maintenance
15
MSP
External Medium Development
11
SCAMPI Sampling Rules
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
12
SCAMPI Sampling Rules
Subgroup
“Cluster” of projects that share common relevant
sampling factor values and exhibit similar process
implementations
–If Customer, Size, and Type of Work are relevant
sampling factors, how many POSSIBLE subgroups are
there?
 Customer: Internal / External
 Size: Small / Medium / Large
 Type of Work: Development / Maintenance
 2 values for <Customer> x
3 values for <Size> x
2 values for <Type of Work> = 12 possible subgroups.
13
SCAMPI Sampling Rules
How Many ACTUAL Subgroups?
Project Location Customer
1
MSP
Internal
2
MSP
Internal
3
MSP
External
4
MSP
External
5
MSP
External
6
MSP
External
7
MSP
External
8
MSP
External
9
MSP
External
10
MSP
External
11
MSP
External
12
MSP
Internal
13
MSP
External
14
MSP
External
15
MSP
External
Size
Small
Medium
Medium
Medium
Small
Large
Medium
Medium
Medium
Large
Small
Medium
Medium
Small
Medium
Type of Work
Maintenance
Development
Development
Maintenance
Maintenance
Development
Development
Development
Development
Development
Maintenance
Development
Maintenance
Maintenance
Development
14
3-tuple
#
Interna l , Sma l l , Dev.
Interna l , Medi um, Dev.
2
Interna l , La rge, Dev.
Interna l , Sma l l , Ma i nt.
1
Interna l , Medi um, Ma i nt.
Interna l , La rge, Ma i nt.
Externa l , Sma l l , Dev.
Externa l , Medi um, Dev.
Externa l , La rge, Dev.
Externa l , Sma l l , Ma i nt.
Externa l , Medi um, Ma i nt.
5
2
3
2
Externa l , La rge, Ma i nt.
SCAMPI Sampling Rules
How Many Projects per Subgroup?
Minimum #
of projects
=
from a given
subgroup
Number of
X
subgroups
Number of
projects in the
given subgroup
Total number
of projects
15
SCAMPI Sampling Rules
Back to Our Example…
Subgroups
Interna l
Medi um
Devel opment
Interna l
Sma l l
Ma i ntena nce
Externa l
Medi um
Devel opment
Externa l
La rge
Devel opment
Externa l
Sma l l
Ma i ntena nce
Externa l
Medi um
Ma i ntena nce
#
2
1
5
# of IMD = (6 x 2) / 15 = 0.8 => 1
# of ISM = (6 x 1) / 15 = 0.4 => 1
2
3
# of EMD = (6 x 5) / 15 = 2.0 => 2
# of ELD = (6 x 2) / 15 = 0.8 => 1
# of ESM = (6 x 3) / 15 = 1.2 => 1
2
# of EMM = (6 x 2) / 15 = 0.8 => 1.
16
SCAMPI Sampling Rules
What if…
Subgroups
Interna l
Medi um
Devel opment
Interna l
Sma l l
Ma i ntena nce
Externa l
Medi um
Devel opment
Externa l
La rge
Devel opment
Externa l
Sma l l
Ma i ntena nce
Externa l
Medi um
Ma i ntena nce
#
200
100
500
# of IMD = (6 x 200)/1500 = 0.8 => 1
# of ISM = (6 x 100)/1500 = 0.4 => 1
200
300
# of EMD = (6 x 500)/1500 = 2.0 => 2
# of ELD = (6 x 200)/1500 = 0.8 => 1
# of ESM = (6 x 300)/1500 = 1.2 => 1
200
# of EMM = (6 x 200)/1500 = 0.8 => 1
17
SCAMPI Sampling Rules
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
18
SCAMPI Sampling Rules
Coverage Rules
Three flavors:
–Coverage Rules for Process Areas
–Coverage Rules for Basic Units (whoops! – Projects)
–Coverage Rules for Support Functions.
19
SCAMPI Sampling Rules
Cover Rules for Process Areas
PA Coverage Rule 1 (in for a penny, in for a pound)
–Evaluation of a given PA must cover all practices, both
specific and generic
–There is a wee bit of wiggle room here as the rule
includes the words, “…for which they supply data”
PA Coverage Rule 2 (no need to keep pounding)
–If a PA is performed in a consistent way across
subgroups, evaluation of that PA may be consolidated
–For example, if CM, DAR, PPQA, and RSKM are
performed in the same manner across all projects, then
they need not be evaluated for all 6 subgroups.
20
SCAMPI Sampling Rules
Project Coverage Rules
Project Coverage Rule 1
– For each subgroup, both artifacts and affirmations shall be
provided for at least one project for every process area
 This sampled project shall provide data for all process areas
Project Coverage Rule 2
– For at least 50% of the sampled projects in each subgroup,
both artifacts and affirmations shall be provided for at least
one process area
Project Coverage Rule 3
– For all sampled basic units in each subgroup either
artifacts or affirmations shall be provided for at least one
process area.
21
SCAMPI Sampling Rules
ATLAS #19a
Can you apply what you’ve learned?
–Well, heard?
Let’s find out…
Work individually or together to answer the
questions in ATLAS #19a
–There is no need to provide comments.
22
SCAMPI Sampling Rules
Cover Rules for Process Areas
PA Coverage Rule 1
–Evaluation of a given PA must cover all practices, both
specific and generic
–HINT: Start with the question,
“Which projects contribute to its fulfillment?”
23
SCAMPI Sampling Rules
Project Coverage Rules
Project Coverage Rule 1
– For each subgroup, both artifacts and affirmations shall be
provided for at least one project for every process area
 This sampled project shall provide data for all process areas
Project Coverage Rule 2
– For at least 50% of the sampled projects in each subgroup,
both artifacts and affirmations shall be provided for at least
one process area
Project Coverage Rule 3
– For all sampled basic units in each subgroup either
artifacts or affirmations shall be provided for at least one
process area.
24
SCAMPI Sampling Rules
ATLAS #19a Results
ATLAS #19a was distributed to all SCAMPI lead
appraisers and only SCAMPI lead appraisers
61 SCAMPI lead appraisers responded
–That’s about a 12% response rate
–But it’s still more than “one expert’s opinion”
Let’s see how YOU did compared with those
responsible for REALLY understanding this stuff…
25
SCAMPI Sampling Rules
Question 1
1. Is Process Area Coverage Rule 1 fulfilled?
<Evaluation of a given PA must cover all practices,
both specific and generic.>
36% Yes / 64% No
Which project(s) contribute to its fulfillment:
P1? 82% Yes / 17% No
P2? 90% Yes / 10% No
P3? 74% Yes / 24% No
P4?
7% Yes / 93% No.
26
SCAMPI Sampling Rules
Question 2
2.
Is Project (BU) Coverage Rule 1 fulfilled?
“For each subgroup, both artifacts and affirmations
shall be provided for at least one project for every
process area”
34% Yes / 66% No
Which project(s) contribute to its fulfillment:
P1? 42% Yes / 56% No
P2? 14% Yes / 86% No
P3? 4% Yes / 96% No
P4? 4% Yes / 96% No.
27
SCAMPI Sampling Rules
Question 3
3.
Is Project (BU) Coverage Rule 2 fulfilled?
“For at least 50% of the sampled projects in each
subgroup, both artifacts and affirmations shall be
provided for at least one process area”
51% Yes / 49% No
Which project(s) contribute to its fulfillment:
P1? 54% Yes / 46% No
P2? 97% Yes / 3% No
P3? 22% Yes / 78% No
P4? 18% Yes / 82% No.
28
SCAMPI Sampling Rules
Question 4
4.
Is Project (BU) Coverage Rule 3 fulfilled?
“For all sampled basic units in each subgroup either
artifacts or affirmations shall be provided for at least
one process area.”
48% Yes / 52% No
Which project(s) contribute to its fulfillment:
P1? 94% Yes / 6% No
P2? 98% Yes / 2% No
P3? 93% Yes / 7% No
P4? 25% Yes / 75% No.
29
SCAMPI Sampling Rules
Optical Illusion – 2 Images in One
30
SCAMPI Sampling Rules
Modified Coverage Rules - #1
 PA Coverage–1: Objective evidence provided for a PA must
address all practices that are part of that PA, both specific
practices as well as all generic practices. The Basic Unit and
Support Function coverage rules indicate the types of objective
evidence that must be provided for each practice.
 Project Coverage–1: For each subgroup, both artifacts and
affirmations shall be provided for each practice for at least one
project.
 Project Coverage–2: For at least 50 percent of the sampled
projects in each subgroup, both artifacts and affirmations shall
be provided for each practice for at least one PA
 Project Coverage–3: For all sampled projects in each subgroup
either artifacts or affirmations shall be provided for each
practice for at least one PA.
31
SCAMPI Sampling Rules
Modified Coverage Rules - #2
 PA Coverage–1: Objective evidence provided for a PA must
address all practices that are part of that PA, both specific
practices as well as all generic practices. [No change]
 Project Coverage–1: For each subgroup, some combination of
both artifacts and affirmations shall be provided for at least one
project.
 Project Coverage–2: For at least 50 percent of the sampled
projects in each subgroup, some combination of both artifacts
and affirmations shall be provided for at least one PA
 Project Coverage–3: For all sampled projects in each subgroup
either artifacts and/or affirmations shall be provided for at least
one PA.
32
SCAMPI Sampling Rules
Conclusion
The new sampling and coverage rules are intended to
provide a truly representative set of projects that
reflect how work is performed in the organization
–There are some issues, however:
 They only solve the “proportional problem,” not the “how
much is enough” problem
 There are multiple ways to interpret the rules as written,
arriving at significantly different results
Based on ATLAS #19a, the SEI has committed to
clarify, or even re-write the coverage rules
With respect to brain function, covering this topic is
equivalent to drinking 6 beers; please drive carefully!
33
SCAMPI Sampling Rules
Questions???
Pat O’Toole
Process Assessment, Consulting & Training (PACT)
PACT.otoole@att.net
www.PACTCMMI.com
If anyone is interested in receiving my monthly
“Do’s and Don’ts” articles, either:
Give me a business card
Send me an email.
34
SCAMPI Sampling Rules