Sampling the SCAMPI Sampling Rules or Trying to Explain the Unexplainable Pat O’Toole, PACT www.PACTCMMI.com May, 2012 1 SCAMPI Sampling Rules SCAMPI Class A v1.3 SCAMPI –Standard CMMI Appraisal Method for Process Improvement SCAMPI MDD – SCAMPI Method Definition Document 250+ page process document for SCAMPI A appraisals Specific source of angst –SCAMPI MDD v1.3 Section 1.1.4 –It’s only 7 pages, but do NOT drive or operate heavy equipment after reading it! 3 SCAMPI Sampling Rules New Terminology with MDD v1.3 Basic Units Sampling Factors Subgroups Coverage Rules. 4 SCAMPI Sampling Rules New Terminology with MDD v1.3 Basic Units Sampling Factors Subgroups Coverage Rules. 5 SCAMPI Sampling Rules Basic Unit <of Work> For CMMI-DEV, “Basic Unit” is roughly equivalent to “Project” –However, “project” doesn’t work as well for the Services or Acquisition constellations –“Basic Unit” is much more generic –Therefore, for purposes of this presentation, I will use “PROJECT!” 6 SCAMPI Sampling Rules New Terminology with MDD v1.3 Basic Units Sampling Factors Subgroups Coverage Rules. 7 SCAMPI Sampling Rules Sampling Factors – Why? Project work may be performed in different ways depending on various characteristics – Small projects may be conducted differently than large ones – Project work conducted by your USA-based team may be done differently than your offshore development group – Scrum projects are run differently than Waterfall projects Sampling factors ensure that the organizational scope provides a robust (and complete) view of the various ways work is performed by the organization. 8 SCAMPI Sampling Rules Mandatory Candidate Sampling Factors The Sampling Factors that MUST be considered: –Location (e.g., Minneapolis vs. Las Vegas) –Customer (e.g., DoD vs. commercial industry) –Size (e.g., small vs. medium vs. large vs. 4XL) –Organizational Structure (e.g., different divisions) –Type of Work (e.g., development vs. maintenance) For each candidate, the question to be answered is: –Do different settings for this factor affect the way that work is performed in the organization to be appraised? Yes – Relevant Sampling Factor No – Irrelevant Sampling Factor. 9 SCAMPI Sampling Rules Other Potential Sampling Factors Other Relevant Sampling Factors may exist and, if so, must also be considered –Life Cycle (e.g., Scrum projects vs. Waterfall projects) –Funding Source (e.g., customer vs. internal funding) –Criticality (e.g., nice-to-have vs. life critical) –Etc. Your organization may have unique characteristics that significantly affect how work is performed –These would be included as Relevant Sampling Factors in a SCAMPI A appraisal. 10 SCAMPI Sampling Rules Example Project Location Customer Size Type of Work 1 MSP Internal Small Maintenance 2 MSP Internal Medium Development 3 MSP External Medium Development 4 MSP External Medium Maintenance 5 MSP External Small Maintenance 6 MSP External Large Development 7 MSP External Medium Development 8 MSP External Medium Development 9 MSP External Medium Development 10 MSP External Large Development 11 MSP External Small Maintenance 12 MSP Internal Medium Development 13 MSP External Medium Maintenance 14 MSP External Small Maintenance 15 MSP External Medium Development 11 SCAMPI Sampling Rules New Terminology with MDD v1.3 Basic Units Sampling Factors Subgroups Coverage Rules. 12 SCAMPI Sampling Rules Subgroup “Cluster” of projects that share common relevant sampling factor values and exhibit similar process implementations –If Customer, Size, and Type of Work are relevant sampling factors, how many POSSIBLE subgroups are there? Customer: Internal / External Size: Small / Medium / Large Type of Work: Development / Maintenance 2 values for <Customer> x 3 values for <Size> x 2 values for <Type of Work> = 12 possible subgroups. 13 SCAMPI Sampling Rules How Many ACTUAL Subgroups? Project Location Customer 1 MSP Internal 2 MSP Internal 3 MSP External 4 MSP External 5 MSP External 6 MSP External 7 MSP External 8 MSP External 9 MSP External 10 MSP External 11 MSP External 12 MSP Internal 13 MSP External 14 MSP External 15 MSP External Size Small Medium Medium Medium Small Large Medium Medium Medium Large Small Medium Medium Small Medium Type of Work Maintenance Development Development Maintenance Maintenance Development Development Development Development Development Maintenance Development Maintenance Maintenance Development 14 3-tuple # Interna l , Sma l l , Dev. Interna l , Medi um, Dev. 2 Interna l , La rge, Dev. Interna l , Sma l l , Ma i nt. 1 Interna l , Medi um, Ma i nt. Interna l , La rge, Ma i nt. Externa l , Sma l l , Dev. Externa l , Medi um, Dev. Externa l , La rge, Dev. Externa l , Sma l l , Ma i nt. Externa l , Medi um, Ma i nt. 5 2 3 2 Externa l , La rge, Ma i nt. SCAMPI Sampling Rules How Many Projects per Subgroup? Minimum # of projects = from a given subgroup Number of X subgroups Number of projects in the given subgroup Total number of projects 15 SCAMPI Sampling Rules Back to Our Example… Subgroups Interna l Medi um Devel opment Interna l Sma l l Ma i ntena nce Externa l Medi um Devel opment Externa l La rge Devel opment Externa l Sma l l Ma i ntena nce Externa l Medi um Ma i ntena nce # 2 1 5 # of IMD = (6 x 2) / 15 = 0.8 => 1 # of ISM = (6 x 1) / 15 = 0.4 => 1 2 3 # of EMD = (6 x 5) / 15 = 2.0 => 2 # of ELD = (6 x 2) / 15 = 0.8 => 1 # of ESM = (6 x 3) / 15 = 1.2 => 1 2 # of EMM = (6 x 2) / 15 = 0.8 => 1. 16 SCAMPI Sampling Rules What if… Subgroups Interna l Medi um Devel opment Interna l Sma l l Ma i ntena nce Externa l Medi um Devel opment Externa l La rge Devel opment Externa l Sma l l Ma i ntena nce Externa l Medi um Ma i ntena nce # 200 100 500 # of IMD = (6 x 200)/1500 = 0.8 => 1 # of ISM = (6 x 100)/1500 = 0.4 => 1 200 300 # of EMD = (6 x 500)/1500 = 2.0 => 2 # of ELD = (6 x 200)/1500 = 0.8 => 1 # of ESM = (6 x 300)/1500 = 1.2 => 1 200 # of EMM = (6 x 200)/1500 = 0.8 => 1 17 SCAMPI Sampling Rules New Terminology with MDD v1.3 Basic Units Sampling Factors Subgroups Coverage Rules. 18 SCAMPI Sampling Rules Coverage Rules Three flavors: –Coverage Rules for Process Areas –Coverage Rules for Basic Units (whoops! – Projects) –Coverage Rules for Support Functions. 19 SCAMPI Sampling Rules Cover Rules for Process Areas PA Coverage Rule 1 (in for a penny, in for a pound) –Evaluation of a given PA must cover all practices, both specific and generic –There is a wee bit of wiggle room here as the rule includes the words, “…for which they supply data” PA Coverage Rule 2 (no need to keep pounding) –If a PA is performed in a consistent way across subgroups, evaluation of that PA may be consolidated –For example, if CM, DAR, PPQA, and RSKM are performed in the same manner across all projects, then they need not be evaluated for all 6 subgroups. 20 SCAMPI Sampling Rules Project Coverage Rules Project Coverage Rule 1 – For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area This sampled project shall provide data for all process areas Project Coverage Rule 2 – For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area Project Coverage Rule 3 – For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area. 21 SCAMPI Sampling Rules ATLAS #19a Can you apply what you’ve learned? –Well, heard? Let’s find out… Work individually or together to answer the questions in ATLAS #19a –There is no need to provide comments. 22 SCAMPI Sampling Rules Cover Rules for Process Areas PA Coverage Rule 1 –Evaluation of a given PA must cover all practices, both specific and generic –HINT: Start with the question, “Which projects contribute to its fulfillment?” 23 SCAMPI Sampling Rules Project Coverage Rules Project Coverage Rule 1 – For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area This sampled project shall provide data for all process areas Project Coverage Rule 2 – For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area Project Coverage Rule 3 – For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area. 24 SCAMPI Sampling Rules ATLAS #19a Results ATLAS #19a was distributed to all SCAMPI lead appraisers and only SCAMPI lead appraisers 61 SCAMPI lead appraisers responded –That’s about a 12% response rate –But it’s still more than “one expert’s opinion” Let’s see how YOU did compared with those responsible for REALLY understanding this stuff… 25 SCAMPI Sampling Rules Question 1 1. Is Process Area Coverage Rule 1 fulfilled? <Evaluation of a given PA must cover all practices, both specific and generic.> 36% Yes / 64% No Which project(s) contribute to its fulfillment: P1? 82% Yes / 17% No P2? 90% Yes / 10% No P3? 74% Yes / 24% No P4? 7% Yes / 93% No. 26 SCAMPI Sampling Rules Question 2 2. Is Project (BU) Coverage Rule 1 fulfilled? “For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area” 34% Yes / 66% No Which project(s) contribute to its fulfillment: P1? 42% Yes / 56% No P2? 14% Yes / 86% No P3? 4% Yes / 96% No P4? 4% Yes / 96% No. 27 SCAMPI Sampling Rules Question 3 3. Is Project (BU) Coverage Rule 2 fulfilled? “For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area” 51% Yes / 49% No Which project(s) contribute to its fulfillment: P1? 54% Yes / 46% No P2? 97% Yes / 3% No P3? 22% Yes / 78% No P4? 18% Yes / 82% No. 28 SCAMPI Sampling Rules Question 4 4. Is Project (BU) Coverage Rule 3 fulfilled? “For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.” 48% Yes / 52% No Which project(s) contribute to its fulfillment: P1? 94% Yes / 6% No P2? 98% Yes / 2% No P3? 93% Yes / 7% No P4? 25% Yes / 75% No. 29 SCAMPI Sampling Rules Optical Illusion – 2 Images in One 30 SCAMPI Sampling Rules Modified Coverage Rules - #1 PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. The Basic Unit and Support Function coverage rules indicate the types of objective evidence that must be provided for each practice. Project Coverage–1: For each subgroup, both artifacts and affirmations shall be provided for each practice for at least one project. Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for each practice for at least one PA Project Coverage–3: For all sampled projects in each subgroup either artifacts or affirmations shall be provided for each practice for at least one PA. 31 SCAMPI Sampling Rules Modified Coverage Rules - #2 PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. [No change] Project Coverage–1: For each subgroup, some combination of both artifacts and affirmations shall be provided for at least one project. Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, some combination of both artifacts and affirmations shall be provided for at least one PA Project Coverage–3: For all sampled projects in each subgroup either artifacts and/or affirmations shall be provided for at least one PA. 32 SCAMPI Sampling Rules Conclusion The new sampling and coverage rules are intended to provide a truly representative set of projects that reflect how work is performed in the organization –There are some issues, however: They only solve the “proportional problem,” not the “how much is enough” problem There are multiple ways to interpret the rules as written, arriving at significantly different results Based on ATLAS #19a, the SEI has committed to clarify, or even re-write the coverage rules With respect to brain function, covering this topic is equivalent to drinking 6 beers; please drive carefully! 33 SCAMPI Sampling Rules Questions??? Pat O’Toole Process Assessment, Consulting & Training (PACT) PACT.otoole@att.net www.PACTCMMI.com If anyone is interested in receiving my monthly “Do’s and Don’ts” articles, either: Give me a business card Send me an email. 34 SCAMPI Sampling Rules