Measurement and Analysis
Practical Software and
Systems Measurement
A foundation for objective project management
PSM
M&A - 1
COCOMO Forum
October 24, 2001
24 Oct 01
Measurement and Analysis
Outline
• Objectives of Measurement and Analysis
Initiatives
• Practical Software and Systems Measurement
• ISO/IEC 15939, Software Measurement
Process
• CMMI Measurement and Analysis Process
Area
• ISO 9000-3, Guidelines for Software
M&A - 2
24 Oct 01
Measurement and Analysis
Objectives of Measurement Initiatives
• Experienced-Based Measurement to Support
Management and Engineering Requirements
• Consistent Measurement Process Across
Measurement Initiatives
• Consistent Measurement Terminology
M&A - 3
24 Oct 01
Measurement and Analysis
Practical Software and Systems
Measurement
ISO
9000-3
INFORMATION
ISO/IEC
DRIVEN
MEASUREMENT 15939
PROCESS
CMMI
M&A
M&A - 4
24 Oct 01
Measurement and Analysis
Measurement Process Model
Measurement Requirements
Information
Needs
Technical &
Management
Processes
User Feedback
Information
Products
Core Measurement Process
Establish
Commitment
Plan
Measurement
Plan
Perform
Measurement
Experience Base
Performance
Information
Evaluate
Measurement
Evaluation
Results
Improvement Actions
Measurement Process Scope
M&A - 5
24 Oct 01
Measurement and Analysis
Practical Software and Systems
Measurement (PSM)
M&A - 6
24 Oct 01
Measurement and Analysis
What is PSM?
• A Primary Measurement and Analysis
Process Used by DoD, Government, and
Commercial Programs
• Technology Based on Actual Experience on
Real Programs
• A Fully Integrated Approach Including
Guidance, Tools, Training, Implementation
Workshops, Support Products, and a
Qualified Technical Team
M&A - 7
24 Oct 01
Measurement and Analysis
Key Measurement Concepts
• Measurement is a Process - Not a Pre-Defined
List of Graphs or Reports
• The Measurement Process is Flexible Adapted To Meet Specific Project and
Organizational Objectives and Information
Needs
• The Measurement Requirements are
Integrated Into the Project Technical and
Management Processes
M&A - 8
24 Oct 01
Measurement and Analysis
Common Information Needs
•
•
•
•
•
•
•
M&A - 9
Schedule and Progress
Resources and Cost
Product Size and Stability
Product Quality
Process Performance
Technology Effectiveness
Customer Satisfaction
24 Oct 01
Measurement and Analysis
Problem Report Status
Priority 1, 2, and 3 Problem Reports
Build 1
1000
900
Start of
Build 1 I&T
Number of Problem Reports
800
700
600
Discovered
Closed
500
400
300
200
100
0
Oct 97
Jan 98
Apr 98
Jul 98
Oct 98
Jan 99
Apr 99
30 Aug 98
M&A - 10
24 Oct 01
Measurement and Analysis
Types of Analysis
Estimation
Performance
Analysis
M&A - 11
Defined
Information
Needs
Feasibility
Analysis
24 Oct 01
Measurement and Analysis
Estimation
Project
Characteristics
Select
Approach
Project Data
Historical Data
Map and
Calibrate
Assumptions
Compute
Estimate
Objectives
Constraints
Adjustments
Evaluate
Estimate
M&A - 12
Estimates
Lack of Information
24 Oct 01
Measurement and Analysis
Integrated Analysis Model
Technology
Effectiveness
Process
Performance
Product Size
and Stability
Resources
and Cost
Schedule and
Progress
Customer
Satisfaction
M&A - 13
Product
Quality
24 Oct 01
Measurement and Analysis
Integrated Analysis Example
Test Progress
Problem Report Status
Open By Priority
1000
100
900
90
700
600
Plan
Actual
500
400
300
200
Number of Problem Reports
Number of Requirements
800
100
0
1 Jan
80
70
60
High
Medium
Low
50
40
30
20
10
22 Jan
12 Feb
5 Mar
26 Mar
0
1 Jan
16 Apr
22 Jan
12 Feb
5 Mar
26 Mar
16 Apr
CPU Utilization
Software Reliability
100
120
90
100
80
Percent of CPU Cycles
Hours Between Failures
Threshold
80
60
40
70
60
50
Threshold
40
30
20
20
10
0
1 Jan
M&A - 14
22 Jan
12 Feb
5 Mar
26 Mar
16 Apr
0
1 Jan
22 Jan
12 Feb
5 Mar
26 Mar
16 Apr
24 Oct 01
Measurement and Analysis
Multi-Level Requirements
Enterprise
Management
•
•
•
•
Performance Measurement
Normative Performance Baselines
Technical and Business Policy
Investment Decisions & Analysis
Organizational
Management
•
•
•
•
Process Improvement
Program Planning Guidelines
Performance Based Acquisition
Organizational Norms & Benchmarks
Program
Management
•
•
•
•
Program Estimation and Planning
Program Performance Tracking
Program Tradeoff Analysis
Resource Management
M&A - 15
Risk
Management
Process
Issue-Driven
Measurement
Process
24 Oct 01
Measurement and Analysis
PSM Project Strategy
Government - Industry Academia Team
Proven Measurement
Technical Guidance
and Tools
Technical Consensus
of Best Practices
PSM
Project
Approach
Incremental Product
Development and
Improvement
Empowered Transition
Organizations
Comprehensive
Transition Support
M&A - 16
Integrated With Other
Key Initiatives
24 Oct 01
Measurement and Analysis
New Technology Areas
• Measurement and New Technologies
-
Object-oriented developments
Spiral/evolutionary approaches
Interoperability
Product lines
• Initial Recommendations Were Draft Drafted
in Early 2001
• Field Trials Are Underway - Additional Trials
Will be Conducted in 2002
M&A - 17
24 Oct 01
Measurement and Analysis
PSM Participating Organizations
Overseas
DoD and Government
• US Air Force AFMC
• US Air Force SSSG
• US Air Force STSC
• US Air Force CRSIP
• US Air Force STRATCOM
• US Army AMC
• US Army ATEC
• US Army CECOM
• US Army ODISC4
• US Army SMDC
• US Army TACOM - ARDEC
• USMC MCTSSA
• US Navy AEDC
• US Navy ASN RDA
• US Navy FMSO
• US Navy NAVAIR
• US Navy NAVSEA
• US Navy NAWC
• US Navy NRL
• US Navy NSWC
M&A - 18
• US Navy NUWC
• US Navy OPTEVFOR
• ODUSD (S&T)
• OSD C31
• OSD PA&E
• Aerospace Corp.
• DCMA
• DFAS
• DISA
• DLA
• DSMC
• FAA
• IDA
• MITRE Corp.
• NDU IRM College
• NASA
• National Park Service
• Sandia National Lab
• SEI
• Soc. Sec. Admin.
• Defence Materiel Org.
(Australia)
• LiveWare (Argentina)
• NRC (Canada)
• Software and Systems
Engineering (Israel)
• Software Improvements
(Australia)
• Software Quality Institute
(Australia)
24 Oct 01
Measurement and Analysis
PSM Participating Organizations
Industry
• ACS GSG
• American Sys. Corp.
• Averstar
• Bloodworth Int. Tech.
• Carnegie Mellon Univ.
• CSC
• CTA
• Distributive Software
• Federal Reserve Bank
• General Dynamics
• General Scientific Corp.
• Graeme & Garland
• GTE
• Hawaiian Electric
• IEEE
• IFPUG
• IITRI
• INCOSE
M&A - 19
• Independent
Engineering
• ITABHI
• James Gregory Assoc.
• Kodak Health Imaging
• Litton/TASC
• Lockheed Martin
• L3 Communications
• Northrop Grumman
• OAO Corporation
• PRC - Litton
• PRC/TASC
• QSM
• Quality Plus Tech.
• Raytheon - Hughes
• Robbins, GIOIA
• Rockwell Collins
• SAIC
• Sallie Mae
• Sentel
• Software Metrics
• SPC
• Sverdrup Tech.
• Tecolote Research
• TeraQuest Metrics
• Tivoli
• TRW
• United Defense
• Univ. Southern CA
• User Trust Network
• US West
• VPI - State University
• West Virginia High Tech.
Cosortium
• West Virginia University
24 Oct 01
Measurement and Analysis
M&A - 20
24 Oct 01
Measurement and Analysis
ISO/IEC 15939, Software
Measurement Process
M&A - 21
24 Oct 01
Measurement and Analysis
ISO/IEC 15939
• PSM Was Base Document for This Standard
• Specifies Required Activities and Tasks of a
Measurement Process
• FCD Ballot Passed in May 2001 With Only 1
Dissenting Vote - All Comments Resolved at
ISO Plenary
• Terminology Was Adjusted for Consistency
With ISO/IEC 9126, 15498, and 14143 Series
• DIS Ballot Will be Released in September
2001 - Ballot Will Close in November 2001 International Standard in Late 2001
M&A - 22
24 Oct 01
Measurement and Analysis
Basic Measurement “Architecture”
ISO/IEC
15939
Measurement
Information
Model
Measurement
Process
Model
Information Driven
Measurement
Requirements
Consistent, Tailorable
Implementation
Process
M&A - 23
24 Oct 01
Measurement and Analysis
Measurement Information Model
Information
Needs
Measurable
Concept
Information
Product
Measurement
Construct
Entities
Attribute
M&A - 24
24 Oct 01
Measurement and Analysis
Information
Needs
Indicator
Analysis
Model
Derived
Measure
Meas.
Function
Quantity Defined as a
Function of Two or More
Measures
Algorithm Combining Two or More
Measures
Base
Measure
Meas.
Method
Meas.
Method
M&A - 25
Algorithm Combining Measures and
Decision Criteria
Derived
Measure
Base
Measure
Attribute
Measure that Provides an Estimate or
Evaluation With Respect to Information
Needs
Attribute
Measure of a Single Attribute Defined
By a Specified Measurement Method
Operations Mapping an Attribute to
a Scale
Property Relevant to
Defined Information Needs
Entities
24 Oct 01
Measurement and Analysis
Information Product
Information
Needs
Information
Product
Decision
Criteria
Indicator
Analysis
Model
Base
Measure
M&A - 26
Measure that Provides an Estimate or
Evaluation With Respect to Information
Needs
Algorithm Combining Measures and
Decision Criteria
Derived
Measure
24 Oct 01
Measurement and Analysis
CMMI - Measurement and Analysis
(M&A)
M&A - 27
24 Oct 01
Measurement and Analysis
CMMI M&A
• ISO/IEC 15939 Was Base Document for the
Measurement and Analysis Process Area
• CMMI M&A Describes How to Evaluate a
Measurement Process
• M&A Is a Level 2 Process Area
• Pilot Assessments Are Being Conducted in
2001
M&A - 28
24 Oct 01
Measurement and Analysis
ISO 9000-3
Quality management and quality assurance
standards - Part 3: Guidelines for the
application of ISO 9001 to the development,
supply, installation, and maintenance of
computer software
M&A - 29
24 Oct 01
Measurement and Analysis
ISO 9000-3
• 9000-3 Is Being Updated by SC7 as Part of the
ISO 9000:2000 Update
• Part 8, “Measurement, Analysis and
Improvement”, Was Updated for Consistency
With 15939
• Standard Is at Working Draft Stage - Expected
Publication Date Is 2002/2003
M&A - 30
24 Oct 01
Measurement and Analysis
For More Information
PSM Support Center
Cheryl Jones - PSM Project Manager
TACOM - ARDEC
AMSTA-AR-QAT
Building 62
Picatinny Arsenal, NJ 07806-5000
(973) 724-2644 (Voice)
(973) 724-2382 (FAX)
cljones@pica.army.mil
PSM Home Page: http://www.psmsc.com
M&A - 31
24 Oct 01