OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Metrics
in the context of the CMM/SPICE
SPIN-UK, 29 September 1998
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Some views on measurement…
• “What is not measurable, make measurable.”- Galileo
• “When you can measure what you are speaking about and express it in
numbers you know something about it; but when you cannot measure
it, when you cannot express it in numbers your knowledge is of a
meagre and unsatisfactory kind.” - Lord Kelvin
• “…This remark [above] has been quoted with an approval it does not
altogether deserve - it does not, for example, do justice to the work of
Darwin or Pasteur, to name only two of Kelvin’s contemporaries.” -
G.U.Yule (statistician)
• “You can’t control what you can’t measure.” - DeMarco
• “Metrics is crap.” - Dijkstra
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
What is measurement?
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
mathematical world
measurement
Formal
relational
system
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
mathematical world
measurement
Formal
relational
system
mathematics
and
statistics
Results
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
mathematical world
measurement
Formal
relational
system
mathematics
and
statistics
Relevant
empirical
information
interpretation
Results
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
mathematical world
measurement
Formal
relational
system
decisions
and
actions
Relevant
empirical
information
mathematics
and
statistics
interpretation
Results
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Characteristics...
• good measurement should not (in general) conflict with
empirical insights
• sound measurement systems reflect the level of understanding
of those things that are measured
• good measurement encourages improved understanding
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
real world
Empirical
relational
system
better
decisions
and
actions
Relevant
empirical
information
mathematical world
refined
measurement
Formal
relational
system
mathematics
and
statistics
more
sophisticated
interpretation
Results
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Measurement - state of practice…
• can be self serving, but increasing awareness of role
• often not well founded, but effort is invested
• data collected but often not valued (even when useful)
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Effective measurement require an appreciation of...
• measurement fundamentals
– how do we measure?
• context
– what do we need to measure?
– what can we measure?
• soft (human) factors
– why should I measure/be measured?
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Measurement within the Capability Maturity Model:
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
The Capability Maturity Model
Optimizing
Managed
Defined
Repeatable
Initial
Organizational abilities are
described as five levels
of increasing capability
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
maturity level
key process areas
common features
OPF
Defined
OPD
ISM
IGC
PR
TP
SPE
commitment
activities
verification
ability
M&A
measurement and analysis
in each KPA from levels 2 to 5
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Measurement within the CMM...
• is a common feature of every key process area at each
described maturity level
• measurement and analysis common feature is least well
described
– SEI recognize need for organizations to derive their own measures
– SEI say nothing about how to measure
• in practice we tend to find measurement and analysis is the last
common feature to be satisfied for any given KPA
– can use this as an informal assessment ‘short cut’.
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
...Measurement within the CMM...
• When investigating measurement capability look for...
– a complementary view of quality throughout the organization
– recognition of why data is collected, and understanding of
definitions and analysis
– data verification (accuracy known) and measures validated
– information used to make decisions and direct actions
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
...Measurement within the CMM
• Level 4
– measure products and process quantitatively
• Software Quality Management
• Quantitative Process Management
– aspires to production engineering (cf project engineering)
– still controversial
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Measurement within SPICE (ISO 15504)
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
The SPICE Architecture
CUS
ENG
PROJ
SUPP
ORG
Proc 1
Practice 1
Proc 2
Practice 2
Eng. 3
Develop S/W Design
Practice 3
Proc 4
Eng 3.4 Establish
traceability
Proc 4
Proc 5
Process
5
4
Well
Capability
0
1
2
defined
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
A comparison of the CMM and SPICE
• SPICE...
– is more complex
•
this is a good thing and a bad thing
– it may be unrealistic
– but does require thought
– is structured by process
• ...not capability - so may make arbitrary target setting more difficult
• allows all processes to be addressed to some extent (c.f. CMM’s SPE)
– has process capability defined by generic processes
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Measurement within SPICE...
• Measures required throughout (like CMM)...
– Generic Practices (define process capability of every process)
• level 2 - plan process (I.e. estimation)
• level 2 - track with measurement
– note - “use of measurement implies measures have been defined
and selected and data has been collected”
• level 3 - use well defined data
• level 4 - measurement oriented (like CMM level 4)
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
...Measurement within SPICE...
• ...Measures required throughout (like CMM)
– Processes
• engineering 2.2 hints at ISO9126
• project 2.5 and 7.2 - size, effort,cost schedule resources risks
• project 5.2 - define quality metrics
• organization 2.9 - define process measures
• organization 2.12 - set performance expectations
• organization 3.6 - define measures of impact
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
...Measurement within SPICE
• introduced at low capability level (like CMM)
• states expectation of what measurement requires (unlike CMM)
– (even at low capability level)
• minimal guidance (examples) on what to measure (less than
CMM)
• implies a distinct capability to define and assess measures
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
Summary
• Measurement
• within the Capability Maturity Model
• within SPICE (ISO 15504)
OXFORD SOFTWARE ENGINEERING
Software Engineering Services & Consultancy
C.C.Shelley
shelley@osel.netkonect.co.uk
+44 (0) 1993 700878
OXFORD
S O F TWAR E E N G I N E E R I N G
LIMITED
www.osel.co.uk