INFO 630
Evaluation of Information Systems
Prof. Glenn Booker
Week 4 – GQ(I)M and
Ishikawa’s Tools
INFO630 Week 4
1
www.ischool.drexel.edu
Why Care About Measurement?
• To quote Albert Einstein, "Not everything
that counts can be counted and not
everything that can be counted counts."
• We are seeking to identify things 1) which
can be counted, and 2) which count
toward achieving our goals
INFO630 Week 4
2
www.ischool.drexel.edu
Reasons for Measurement
• Measurements are required by all major
process and quality models (CMM, ISO
9000, etc.) for four good reasons:
– To Characterize or understand the current
status of activities or products
– To compare that understanding to our
objectives, and Evaluate whether the current
status is “good” or “bad”
INFO630 Week 4
3
www.ischool.drexel.edu
Reasons for Measurement
– To Predict future performance, based on past
trends
– To form the basis for measuring Improvement
• You won’t know if you improved if you don’t know
where you started!
INFO630 Week 4
4
www.ischool.drexel.edu
Where do we get Information?
• Metrics are often used to support decision making (the
Evaluate step in the previous slide)
• Decisions should be based on quantified information
• To get that information, we calculate measures from raw
data called data elements
• The data elements each come from a data source
Data
Source
Data
Elements
Measures
INFO630 Week 4
Information
5
www.ischool.drexel.edu
How do we Choose What
to Measure?
• A commonly used method for selecting
measurements is called GQ(I)M for Goal,
Question, Indicator, and Measurement
– It is based on GQM work by Victor Basili
(first reported circa 1988-89)
INFO630 Week 4
6
www.ischool.drexel.edu
How do we Choose What
to Measure?
• The GQ(I)M method uses ten steps to describe
measurements systematically
• The steps don’t have to be followed in the
order presented
– Their main purpose is to help ensure that
measurements have been fully thought out, so you
can start at the top, or the bottom, or the middle
– We’ll discuss them in a top-down approach, just to
be linear
INFO630 Week 4
7
www.ischool.drexel.edu
1. Identify Business Goals
• These are big, vague, lofty desired
accomplishments or objectives for
the organization:
– Reduce cycle time
– Improve customer satisfaction
– Develop detailed process history
– Respond to changing business environment
INFO630 Week 4
8
www.ischool.drexel.edu
1. Identify Business Goals
– Reduce overhead
– Improve competitive position
– Increase market share
– Improve product quality
• Think of what you’d find cited in a company’s
annual report – those are often high level
business goals
INFO630 Week 4
9
www.ischool.drexel.edu
2. Identify Desired Knowledge
• Break down each goal into products,
resources, and activities (processes)
needed to meet that goal
• Think of questions like:
– What activities do I manage or execute?
– What do I want to achieve or improve?
– To meet this, I will need to …
INFO630 Week 4
10
www.ischool.drexel.edu
3. Identify Subgoals
• Set Subgoals (objectives) for each
of the entities you manage
• What do you want to know about the
results of step 2? What kind of information
is important to you?
• How big, fast, expensive, complex, or
much time will a process, product, tool, or
resource take?
INFO630 Week 4
11
www.ischool.drexel.edu
4. Identify Entities and Attributes
• Formulate questions to identify entities
(document, product) created by your
process, and the attributes of them you
are interested in (size, quality, duration,
cost)
– Entities in this sense are often part
of the inputs, outputs, or process associated
with an activity
INFO630 Week 4
12
www.ischool.drexel.edu
4. Identify Entities and Attributes
• Don’t get too detailed at this point - just
identify the type of information desired
(what are you studying), and the subject of
that information (what about it do you want
to know)
– Then once the entities have been described,
find the particular characteristics of interest
(attributes)
INFO630 Week 4
13
www.ischool.drexel.edu
4. Identify Entities and Attributes
Input
(receives)
(products) entities
(traits)
attributes
<Process name>
Consists of…
Holds...
Output
(produces)
entities
entities
attributes
attributes
INFO630 Week 4
14
www.ischool.drexel.edu
5. Define Measurement Goals
• Form structured statements of the
measurement goals for each attribute
– This step is the heart of defining a metric in the
form of a sentence
• Two types of goals:
– Active goals: reduce or improve something
– Passive goals: identify, assess, understand
INFO630 Week 4
15
www.ischool.drexel.edu
5. Define Measurement Goals
• Lower maturity organizations start with passive
goals, then work on active goals
– First measure existing trends, before predicting
improvements
• Description of a measurement goal includes
the target entity, a purpose, a perspective, the
environment and constraints
INFO630 Week 4
16
www.ischool.drexel.edu
5. Define Measurement Goals
• It should include the quantity to be measured,
the active or passive verb, describe the
independent variable (e.g. time), and for active
goals, quantify a desired amount or level
– DO NOT report traits of individual people (fear
of judgment) unless it’s a known part of their job
INFO630 Week 4
17
www.ischool.drexel.edu
5. Define Measurement Goals
• Also need to balance how often measurements
are made
– More frequent measurement gives finer
control, but excessive measurement wastes
time and slows the process being measured
• Can measure per release, per component,
or some basis other than time
INFO630 Week 4
18
www.ischool.drexel.edu
5. Define Measurement Goals
• Examples of measurement goals:
– Passive: “Measure the number of requirements which
changed each month.”
“Identify the voluntary turnover rate per month for
programmers and software engineers.”
– Active: “Reduce the defect rate of developed code
over time to under 20 defects/KLOC”
“Improve the percent of satisfied customers after 30
days of product use to 95% or more”
INFO630 Week 4
19
www.ischool.drexel.edu
5. Define Measurement Goals
• The general format is:
<verb> <measure> <qualifier(s)> [objective]
• Where the <qualifiers> indicate the scope
or time frame of the measurements (a.k.a.
independent variables), and [objective] is
only given for active measurement goals
INFO630 Week 4
20
www.ischool.drexel.edu
6. Quantify Questions and
Indicators
• Pose questions to address your measurement goals
(quantifiable ones, if addressing active goals)
– Active: “Can we resolve customer emergencies, on
average, in under 24 hours?”
– Passive: “How many requirements do we have at the
end of the Requirements Definition phase?”
INFO630 Week 4
21
www.ischool.drexel.edu
6. Quantify Questions and
Indicators
• Sometimes the search for a meaningful
metric starts with a question, and that
leads to filling out the GQ(I)M from the
middle
• Identify indicators to show the
results effectively, such as:
– Pie charts
– Bar graphs
– Scatter plots, etc. (see Ishikawa later)
INFO630 Week 4
22
www.ischool.drexel.edu
7. Identify Data Elements
• Identify the data elements and equation(s)
needed to prepare (calculate) each
indicator
– E.g. “defect rate by module each month”
needs a table of defects found, for the last
month, with the module each came from
• Determine the source for each data
element
INFO630 Week 4
23
www.ischool.drexel.edu
8. Define Measures
• Describe how the data elements will be
used to produce the indicators (i.e. what is
the equation for the measure)
– Even the most obvious measure could be
defined many ways, so state the definition of
each measure clearly
• E.g. does your ‘turnover rate’ include all project
employees, or only salaried ones?
INFO630 Week 4
24
www.ischool.drexel.edu
8. Define Measures
• Define exactly what you mean by each
measure - use a checklist if needed to
show what is and isn’t included in its
definition
– Include rules, assumptions, constraints, and
environment
– Cite source if an unusual measure is used; or
if you made it up, explain why
INFO630 Week 4
25
www.ischool.drexel.edu
An Aside
• The first eight steps of the GQ(I)M
approach define traceability from the
business goals to the exact definition of
each measure, and the way it is calculated
from its data elements
• The last two steps focus on the broader
issue of planning a measurement
approach for a project
INFO630 Week 4
26
www.ischool.drexel.edu
9. Identify Measurement
Implementation
• Analyze what measures are currently
collected (if anything) and how they’re
being used
• Diagnose how well the current
measurements meet your goals – ask
‘what’s missing?’
• Act on implementing new measurements,
possibly in a phased approach based on
priorities
INFO630 Week 4
27
www.ischool.drexel.edu
10. Prepare Measurement Plan
• Take all of the aforementioned information
and create a complete plan to identify and
implement measurement for your
organization or project
• This is generally called a Metrics Plan or a
Measurement Plan
INFO630 Week 4
28
www.ischool.drexel.edu
Summary of Core Steps
•
•
•
•
•
•
•
Goal (the big picture this is describing)
Subgoal (objective; why collect this metric)
Question(s) (answered by this metric)
Indicator (how display metric)
Measurement (the actual metric and its definition)
Data Elements (used to calculate various metrics)
Source (of each data element)
INFO630 Week 4
29
www.ischool.drexel.edu
Indicators
• Indicators are the means used to present
measures, such as charts, graphs, etc.
– (yes, ‘indicator’ is an odd term for it, but
pretend it makes sense)
INFO630 Week 4
30
www.ischool.drexel.edu
Indicators
• To choose a good indicator, consider:
1. The Amount of Data to be presented for each
interval (e.g. one measure at a time, or five
different survey responses at once), and
2. The Number of Intervals to be shown, such
as time units, modules of code, etc.
INFO630 Week 4
31
www.ischool.drexel.edu
Indicators
• Different indicators are better at different
Amount or Number characteristics
– Consider also how your data will be presented
- in color or B/W, live or printed?
– Will your audience see pristine originals, or
will it be copied a zillion times?
INFO630 Week 4
32
www.ischool.drexel.edu
Indicators
• For graphs:
– The X-axis of a graph (the horizontal line)
is the independent variable
• Is often a <qualifier> you choose before the measurement,
such as time, severity, etc.
• If X is time, describe how often measurements
are made (weekly, monthly, every release, etc.)
– The Y-axis of a graph (the vertical line) is the
dependent variable; the thing you are measuring
(the Measure)
INFO630 Week 4
33
www.ischool.drexel.edu
Pie Chart
• The lowly pie chart is good for presenting a limited amount of
information attractively
– % of customers satisfied and not satisfied
– % of defects by severity at this moment
• Amount of Data: Shows a few data points (2-10)
• Number of Intervals: One - it generally shows only one
moment in time, or one set of measurements
Sales ($M)
1.2
West
4.6
2.4
Northeast
South
Midwest
East
1.1
1.7
INFO630 Week 4
34
www.ischool.drexel.edu
Ishikawa’s Seven Basic Tools
• Developed circa 1950 for manufacturing
production
– Used widely in manufacturing quality control
– Focuses on project level concerns; is this
batch good enough to accept?
• Not very useful for research; has little
theoretical basis
INFO630 Week 4
35
www.ischool.drexel.edu
Ishikawa’s Seven Basic Tools
• From a software perspective, these tools
are often best for managers – focuses on
identifying process and/or quality control
issues
– Not typically helpful for individual developers
INFO630 Week 4
36
www.ischool.drexel.edu
1. Check sheet
• Used to gather data easily, consistently,
and in a standard format
– Helps to define key parts of a process, and
make sure they are all performed
– Examples include code inspection checklist,
detailed test procedures
• A check sheet used to help the quality of a
process or product is a “checklist”
INFO630 Week 4
37
www.ischool.drexel.edu
2. Pareto Diagram
• Used to identify problem areas - where to fix first; what are
the biggest fires to put out
• Defects tend to cluster in portions of code e.g. plot % of
defects by the type of defect (logic, data definitions, etc.), and
plot the total % of defects above it
• Must list categories in descending order of frequency
Helps look for the few
components where
most defects reside
X axis must be a
nominal variable
Defect Rate (defects/KLOC)
Pareto Diagram of Component Defect Rate
25
22.3
20
15.6
13.3
15
11.1
8.9
10
5.6
5
0
Interface
Query
Core
Import
Export
Reports
Component
INFO630 Week 4
38
www.ischool.drexel.edu
3. Histogram
No. of Problem Reports
• A bar chart is used to break down data by an ordered
category (e.g. defect severity, satisfaction rating, etc.)
• Can choose to put bars next to each other, or stack them.
Stack when they add to a constant (e.g. 100%), or when the
total is also a useful measure
• Can show limited time spans, e.g. a few
time intervals
No. of Problem Reports by Status
234
250
200
150
100
50
23
50
34
15
8
0
New
Closed
Pending Withdrawn Analysis
Testing
Status
INFO630 Week 4
39
www.ischool.drexel.edu
3. Histogram
Problem Reports in each Status over Time
Testing
400
Analysis
300
Withdrawn
200
Pending
Closed
100
New
0
Jan-02
Feb-02
Mar-02
Apr-02
May-02
Time
Problem Reports in each Status over Time
300
Stacked and
cluster bar
graphs for the
same data set
Number of Problem
Reports
Number of Problem
Reports
500
250
New
200
Closed
Pending
150
Withdrawn
100
Analysis
50
Testing
0
Jan-02
Feb-02
Mar-02
Apr-02
May-02
Time
INFO630 Week 4
40
www.ischool.drexel.edu
3. Histogram
• A true histogram uses ranges of X axis
values to show, for example, the
number of events or data points in that
range of values
Histogram
7
6
Frequency
Example shows how
many customers were
in age ranges of 0-18,
19-30, 31-40, 41-50,
51-60, and 61+
5
4
3
2
1
0
18
30
40
50
60
More
Bin
INFO630 Week 4
41
www.ischool.drexel.edu
4. Run Charts
Usually uses
a line chart
Number of Defects Found
• Purpose is to plot something important versus time
• Often compare values to a desired or target value,
especially at higher process maturity levels (CMM Level 3
and up) and for comparing to active goals
• Special case: The “S” curve plots (cumulative % completion
of something) versus time
Number of Defects Found by Severity
45
40
35
30
25
20
15
10
5
0
Minor
Major
Total
Jan-02
Feb-02
Mar-02
Apr-02
May-02
Time
INFO630 Week 4
42
www.ischool.drexel.edu
5. Scatter Diagram
CURRENT SALARY
• Is used to plot two measures
against each other to see if
there’s a correlation between
them
– E.g. Defect rate per module
versus module complexity,
or productivity versus
experience
– When we say ‘plot Blah
versus Ick,’ Blah is the Y
axis, and Ick is the X axis
INFO630 Week 4
Current Salary vs. Educational Level
60000
50000
40000
30000
20000
10000
0
6
8
10
12
14
16
18
20
22
EDUCATIONAL LEVEL
43
www.ischool.drexel.edu
5. Scatter Diagram
• If there’s enough data, can try to add
curve fitted lines – we’ll cover this in the
regression analysis discussion
• This and control charts are the most
powerful types of graphs for
understanding your data
– Many estimation rules and equations are
derived from scatter diagrams
INFO630 Week 4
44
www.ischool.drexel.edu
6. Control Chart
• Is a key Statistical Process Control tool
• Hard to apply to software development
– Specifications poorly defined
– Each project takes a long time
– Too many uncontrolled process variances
– Process-quality relationship not well defined
– Too many processes used
– Rapidly changing technology
• There are many varieties of control chart
INFO630 Week 4
45
www.ischool.drexel.edu
6. Control Chart
• Control charts are often used at very
high levels of process maturity (CMMI
Control Chart: Percent Passed
levels 4 & 5)
93.000
89.000
85.000
Percent Passed
81.000
UCL = 92.33 43
Average = 85.0833
77.000
LCL = 77.8324
1
2
3
4
5
6
7
8
9
10
11
12
Sigma level: 3
INFO630 Week 4
46
www.ischool.drexel.edu
6. Control Chart
• Pseudo-control charts include:
– The ‘u’ chart for defect rates by component, BMI,
etc. versus time
– The ‘p’ chart for percentages, such as inspection
effectiveness or customer satisfaction rating
• Upper and Lower Control Limits (UCL and LCL)
are +/- 3 sigma from the mean (average)
– Can add a warning limit at +/- 2 sigma
INFO630 Week 4
47
www.ischool.drexel.edu
7. Cause and Effect Diagram
• A.k.a. the Fishbone chart
– Is the least used Ishikawa tool, in my experience
– Is not an Indicator per se, just a tool for capturing
thoughts about a problem
• Used with brainstorming to trace the causes of some
outcome or result (good or bad)
• Ask “what causes that” or “what influences that” to
determine the major types of causes, then break them
into more detailed events
INFO630 Week 4
48
www.ischool.drexel.edu
7. Cause and Effect Diagram
• Sample fishbone diagram to analyze causes of
Incorrect Deliveries
Note the choices of types of causes (communication, skills,
transport, procedures) vary from one problem to the next
INFO630 Week 4
49
www.ischool.drexel.edu