INFO 630
Evaluation of Information Systems
Dr. Jennifer Booker
Week 3 – Quality Management
and Earned Value
INFO630 Week 3
1
www.ischool.drexel.edu
Quality Management System
• Every organization should have a Quality
Management System (QMS)
– The QMS describes the quality organization
(members, duties, and to whom they report) and
activities which they need to perform
– Quality Assurance, Quality Management, and Quality
Control are often used interchangeably for the QMS’
name
INFO630 Week 3
2
www.ischool.drexel.edu
Typical QMS Activities
• Activities by a quality management organization
typically include
– Process Evaluation – to determine if process is clear,
and consistent with other related processes
– Process Assessment - help ensure processes meet
quality standards (CMMI, ISO 9000)
INFO630 Week 3
3
www.ischool.drexel.edu
Typical QMS Activities
– Process Verification – to ensure that processes are
actually being used
– Product Evaluation – to inspect documents or other
products (SW, HW) to verify
their quality
• Think of the tag ‘Inspected by 19’ in new clothes – that’s a
product evaluation
INFO630 Week 3
4
www.ischool.drexel.edu
QMS Activities
• Quality control for manufacturing,
assembly, shipping, receiving, etc.
• Support definition, collection, and analysis
of metrics for each organization
• The quality organization should be able to
stop any activity if the product quality is
inadequate (what defines inadequate?)
INFO630 Week 3
5
www.ischool.drexel.edu
Quality Road Map for
Software Maintenance
• An organization’s Quality Management
System may be guided by some sort of
Quality Road Map (high level plan)
• The Quality Road Map is the basis for
each release’s Quality Plan
• The Road Map is the highest level
corporate strategy for improving the
product
(Warning – IBM terminology here…)
INFO630 Week 3
6
www.ischool.drexel.edu
Quality Road Map
• The Road Map should systematically introduce more
comprehensive quality activities and measures
• Let each team develop own quality goals
• Consider process, product, people, and tools; how
will each contribute to quality?
– FAA uses 3-year overall plan, with one-year goals to
help meet it
INFO630 Week 3
7
www.ischool.drexel.edu
Supplier Buy-in
• Your quality efforts must be matched by
your suppliers
• This must be stated in the Quality Plan or
equivalent
• Hence the quality measurements need to
fulfill the overall goals of the organization,
and define objectives for each release or
product, and for major suppliers
INFO630 Week 3
8
www.ischool.drexel.edu
Tracking, Measurement,
Analysis
• These are key to making quality
management work
• Software development must become
quantitatively predictable to be regarded
as a science
• Large volume of data not needed; instead,
focused, accurate, useful data is better
INFO630 Week 3
9
www.ischool.drexel.edu
Customer Problems
• Problems include defects, user errors,
usability problems, unclear instructions,
duplicates of known problems, etc.
• Usually express as Problems per User
Month (PUM)
– PUM = (Problems reported) /
(total license-months)
INFO630 Week 3
10
www.ischool.drexel.edu
Customer Satisfaction
• Often measured with 5-point Likert scale
(with optional Net Satisfaction Index
(NSI)):
– Very Satisfied (NSI = 100%)
– Satisfied (NSI = 75%)
– Neutral (NSI = 50%)
– Dissatisfied (NSI = 25%)
– Very Dissatisfied (NSI = 0%)
INFO630 Week 3
11
www.ischool.drexel.edu
Customer Satisfaction
• Note that average NSI is a weak metric, since it
hides the type of distribution
• The average NSI for half Very Satisfied and half
Neutral customers looks the same as for all Satisfied
– Like measuring the mean of a distribution, without
measuring the standard deviation (width of the bell
curve)
INFO630 Week 3
12
www.ischool.drexel.edu
In-Process Quality Metrics
• Defect density during automated (machine) system
testing
– Good for comparing different releases of
one product, or very similar products
• Defect arrival rate
– # of defects found vs. time (e.g. week, month)
– Cumulative defects found vs. time
– Look for stability to decide done with testing
INFO630 Week 3
13
www.ischool.drexel.edu
In-Process Quality Metrics
• Clarify actual defects, vs. reported
problems, vs. backlog of defects
• Actual defects measures how many known
defects have been found in
the code
• Reported problems includes code defects,
user errors, etc.
• Backlog shows how fast defects
fixed, as well as found
INFO630 Week 3
14
www.ischool.drexel.edu
In-Process Quality Metrics
• “Phase-based defect removal pattern”
tracks defects during the entire life cycle
• Ways to identify defects include
requirements and design reviews, code
inspections, formal verification, and all
phases of testing
• Want most defects found early in project
(key theme)
INFO630 Week 3
15
www.ischool.drexel.edu
Software Maintenance Metrics
• Focus on fixing existing problems, not
changing features or requirements
– Fix backlog = number of outstanding known
problems
– Backlog Management Index = (# closed
problems) / (# new problems) * 100
• BMI > 100 means backlog of problems
is being reduced
INFO630 Week 3
16
www.ischool.drexel.edu
Software Maintenance Metrics
• New, closed, and the status of problems can also
be tracked
• Fix response time, measured by the
“mean time problems are open”
• Percent delinquent fixes = % of problems which
exceeded desired response time
– Note that this implies an active goal, since the
expected fix time has been defined
INFO630 Week 3
17
www.ischool.drexel.edu
Fix Quality
• Fixes are, in general, not all perfect - they
may create new problems!
– Fix quality = % of fixes which are defective
(i.e. result in new problems)
– Fix propagation rate = average number of
new problems resulting from a fix
INFO630 Week 3
18
www.ischool.drexel.edu
What is Earned Value?
• Earned Value is a method for planning and
tracking the cost, schedule, and technical
accomplishments of a project
• Earned Value focuses on comparing the
ACTUAL cost, schedule, and technical
accomplishments to their BUDGETED (or
planned) values
INFO630 Week 3
19
www.ischool.drexel.edu
Earned Value
Cost
Earned Value
balances
measurement
of cost,
schedule, and
technical
performance.
Earned
Value
Schedule
INFO630 Week 3
Technical
20
www.ischool.drexel.edu
Why do we care?
• Some organizations are contractually
required to use earned value to report
their progress
• Payments are then based on the
accomplishments claimed under earned
value; hence earned value reporting
claims can be legally binding!
INFO630 Week 3
21
www.ischool.drexel.edu
Foundation
• Earned value requires having a written
plan which describes each task, before
that task is performed.
• The plan must define, for each task:
– Schedule (start, stop, and calendar duration)
– Cost (or earned value) it is worth, and
– Earned value measurement method (see
later)
INFO630 Week 3
22
www.ischool.drexel.edu
Budget Basics
• The total amount of money available on a
contract is the Contract Budget Base
(CBB)
• Wise managers leave a Management
Reserve (MR) of about 10-15%, and
budget the rest to the project’s
Performance Measurement Baseline
(PMB)
– Hence CBB = MR + PMB
INFO630 Week 3
23
www.ischool.drexel.edu
Budget Basics
• Most mortals (project staff) are only aware
of the PMB, and do not know the extent of the MR
– The PMB may be broken into the undistributed
budget (a more visible reserve than MR), plus
distributed budget (which is the sum of all tasks’
cost accounts)
– PMB = undistributed budget + distributed budget
INFO630 Week 3
24
www.ischool.drexel.edu
Budget Basics
Contract Budget Base (CBB) - all money available to project
Performance Measurement Baseline (PMB)
Management
Reserve
(MR)
Performance Measurement Baseline (PMB)
Distributed Budget (maps to WBS and
master schedule)
INFO630 Week 3
Undistributed
Budget
25
www.ischool.drexel.edu
Earned Value Terminology
Data Element
Term
Acronym
Scheduled Work
Budgeted Cost for Work Scheduled
BCWS
Earned Value
Budgeted Cost for Work Performed
BCWP
Actuals
Actual Cost of Work Performed
ACWP
Authorized Work
Budget At Completion
BAC
Forecasted Cost
Estimate At Completion
EAC
Work Variance
Schedule Variance
SV
Cost Variance
Cost Variance
CV
Completion Variance
Variance At Completion
VAC
INFO630 Week 3
26
www.ischool.drexel.edu
The Plan
• The project plan produces a curve for the
BCWS (budgeted cost of work
scheduled) through its planned
completion. This curve is typically Sshaped, and is plotted as cost versus
time (e.g. $K versus months).
• The total expected cost of the task,
located at the end of the BCWS curve, is
the BAC (budget at completion)
INFO630 Week 3
27
www.ischool.drexel.edu
Budgeted Cost of Work Scheduled
BCWS
3.5
BCWS ($K)
3
BAC
2.5
2
1.5
1
0.5
0
0
2
4
6
8
10
12
14
Month
INFO630 Week 3
28
www.ischool.drexel.edu
Tracking Progress
• As the task is being performed, measure
the amount of money actually being
expended (ACWP, actual cost of work
performed)
• Keep track of the tasks which have
actually been worked on, and look up how
much earned value you get credit for that
work (BCWP, the budgeted cost of work
actually performed)
INFO630 Week 3
29
www.ischool.drexel.edu
Tracking Progress
• Plot the ACWP, BCWP, and BCWS over time
• Notice that ACWP and BCWP can only be shown to
date, whereas the BCWS covers the entire project’s
schedule
– BCWS is the original project plan
– ACWP is how much real money has been spent
– BCWP is how much real work has been
accomplished with that money
INFO630 Week 3
30
www.ischool.drexel.edu
Earned Value Example
Actual Expenses and Accomplishments
3.5
3
$K
2.5
BCWS
2
ACWP
1.5
BCWP
1
Now
0.5
0
0
5
10
15
Month
INFO630 Week 3
31
www.ischool.drexel.edu
Analysis
• At any time during the project:
– Cost Variance (CV, in $) = ACWP - BCWP
– Schedule Variance (SV, $) = BCWS - BCWP
– Schedule Slip (time) = (current date) minus (date
when current value of BCWP was found on the
BCWS curve)
• Positive numbers are bad for all of these; negative
is good
INFO630 Week 3
32
www.ischool.drexel.edu
Projection
• The ACWP curve is used to estimate
when the task will actually be completed
• The projected ACWP curve is called
Estimate To Completion (ETC)
• The end of the projected ETC curve is
the Estimate At Completion (EAC)
– That’s the expected end of the project (in $
and schedule)
INFO630 Week 3
33
www.ischool.drexel.edu
Projected Project Completion
Projected Completion
$K
4
3.5
3
2.5
BAC
EAC
BCWS
ACWP
2
1.5
1
0.5
0
BCWP
ETC
0
5
10
Month
15
Notice the ACWP line becomes
the ETC line after the present date
INFO630 Week 3
34
www.ischool.drexel.edu
Project Completion
• Compare BAC (the originally planned completion of
the project) and EAC (the current estimated
completion of the project)
– Projected Program Delay (schedule overrun) = Time
difference between BAC and EAC
– Variance At Completion (VAC, cost overrun) = Cost
difference between BAC and EAC
INFO630 Week 3
35
www.ischool.drexel.edu
Earned Value Summary
T/N is Time Now
6000
T/N
EAC
$
5000
ETC
VAC
4000
CBB
MR
BAC
(PMB)
3000
2000
BCWS
1000
Projected
Program
Delay
ACWP
BCWP
Cost Variance
Schedule Variance
0
Jan
Feb Mar Apr May Jun
Jul
Aug
Sep
Oct
Nov Dec
Jan
Feb Mar Apr May Jun
Schedule Slip
INFO630 Week 3
36
www.ischool.drexel.edu
Earned Value Measures
Term
Symbol
Formula
Checklist Actions
% Done
BCWP
BAC
Ratio of work accomplished in terms of the total amount of work to do.
CPI or PF
BCWP
ACWP
Ratio of work accomplished against money spent (an efficiency rating: Work
Done for Resources Expended)
To Complete
Performance Index
or Verification Factor
TCPI or VF
BAC - BCWP
EAC - ACWP
Ratio of work remaining against money remaining (Efficiency which must be
achieved to complete the remaining work with the expected remaining money)
Schedule Performance Index
SPI
BCWP
BCWS
Schedule Correlation
SC or S/C
PCUM
SV
Percent Complete
Cost Performance Index
or Performance Factor
1)
Independent Estimate
At Completion
IEAC
Average Performance
PCUM
Average Expected
Performance To Finish
P TO GO
BAC
PF
2)
BAC - BCWP
ACWP +
.8CPI + .2SPI
Ratio of work accomplished against what should have been done (Efficiency
Rating: Work done as compared to what should have been done)
Ratio of Schedule Variance (SV) in terms of average amount of work
accomplished (in weeks or months). It indicates a correlation to program true
schedule condition
Calculation of a projected Estimate At Completion to compare with the CAM's
Estimate At Completion:
1) Ration of total work to be done against experienced cost efficiency
2) Sunk costs added to a ratio of remaining work against weighted cost and
schedule efficiencies
BCWPcum
Duration (wks or mos)
Since ACWP Began
Average rate at which work has been accomplished since work began
BCWPcum
Duration (wks or mos)
From Time Now to
Manager's Stated
Completion Date
Average rate at which work must be accomplished in the future to finish on the
date the CAM has forecasted for completion of the work.
Key measures we’ll use are SPI and CPI
INFO630 Week 3
37
www.ischool.drexel.edu
More Earned Value Measures
Term
Risk Exposure
Performance Index
Risk Schedule
Performance Index
Management
Reserve Ratio
Symbol
Formula
REPI
ARWP
REWP
Ratio of work accomplished against
money spent (Efficiency Rating:
Work Done for Resources Expended)
REWS
REWP
Ratio of work accomplished against what
should have been done (Efficiency Rating:
Work done as compared to what should
have been done)
RSPI
MRR
Checklist Actions
TFA
= EAC
Ratio of Total Funds Available against
total Estimated funds expended At
Completion; or Total Funds Available
against the Budgeted cost At Completion
TFA
=
CPI
multiplied by the Cost Performance Index
BAC *
INFO630 Week 3
38
www.ischool.drexel.edu
What are “good” values?
Performance
Metric
Green
Yellow
Red
SPI
0.95 and
greater
0.90 and
greater
Less than
0.90
CPI
0.95 and
greater
0.90 and
greater
Less than
0.90
VAC
0.05 and
lesser
0.10 and
lesser
Greater than
0.10
Per the “U.S. Marine Corps Acquisition Procedures Handbook,” June 1997
INFO630 Week 3
39
www.ischool.drexel.edu
Projection
• Another method for projecting the end of
the actual project is
– Plot CPI and SPI over time
– Based on the trends seen, estimate the CPI
and SPI at the end of the project
• EAC(cost) = BAC(cost) / CPI
• EAC(schedule) = BAC(schedule) / SPI
INFO630 Week 3
40
www.ischool.drexel.edu
Example - Railroad Project
• Given the following project: build 4 miles of railroad
track in 4 weeks for $4 million
• Actual performance to date is: After 3 weeks, $2
million has been spent and one mile of track has
been built
• Pure cost analysis would only show that the project
has spent $1 million less than expected ($3M - $2M)
– Is this good?
INFO630 Week 3
41
www.ischool.drexel.edu
Example - Railroad Project
Earned Value (track miles)
Work Accomplished and Projected
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
BCWS
BCWP
ETC
0
5
10
15
Weeks
INFO630 Week 3
42
www.ischool.drexel.edu
Example - Railroad Project
Earned Value Example - Cost
Millions of Dollars
8
EAC
6
BCWS
ACWP
4
BAC
BCWP
ETC
2
0
0
4
8
12
Weeks
INFO630 Week 3
43
www.ischool.drexel.edu
Example - Status Analysis
• Today’s earned value status:
– BCWS = $3M (expected expenditure & work)
– ACWP = $2M (actual expenditure)
– BCWP = $1M (actual work accomplished)
• Calculate
– Cost Variance (CV) = ACWP - BCWP = $1M
– Schedule Variance = BCWS - BCWP = $2M
– Schedule Slip = (3 weeks) - (1 week) = 2
weeks
INFO630 Week 3
44
www.ischool.drexel.edu
Example - Status Analysis
• Earned value analysis:
– Planned $1M/week expenditure for 4 weeks
– Hence planned $3M after 3 weeks (BCWS)
– Earned value of only $1M (BCWP is 66% behind the
plan)
– Actual expenditure of $2M to date (ACWP has 100%
overrun, given how much was accomplished)
INFO630 Week 3
45
www.ischool.drexel.edu
Example - Projected Outcome
• Find EAC using SPI and CPI
• Only are given one data point for status,
so use:
– SPI = BCWP/BCWS = $1M / $3M = 0.333
– CPI = BCWP/ACWP = $1M / $2M = 0.500
INFO630 Week 3
46
www.ischool.drexel.edu
Example - Projected Outcome
• BAC was 4 weeks and $4 million
• EAC is
– EAC(cost) = BAC(cost)/CPI = $4 million /
0.500 = $8 million
– EAC(schedule) = BAC(schedule)/SPI = 4
weeks / 0.333 = 12 weeks
• Hence projected cost overrun is $4M, and
schedule overrun is 8 weeks
INFO630 Week 3
47
www.ischool.drexel.edu
Measurement Methods
• The methods used to take credit for
earned value are of key significance
• It is critical that earned value claims are
accurate and consistent
• Credit, or Earned Value, for performing
each task may be obtained in many
different ways, depending on the type of
task.
INFO630 Week 3
48
www.ischool.drexel.edu
Measurement Methods
•
•
•
•
Milestone Events
Weighted Milestone
Fixed Formula
Percentage
Complete
Estimates
• Percentage
Complete and
Milestone Gates
• Equivalent
Completed Units
• Earned Standards
• Apportioned
Relationship to
Discrete Work
• Level of Effort
INFO630 Week 3
49
www.ischool.drexel.edu
Milestone Events
• Milestone events earn 100% credit when:
– The event has occurred, AND
– All follow-up work is done (e.g. preparing and
distributing minutes after a formal review) AND
– All action items are closed
• Until all of the foregoing activities are completed,
no earned value may be taken
INFO630 Week 3
50
www.ischool.drexel.edu
Weighted Milestone
• Useful for short-span tasks
• Weighted budget amounts are applied to
milestones distributed across the duration
of the task
• As each milestone is reached, the budget
amount is earned
INFO630 Week 3
51
www.ischool.drexel.edu
Fixed Formula
• Useful for detailed short-span tasks
• An amount ranging from 0 to 100 percent of the
task's budget is earned when the task begins
• The remaining percentage is earned when the task
is complete, e.g. 0/100%, 25/75%, or 50/50%
• These measures are typically used to track earned
value on nonrecurring tasks
INFO630 Week 3
52
www.ischool.drexel.edu
Percentage Complete Estimates
• Easiest method to administer
• A "subjective" estimate of the percentage
of work completed is used for the earned value
• Requires well-defined work packages and guidelines
to determine an accurate percent-complete value
• This is the biggest source of inconsistencies in
earned value reporting
INFO630 Week 3
53
www.ischool.drexel.edu
Percentage Complete
• For example, define the following
allowable earned value for each phase of
new feature development:
– Analysis and design are ready for peer
review: up to 30% earned value
– Successful completion of peer review: 35%
– Preparation for customer review: 45%
INFO630 Week 3
54
www.ischool.drexel.edu
Percentage Complete Estimates
– Completion of customer review and approval
to proceed: 55%
– Coding and unit testing completed: 75%
– Integration testing completed: 85%
– System testing completed: 95%
– New feature released: 100%
INFO630 Week 3
55
www.ischool.drexel.edu
Percentage Complete and
Milestone Gates
• Popular method used by government organizations uses a combination of weighted milestones and
percent complete
• "Subjective" percent-complete estimates are allowed
up to a specific ceiling associated with each
milestone
• Advancement past the milestone is not allowed until
tangible criteria have been met, hence the term
"gate"
INFO630 Week 3
56
www.ischool.drexel.edu
Equivalent Completed Units
• Useful for extended duration and repetitive tasks
• The overall project is divided into distinct units of
accomplishment
• Earned value is computed by summing
the “units” completed
INFO630 Week 3
57
www.ischool.drexel.edu
Earned Standards
• Useful for production-type work
• Perhaps the most sophisticated method to compute
earned value
• Standards of performance based on historical cost
data, time and motion studies, etc., are used to
compute the earned value on a given task...
INFO630 Week 3
58
www.ischool.drexel.edu
Earned Standards
• Often, several standards are used simultaneously,
and a management consensus determines which
standard is ultimately used
• These measures are typically used to track earned
value on either nonrecurring or recurring tasks
INFO630 Week 3
59
www.ischool.drexel.edu
Apportioned Relationship to
Discrete Work
• Useful for tasks in which their performance has a
direct relationship to other tasks
• The earned value for apportioned tasks is
a summary of the earned value measurements
made on the work to which it is related
• Schedule variances in the apportioned work are
usually identical to schedule variances in its related
work
INFO630 Week 3
60
www.ischool.drexel.edu
Apportioned Relationship to
Discrete Work
• However, cost variances in the apportioned work are
substantially different from the related work because
of the dynamics of actual costs
• This measurement can employ any of the previous
six methods
INFO630 Week 3
61
www.ischool.drexel.edu
Level of Effort
• Level of effort tasks are those earn value based
entirely on the actual amount of effort expended,
hence whatever is set as the planned value is
always the earned value, regardless of what work
was done
• This method is generally not recommended
to track earned value – no accountability!
• Used mainly for pure research or overhead tasks
(e.g. clerical support)
INFO630 Week 3
62
www.ischool.drexel.edu
References
• Earned Value Management
– Sean Alexander, “Earned Value Management
Systems Basic Concepts”
– Wayne Abba, “The New Management
Revolution: Integrating Contractor
Performance with Agency Budgets”
• See Balanced Scorecard for another
management measurement system
INFO630 Week 3
63
www.ischool.drexel.edu