Performance Metrics
Performance Metrics – What Are They And How Do I Write Good
Ones?
Chris Hamm
Operations Director
General Services Administration
GSA Training
Conference
and Expo 2010
1
Performance Metrics
This session is sponsored by the
Federal Acquisition Institute
The primary organization providing knowledge
and support to the federal civilian acquisition
workforce.
For more information about FAI, please visit
our website at www.fai.gov
2
Performance Metrics
Topics
•
•
•
•
Introduction to Performance Metrics
Step by Step Process for Creating Metrics
Post Award Management Process
Pitfalls
3
Performance Metrics
 “If you can’t measure it, you can’t manage it”

Peter Drucker
4
Performance Metrics
Stickk.com
&
The Biggest Loser
 Stickk
 Dean Karlan - Economics professor at Yale University
 Ian Ayres - a Law professor at Yale University
 Jordan Goldberg - a student from Yale School of Management

The mere act of writing down your goals increases your chances of success. Tying a
financial incentive (e.g., make a bet or Commitment Contracts) increases your odds
even more!
 Proof: Years of rigorous academic research using two of the hardest goals to achieve:
Losing weight and quitting smoking.
 The Biggest Loser
 Incredible results are possible with regular measurement, quality assurance
surveillance
5
Performance Metrics
What are Performance Metrics?
 They are the short hand for telling where we are.
• Number on your speedometer
• Grades your children bring home
• Performance results on your service contract
• Customer satisfaction level
• Uptime
• Usage
Also called
SLAs, KPIs
 A means of measuring that indicates how well our performance
objectives are being met
 An integral part of your requirements development and performance
management strategy
6
Performance Metrics
Why are we talking about Metrics?
FY08 – 50% of all eligible services acquisitions should be PBA
FAR Subpart 37.6
 (b) Performance-based contracts for services shall include—
 (1) A Performance Work Statement (PWS);
 (2) Measurable performance standards (i.e., in terms of
quality, timeliness, quantity, etc.) and the method of
assessing contractor performance against performance
standards; and
 (3) Performance incentives where appropriate. When used, the
performance incentives shall correspond to the performance
standards set forth in the contract.
7
Performance Metrics
Cost Reimbursable to Fixed Price Continuum
Research
Production / Sustainment
Development
CPFF
CPFF / CPAF
R&D
Development
CPIF/CPAF
Enhancements
Higher risk, less-defined
requirements
Government assumes
more cost risk
FPAF/FPIF/ FFP
/ Production
FPIF /FFP
Operations &
Maintenance
Lower Risk, well-defined requirements
Contractor assumes
more cost risk
As you move to the right, the more detailed and meaningful the
performance measures become. . . .
8
Performance Metrics
Performance Metrics are. . . .
•
A communications tool - The value of an agreement is not just in the
final product; the very process of establishing metrics helps to open up
communications
•
A conflict-prevention tool - An agreement helps to avoid or alleviate
disputes by providing a shared understanding of needs and priorities.
And if conflicts do occur, they tend to be resolved more readily and
with less gnashing of teeth
•
A living document - This is one of its most important benefits. The
agreement isn't a dead-end document consigned to the Forget
Forever file. On a predetermined frequency, the parties get to review
the metric to assess service adequacy and negotiate adjustments
•
An objective basis for gauging service effectiveness - A metric
ensures that all parties use the same criteria to evaluate service
quality
9
Performance Metrics
Seven Steps to PBSA
Performance-Based Acquisition
12
©Acquisition Solutions, Inc.
http://www.acquisition.gov/comp/seven_steps/index.html
10
Performance Metrics
Establish the Team
 Create an Integrated Project Team! The IPT should
have your technical expert, stakeholders, financial
person, and acquisition. Be sure to include
contracting so that you start off on the right strategy
and discuss the requirement.
 Invite the technical staff that currently have access to
any performance data. There is data somewhere.
Find it. Get copies of it. Ask the current contractor for
it.
11
Performance Metrics
Describe the Problem that Needs Solving

Interview all internal and external parties to determine the needs,
desired outcome, potential solutions, obstacles. Get BUY IN!
 Take notes during the meeting, send them out to everybody.
Documentation takes time, but eliminates complaints and issues
that may arise. You cannot over-communicate. . .
 Finalize definitions on the terms you will be using for measuring
performance (e.g., Service Level Agreement [SLA], Key
Performance Indicator [KPI])
12
Performance Metrics
Conduct Market Research
 Research the public sector to see what has worked for
your counterparts in other agencies
Locate sources for performance measures. Type what
you are seeking into any search engine and you can find
many examples. There are dozens of companies that
track performance and create standards
 Meet with vendors one-on-one to learn about your
industry. Ask them what metrics they use
13
Performance Metrics
Due Diligence / Industry Days / RFIs

What if you don’t know what you want to measure?
 Industry Day: This is our current environment and requirement,
we give a presentation to interested vendors.
 Due Diligence: The Government releases a draft description of
the requirement and opens up the doors to interested parties,
helping each side clarify the requirement.
 Request for Information: Can include draft requirements, but is
geared most to soliciting ideas and options from the vendors.
 Industry Day is good, Due Diligence and RFI’s are better, because
you can ask for specific information on industry standard measures,
metrics, and approaches. Both will increase the quality of your RFP
14
and the vendor proposals.
Performance Metrics
How to Include Metrics in the Solicitation
• Draft Incentive/ Award Fee Plan and SLA Format included in
Section J
• Option 1: Specify Metrics & have all vendors bid the same
• Option 2: Let vendors chose Metrics & targets
• Review requirements with the client before release and see if
tasks are sufficiently defined to create notional metrics to
include as recommendations
• Remember that the Government’s strongest negotiation position
is pre-award.
• Fee Plans are unilateral before the period begins, bi-lateral
during the period (if the CO allows any changes at all)
Option # 2 is harder to evaluate but better post award: Better measures, more leverage after award.
15
Performance Metrics
PBA Matrix Template
Outcomes
Required Performance Monitoring
Services Standards
Methods Incentives
16
Performance Metrics
Map the Matrix to the Solicitation
Contract Line
Item structure
for RFP
PBA Matrix
Objective
Standards AQL
Performance
Work Statement
or SOO
Inspection
Quality
Assurance
Surveillance
Plan
Incentive
Evaluation
Factor
weightings for
Section M of
RFP
17
Performance Metrics
Decide how to Measure & Manage
Performance
• People get really wrapped up in performance measures and
SLAs, but this can be made simple if you want it to be.
• Think about the outcomes rather than the processes or org
charts
• Use the systems that you have in place
• Only measure things that are meaningful, not things because
you can
Bad Example: Cost Savings over IGCE
18
Performance Metrics
Contract Types for Measuring
Good Contract Types:
• “Incentive” types (CPIF / FPIP)
• Firm Fixed Price (FFP) and Fixed Unit Price (FUP)
• Award Fee (CPAF / FPAP)
Less Good Contract Types:
• Time & Materials and Labor Hour can have metrics, but cannot have metrics with
financial incentives
Regardless of contract type, the Government must still perform quality assurance.
The level of effort on surveillance is higher on the good contract types
19
Performance Metrics
Source Selection Process
• Include the vendor’s proposed metrics in the technical evaluation
• Review the feasibility of the measure, whether it is appropriate,
meaningful, and end-to-end
• Is the measure entirely within the control of the vendor?
• Most vendor proposed metrics will require some changes after
award
20
Performance Metrics
Manage Performance (immediately after post award)
 Ensure Contract Award includes any contractor proposed metric
and/or changes from Discussions and Negotiations
 Set up separate meeting from Kickoff Meeting with key stakeholders
to review metrics
 - Ensure that all stakeholders agree to the metrics
 - Set the “weight” of each metric
 - Make sure the metric is entirely within the control of the
contractor (no mixed responsibility)
 - Add further definition of the metric and all exclusions
21
Performance Metrics
Manage Performance (immediately after post award)
 Develop the methodology of how to measure performance
 Periods (monthly, quarterly, bi-annually)
 Set Levels (Stretch, primary, secondary, tertiary)
 Set Measurement Scale with Incentive / Disincentive
– For Example:
 Stretch:
100% + rollover
 Primary:
100% of available fee
 Secondary:
90%
 Tertiary:
75%
 Failure:
0 % Fee
22
Not all levels are required. Some only have one target. It depends on the requirements.
Performance Metrics
Manage Performance - Measurement Scale
 Award Fee Plan Scale:





Outstanding (95-100)
Excellent
(90-95)
Good
(80-90)
Average
(70-80)
Poor
(0-70)
SLA Measurement Scale:





Outstanding
Excellent
Good
Average
Poor
99.99+
99.9 – 99.99
99 – 99.9
98 – 99
>98
23
Performance Metrics
Manage Performance - Examples of Measurements
 Percentages
 See Chart

Hours
 Indicate normal business
hours for response!

Days
 Always clarify business
vs. calendar days!

Yes / No
Measure
Downtime in one year
99%
3.7 days
99.90%
8.8 hours
99.99%
53 minutes
99.999%
5.3 minutes
Most projects do not need Five 9s
and could not afford it if they
wanted it. The cost is prohibitive!
24
NEW EXCEPTION - CLOUD availability
Performance Metrics
Manage Performance – a few months post award
• Hold a regular review of the metrics to ensure that the levels are set
correctly
• Set up a quality assurance function to check the contractor’s self
reporting metrics
• Insist on access to the raw data from the system tools used to
generate the metrics reports
• Review all trends and focus on failures
• Root cause analysis can identify process issues
25
Performance Metrics
Manage Performance – Build Dashboard
Summary Scorecard
CLIN 0001
Contractor #
2
Dinning
Facility
Supplier
A&AS
Supplier
Cost
Customer Satisfaction
Performance
Small Business Goals
26
Performance Metrics
Manage Performance – a few months post award
•Set a structure for gradual improvement of metrics
over time
•For particularly difficult metrics to negotiate, consider
benchmarking them for a period without making them
SLAs to obtain buy-in and set the appropriate target
•Gradually increase the overall weight of metrics in
the fee plan overtime, with the goal of eliminating
subjectivity as much as possible
27
Performance Metrics
Manage Performance - Exceptions
• Scheduled Downtime
• Government Intervention in Contractor Response
• Other Contractor Intervention
• Acts of God / Force Majure
Make sure someone from the Government is aware of and approves all
scheduled downtime or deviations from SLAs. No fox in the hen house!
Another benefit of outsourcing your infrastructure to the “cloud” . . . No more
scheduled downtime!
28
Performance Metrics
Manage Performance – Rollover
Good metrics are designed to challenge the vendor to
deliver good performance. When the vendor does not
achieve the performance goal, the Government has the
option of “Rollover” *, where the financial incentive is
not lost, but moved to the next period.
If mitigating circumstances exist (e.g., Government
redirected performance), Rollover to the next period
may be appropriate. If the vendor simply failed to meet
the goal, Rollover is not appropriate.
* Interim Rule Section 814 – Reporting on Rollover
29
Performance Metrics
Manage Performance – Pitfalls
 Metrics should align with the client and project objective AND
motivate good contractor behavior
 Do not design a metric that, when missed, motivates the
contractor to give up on that measure for the rest of the
period
– Example: Doing an “average” over a six month period. If
the contractor fails in the 1st month, there is no incentive
to maintain service levels or improve over time
 Just because something can be measured doesn’t mean it
should be a metric
 Example: It is easy to measure how fast someone picks up a
phone, but a better measure is how long it takes to actually
solve the problem
30
Performance Metrics
Manage Performance – Pitfalls
 Having too many measures
 Example: Too many metrics, some of which were redundant
and worth less than 1% of the pool
 Example: Aligning metrics to Government staff. At least one
metric (and sometimes more) for each of 25 tasks to
appease stakeholders
 Targets
 Just like performance reviews, if you set a target, the
contractor will plan and staff to meet that target. If the target
is an average, make sure that they plan to make the
average, not to always beat the target
31
Performance Metrics
Manage Performance – Pitfalls
 Funding
 If the metric has an incentive, that amount must be fully
funded to prevent any anti-deficiency situation
– This is what makes really good incentives difficult level of
oversight
 Oversight
 COR and Contractor may initially balk at the level of effort
required to stand up a metric management process
32
Performance Metrics
Manage Performance – Pitfalls
 Over reliance on Industry Standards
 Benchmarks are great, but that does not mean they are right
for your client’s infrastructure. Just because Gartner says
five 9s is normal for a call center doesn’t mean that your
requirement needs that
 Customer Satisfaction
 This one is difficult. It will take time to reach agreement on
what scale to use (1-5, Y/N, etc), what survey to use, how to
count the no-response pool.
 At the end of the day, customer satisfaction may be the most
important measure but the hardest to get right
33
Performance Metrics
Learn more about the Federal Acquisition Institute and the
resources it offers at www.fai.gov
34
Performance Metrics
Questions
35