May 27, 2010
Defining and Assessing Project Metrics for
UC Berkeley Service Projects
Introduction
This document describes how project managers and other key project stakeholders can
define and assess project metrics to help make their campus online service projects be
more successful.
In order to ensure that a service project achieves its goals, the project must have clearly
defined measures of success that will allow all key stakeholders, including the sponsor,
functional owner, and the project team members, to assess whether or not the project is
progressing successfully and whether or not the service created by the project meets the
goals it was designed to achieve. A project should have sufficient metrics to answer five
questions:
1. Is the project being developed on time to meet any time constraints required for
the launch of the service?
2. Is the development of the project progressing on budget?
3. Is the service created by the project being used by the intended categories and
number of campus users?
4. If the system is not achieving the intended user adoption goals, what changes
should be made to the service?
5. Is the service achieving the business goals (e.g., time or financial savings) for
which the campus invested in the project?
In order to answer these questions, each project should have at least three categories of
metrics that serve as key performance indicators to track the progress of the project and
the success of the service the project creates:
1. Project Metrics
2. User Adoption and Satisfaction Metrics
3. Business Metrics
Defining and Measuring Project Metrics
Project metrics are designed to determine if the project is being developed on time and on
budget by tracking the progress of project milestones. A project milestone is a scheduled
event signifying the completion of a major deliverable or a set of related deliverables for
a project. Typically, a project will be divided into five to ten major milestones. Each
milestone has a deliverable, a scheduled completion date, and a cost estimate for its
development. Project milestone dates are defined in the project charter and refined in the
project schedule. If your project does not already have a charter, the Technology Program
Office has information about how to create a project charter, schedule, and budget on its
website at http://cio.berkeley.edu/cio/tpo/project/pmresources/tools/. To calculate the cost
of completing each milestone, the project manager will need to analyze the project budget
and vendor contracts and include the total cost of labor, equipment, software, and
supplies.
After defining the project milestones and each of their completion dates and costs, the
project manager can track the project metrics by using the Project Metrics Tracking
Form that is available on the Technology Program Office website at
http://cio.berkeley.edu/cio/tpo/project/pmresources/tools/.
As the actual milestone completion dates and actual costs are tracked, the project
manager may need to take corrective action if the milestones are delivered late or exceed
the cost estimate. If corrective action is needed, the project manager needs to revise the
project plan in consultation with the project team and sponsor. The plan should be revised
based upon the most important of the three constraints of the project: time, cost, and
scope. If the most important goal is to deliver the project by a specific date, then the
project manager should revise the plan to reduce the scope of work for the project and/or
increase the budget so more resources including staff can be applied to the project. When
possible, reducing the project scope is often the most effective strategy because adding
new staff to the project team requires additional time for the new project team members
to get up to speed. Also, project communication among the team members will become
more complex and require more work as the size of the project team grows.
Defining and Measuring User Adoption and Service Satisfaction
Metrics
In order for the project to be successful, the service created by the project needs to be
adopted by the number and categories of campus users described in the business case. A
business case is a written document that describes the reasons for initiating a project. The
business case also describes the resources such as money, staff time, and equipment that
are required to achieve the specific business goals. If user adoption goals are not
described in the business case, the project manager should ask the sponsor and functional
owner to identify the number and types of users that will be using the service by a
specified deadline after its launch. Examples of these goals include:
 The number of users who are utilizing the service by a specified date after
it is introduced.
 The maximum number of concurrent users of the service.
 The campus departments and units using the service.
 The total number of transactions processed by the system hosting the
service per unit of time.
 The maximum number of transactions the system should be able to
process per unit of time.
To determine if user goals have been met, the information technology system that
delivers the service needs to include functionality to measure each of these user adoption
metrics. This way, the sponsor, functional owner, and any other key stakeholder can
determine if the goals have been achieved and identify the causes if they have not. The
metrics should also provide the functional owner with data about ways the service
process can be improved.
2
For example, if the campus is implementing a new travel reimbursement system, the
project team needs to identify the user adoption goals. The project team could obtain this
information by interviewing the sponsor and the manager of the Travel and Entertainment
Services Office (the functional owner of the new system), and reviewing the existing
campus travel reimbursement data. The project might have the following requirements
for the service:
 Can be used by up to 7,000 campus employees.
 Can process up to 600 travel reimbursements per day.
 Can support up to 200 users simultaneously logged into the system.
 Takes an average of no more than 10 minutes for a traveler seeking a
reimbursement to enter data into the system.
 Takes an average of no more than 4 minutes for a campus staff person
who approves the reimbursements to perform a review.
 Processes the travel reimbursements in no more than an average of 3
business days after the request is initially entered into the system.
The travel reimbursement systems must be developed so the functional owner, sponsor,
and any other identified key stakeholder can determine if the online service is meeting the
user adoption goals. The system should generate reports that are easily available to
authorized staff assessing the metrics. When designing this functionality and reports, the
project team should consult with the staff that will be using the reports to ensure that the
information presented is complete and easy to understand.
After identifying the numbers and types of users that will use the service, the project team
needs to identify the most important factors that lead to user satisfaction with the service.
If the business case does not describe the user satisfaction goals and the sponsor and
functional owner cannot provide this information, the project team should conduct
surveys and/or focus groups with the intended users to identify the factors and system
functionality that will promote user satisfaction.
Surveys are useful for identifying basic information from a large number of potential
users, and the Web offers a variety of tools that make it easy and inexpensive to conduct
online surveys. SurveyMonkey and Zoomerang are two of the most commonly used webbased survey tools. For information about designing and conducting online surveys,
please consult the following books:
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
by Don A. Dilman et al.
Designing and Conducting Survey Research: A Comprehensive Guide (Jossey Bass
Public Administration Series) by Louis M. Rea et al.
To obtain a deeper understanding of the users’ needs, the project team should conduct
focus groups. A focus group is a form of qualitative research in which a group of people
are asked about the business issues and functionality that the service should address.
3
Questions are asked in an interactive group setting in which participants are free to talk
with other group members. To conduct a focus group, the project manager or another
member of the project team should invite members of the campus community that
represent a cross-section of people who will be using the online system. Prior to
conducting the focus group session, the project team should identify five to ten questions
to ask the focus group participants. When conducting the focus group, the project team
needs to have one member of the team facilitating the session and another team member
taking notes during the meeting. For more information about conducting focus groups,
please visit the following websites:
Focus Groups
http://www.usability.gov/methods/analyze_current/learn/focus.html
Conducting A Focus Group
http://www.cse.lehigh.edu/~glennb/mm/FocusGroups.htm
Basics of Conducting Focus Groups
http://managementhelp.org/evaluatn/focusgrp.htm
The information gathered from the surveys and focus groups should be used to design the
service’s functionality.
After the service has been operating for a specified duration, the functional owners of the
system should conduct a survey of the users to determine if they are satisfied with the
quality of the service. The survey should include questions about issues that were
identified during the previously conducted surveys and focus groups. The information
gleaned from the new survey should be used to address any important inadequacies with
the service and to make refinements that will improve user adoption and satisfaction.
When creating the budget, the project team should set aside five to ten percent of the
project budget for assessing and improving the service after launch in case it does not
meet the specified goals.
Defining and Measuring Business Metrics (Long-term success metrics)
Finally, for the service to be a success, it needs to achieve specific business goals
described in the business case for the project. Typical goals described in a business case
include:
 Reduces the amount of time to process a business transaction.
 Reduces the cost of performing a business transaction.
 Ensures that sensitive data is managed in a way that complies with state
and federal laws.
 Creates new business services that meet the specific needs of the members
of the campus community.
 Improves student services.
 Improves teaching and learning.
4
In order to determine if the service created or improved by the project has achieved the
business goals that were specified in the project’s business case, the service must have
defined business metrics that will allow the sponsor, functional owner, project manager,
and any other key stakeholder to assess the performance of the service. When defining
the business metrics, the functional owner and project manager need to identify all
aspects of the service that will either help or prevent the service from achieving the
specified business goals. Next, the project team should prioritize these factors from most
to least important and determine which of these factors should be addressed in the design
of the new service. Once the key factors are identified, the functional owner and project
manager must define the metrics that will be used to asses how effectively the new
service is achieving the business goals and then, in cooperation with the rest of the
project team, design and implement functionality into the system that will capture the
relevant data and make it available to the functional owner and other designated staff in
easily accessible and readable reports. As with the user adoption and satisfaction reports,
the project team should consult with the staff that will be using the reports when
designing them to ensure that the information presented is complete and easy to
understand.
To create services that meet the project’s business goals, the project team usually must
conduct a business process management analysis that identifies both the existing “as-is”
business process and the new business process that will be implemented in the revised
service. To learn more about business process analysis and design, campus staff can
register for courses on business process management by visiting the UC Berkeley
Learning Center website at http://hrweb.berkeley.edu/ice/home/. The book, Process
Mapping, Process Improvement and Process Management, by Dan Madison is also a
good reference for learning how to document existing business processes and improve
them.
Here is an example of defining business process metrics for a revised travel
reimbursement service that will reduce the average time required to process a
reimbursement from eight business days to four. The project team needs to identify the
business process steps and their sequence in the existing reimbursement process and
identify which steps cause the greatest delays in processing. In this example, when
reviewing the existing as-is business process, the project team discovers that campus
travelers seeking reimbursement frequently fail to provide all the necessary information
when entering their data into the reimbursement system. These omissions require the
Travel and Entertainment Services Office staff to contact the traveler to obtain the
additional information, which causes significant delays in the reimbursement process.
The project team needs to identify the information most frequently omitted by travelers
and design the new system to reduce these errors. The project team will also need to
identify the rate at which these errors occur in the as-is process. This error rate will
establish a baseline measurement to which the functional owner and other key
stakeholders can compare the error rate in the new travel reimbursement system to
determine if the new service delivers the efficiencies sought.
5
The project team will also want to identify any steps in the as-is business process that
could be eliminated in the new business process. For example, the Travel and
Entertainment Services Office staff may currently be manually reviewing every travel
reimbursement submission. The Travel and Entertainment Services Office staff may
replace this process with a statistical approach in which only a percentage, say ten
percent, of the requests are reviewed. If a traveler makes a mistake in the reimbursement
submission that exceeds two percent of the requested reimbursement, all of this traveler’s
reimbursement submissions will be flagged for review during the next year. This new
process will significantly reduce the amount of staff resources required to review travel
reimbursement requests and speed up their completion.
Finally, because the new travel reimbursement system is designed to reduce the time to
process a reimbursement, the system will also need to record how much time each stage
of the new business process takes to complete, the average number of business days it
takes to complete a reimbursement, and generate reports showing this information. The
functional owner can use the reports to identify causes for the delays in the process and
implement future revisions to the system that will further improve service performance.
6
Summary of Metrics Resources
Here is a summary of the resources referenced in this document.
• Project Management Metrics
Project Metrics Tracking Form
http://cio.berkeley.edu/cio/tpo/project/pmresources/tools/
• User Adoption and Service Satisfaction Metrics
Online Survey Services
SurveyMonkey
Zoomerang
Books about Creating Online Surveys
Madison, Dan, Process Mapping, Process Improvement and Process Management:
Chico, CA, Paton Press 2005
Dillman, Don, Internet, Mail, and Mixed-Mode Surveys: The Tailored Design
Method (3rd edition): Hoboken, NJ, John Wiley & Sons, 2008
Rea, Louis et al, Designing and Conducting Survey Research: A Comprehensive
Guide (A Comprehensive Text for Nonprofit and Public Leadership and
Management) 3rd edition: San Francisco, CA, Jossey Bass Public,
Information about Conducting Focus Groups
Usability.gov
http://www.usability.gov/methods/analyze_current/learn/focus.html
Lehigh University Department of Computer Science and Engineering
http://www.cse.lehigh.edu/~glennb/mm/FocusGroups.htm
Free Management Library
http://managementhelp.org/evaluatn/focusgrp.htm
• Business Metrics
Madison, Dan, Process Mapping, Process Improvement and Process Management:
Chico, CA, Paton Press 2005
7