Preparing Metrics
that Matter
Enhancing Customization and Automation
www.regoconsulting.com
Phone: 1-888-813-0444
Agenda
●
●
●
●
●
Functional Best Practices
Case Study – Emerging Health
Case Study – HSBC
Open Discussion
Technical Best Practices
2
www.regoconsulting.com
Phone: 1-888-813-0444
Why do we utilize metrics?
● Align behaviors with organizational strategy
● What gets measured gets done
● Enables self-service performance management
● Benchmarking – Internal and External
● Manage based on data not gut feel
3
www.regoconsulting.com
Phone: 1-888-813-0444
What makes a good Metric?
• SMART
• Specific
• Meaningful
• Aligned
• Realistic goals
• Time based
4
www.regoconsulting.com
Phone: 1-888-813-0444
What makes a good Metric?
•
•
•
•
•
•
•
Balanced – Do not want to drive the wrong behavior
Drill Down – Need to be able to answer why
Automated – consistency, availability, accuracy
Incentives – rewards drive behavior
Positive Focus – Highlight good behavior not bad
Assign Ownership – Monitor, report improve
Leading vs Lagging – measure things that can be
changed
5
www.regoconsulting.com
Phone: 1-888-813-0444
How to Create KPIs?
1. Understand strategic goals of your organization
2. Determine the Key Success Factors in reaching the goals
3. Define Key Performance Indicators that measure the success
factors
4. Determine the Success Factors for these metrics
5. Define Performance Indicators that measure the success
factors
6. Make KPIs, PIs and data available
7. Communicate to all levels
6
www.regoconsulting.com
Phone: 1-888-813-0444
What Metrics Do You Use Today?
●
○
○
●
○
○
○
% of open tasks with finish date < one week ago (stale)
% projects on-time
% of tasks completed on time
% of tasks completed within baseline duration
% of tasks completed within baseline hours
% Milestones on time
Actual/Estimate - % to estimate
Average task duration or effort (good plans)
% tasks assigned 3 months out (how well advanced
planning)
# of baselines
Milestone Burndown
Cross Project Dependency Slack
–
–
–
–
–
–
–
–
•
○
–
–
–
% projects on-budget
% complete vs. % money expended
Actual / budgeted ROI
Total support costs for x months after solution is
completed
Amount of project manager time vs. overall effort hours
% change in NPV/IRR from gate to gate
Benefits realized vs. benefits planned
% projects with current financial forecasts
•
Subjective R/Y/G on projects by the PM (PMs thoughts)
Duration between steps, phases, stages, etc (diagnose
where issue. This will not solve issues)
# issues / project duration in days
How long issues remain open
% projects with current status reports
Number of defects discovered after initial acceptance /
development hours
Counts and amounts for projects in Red/Amber/Green
status
# issues by severity: opened, closed, on-hold
% projects failed gate review
% projects that delayed gate review
Time (Not Project Related)
–
–
–
–
–
Budget
○
○
○
○
Overall Project
–
–
# changes / project duration in days (bad req. or bad PM)
Customer satisfaction (survey 1-10)
Schedule
○
○
○
○
○
○
○
○
○
●
•
Scope
Time not submitted on time
Ave hours worked by staff
% internal vs. external resources
Time to assign WR (response time)
Time by % Admin, Projects, Maintenance (Doing the right
things)
Resource Management
–
–
–
–
–
% of resources with “appropriate” allocations 6-9 months
in the future
% of resources with “appropriate” assignments 0-3 months
in the future
Forecasted Utilization % by project manager for next 0-3
months
Actual Utilization % by project manager for last 3 months
Workloads by resources or manager - # projects they
support
7
www.regoconsulting.com
Phone: 1-888-813-0444
KPI Project Dashboard
8
www.regoconsulting.com
Phone: 1-888-813-0444
What will be Covered
• PM Dashboard portlet, Sub-Page & Drill Down
• Key Performance Indicators
• Governance
9
www.regoconsulting.com
Phone: 1-888-813-0444
Overview & Importance
• PM Dashboard
– Provides PMs and Management with important information
related to project staffing, schedule and level of effort.
– Information drives Executive decisions related to project
portfolios.
– Can be used by the PM as a project maintenance check list.
– Makes it easier for PMs to identify and correct project
schedule and staffing issues.
– Accessible to all users
10
www.regoconsulting.com
Phone: 1-888-813-0444
Accessing PM Dashboard sub-page
Accessing from within a project
Click the PM Dashboard sub-page link on the project properties page.
The sub-page will show all indicators related to the project.
11
www.regoconsulting.com
Phone: 1-888-813-0444
Project Dashboard Detail
Use: Provides detailed information related to Red and Yellow Indicators.
To launch the Detail information click on a Red or Yellow icon
The detail page for ETCs Past is the Project
Team portlet. All others indicators link to the
Project Dashboard Popup Page as shown to
the right
12
www.regoconsulting.com
Phone: 1-888-813-0444
Portlet Defaults
• The portlet returns all active projects
• Indicator Update Frequency
Hourly Updates
Daily Updates at 4:00 am
Zero ETC
ETCs Past
Milestone Schedule
Next Go Live Dt Past
Comment Dt Past
Resource Budget
Unfilled Roles within 30 days
Allocation Dt Past
13
www.regoconsulting.com
Phone: 1-888-813-0444
KPI Indicator Colors
A Gray icon means the indicator does not apply to the
project in its current state.
A Green icon means the indicator is within the acceptable
range and no action is required at this time.
A Yellow icon means there is a component of the
project that needs attention.
A Red icon means there is a component of the project
that needs immediate attention.
14
www.regoconsulting.com
Phone: 1-888-813-0444
Key Components of Governance
• Establish a governance structure with a Project Committee
including appropriate subgroups
• Define your processes – e.g. start identifying what projects align
with your strategic goals
• Ensure priorities are set by Senior Management and Executives
• Communicate priorities and progress on a regular basis
• Monitor Resource, Budget and Schedule regularly
• Implement effective Governance reporting based on the
processes
15
www.regoconsulting.com
Phone: 1-888-813-0444
HSBC
16
www.regoconsulting.com
Phone: 1-888-813-0444
}
HSBC TECHNOLOGY AND SERVICES
HSBC background
} Clarity/Niku customer since 2005
} Rego Consultant services provided since
2010
} One partition
– 3 Main Department Functions
– Global system
} 37,200 active resources
– 31,000+ OTE
– 27,500+ Billable
} 39,500 + active projects
– 27,000+ Billable
} Integrated with HR and Finance systems
} Yearly Financial planning process
– Project setup
– Billable Product tie back
} Actuals = financial transactions
(chargebacks to customers)
} 3 Main departments
INTERNAL
Restricted for company use only
HSBC TECHNOLOGY AND SERVICES
}
Overview Resource/Project Details
} Gaining value from Clarity depends on effective use of the tool and
increasing end-user adoption. Project and Resource management metrics
are required to baseline and track adoption as part of monitoring and
improving the value of the data on Clarity
} Clarity Maturity Model (CMM) was implemented in 2008 in response to an
audit requirement to track Clarity usage and data
} Clarity SWAT are looking to materially improved the existing CMM model to
eliminate “box ticking” and implement metrics that will drive sound resource
and project management behavior
18
INTERNAL
Restricted for company use only
HSBC TECHNOLOGY AND SERVICES
}
Key Roles
} LQSOs – Local Quality Service Organization; audit
regional/departmental project and resource data
} Clarity PPMT (Project and Portfolio Management
Tooling) – Enhancement, defect and maintenance.
Make CMM available and enhance/create new
Metrics/KPIs
} PMOs, Deployment Leads, Middle and upper
Management; Direction and Take action on
deployments
19
INTERNAL
Restricted for company use only
CMM – Clarity Maturity model
Clarity/Niku customer since 2005
Rego Consultant services provided since 2010
37,200 active resources
– 31,000+ OTE
– 27,500+ Billable
39,500 + active projects
– CTB 28,000+ RTB 11,500+
– 27,000+ Billable
Integrated with HR and Finance systems
Yearly Financial planning process
– Project setup
– Billable Product tie back
Actuals = financial transactions (chargebacks to customers)
INTERNAL
20
CMM – Clarity Maturity model
Clarity/Niku customer since 2005
Rego Consultant services provided since 2010
37,200 active resources
– 31,000+ OTE
– 27,500+ Billable
39,500 + active projects
– CTB 28,000+ RTB 11,500+
– 27,000+ Billable
Integrated with HR and Finance systems
Yearly Financial planning process
– Project setup
– Billable Product tie back
Actuals = financial transactions (chargebacks to customers)
INTERNAL
21
22
INTERNAL
Discussion Questions
For your organization:
● What are your process pain points?
● What behaviors are you trying to drive?
● What are your reporting challenges?
23
www.regoconsulting.com
Phone: 1-888-813-0444
Metrics Development Cycle
5. Analyze the
Trends
1. Process
improvement
Identified
OR
Other Impacts
Strategic Goals
2. Requirements
for a Metric
4. Reporting
Happy Customer
3. Technical
Development
And
Implementation
24
www.regoconsulting.com
Phone: 1-888-813-0444
Technical Best Practices
Check what is already available
● OOB Metrics
○ Earned Value (CPI, SPI)
● RegoXchange
○ Search for keyword “Dashboard” on the RegoXchange website
○ Rego Adoption Metrics (coming soon!)
25
www.regoconsulting.com
Phone: 1-888-813-0444
RegoXchange Content
Portlet Name
Description
Dashboard: Baseline Comparison
Compares multiple baselines on a project
Dashboard: Change Requests
Provides an overview of all change requests on the project
Dashboard: Financial Summary by Charge Code
Displays summary financial information by charge code on the
project dashboard
Dashboard: Open Issues
Provides an overview of all issues on the project.
Dashboard: Open Milestones
Provides an overview of all milestones on the project.
Dashboard: Open Risks
Provides an overview of all risks on the project.
Dashboard: Phase Gantt
Provides an overview of highest-level tasks on the project.
Dashboard: Programs Portfolios
This portlet displays all programs and portfolios a project is a
subproject of
Dashboard: Project Information
Provides an overview of the project.
Dashboard: Project Status Indicators Multi-line
Dashboard: Project Status Indicators Single-line
Provides an overview of the project status on multiple lines.
Provides an overview of the project status on a single line.
Executive Dashboard
A one-stop place for the PMO or management to view all critical
information about projects.
Incident Dashboard
Displays an overview of incident and related task information
Program Executive Dashboard
Displays project information for projects within a program
Project Manager Dashboard
Provides an overview of all projects.
www.regoconsulting.com
Phone: 1-888-813-0444
26
Rego Adoption Metrics
• We will have a set of metrics defined for projects, and a set of
defined for resources.
• Each metric will have a set criteria for levels 0-5 maturity, so that
for each metric that item (i.e. project) will be given a maturity
level from 0-5.
• Project Metrics will apply only to active projects
• Resource Metrics will only apply only to active users (exclude
locked resources)
27
www.regoconsulting.com
Phone: 1-888-813-0444
Rego Adoption Metrics
Metric Focus
Project
Metric
Description
Project Allocation Adoption This adoption metric indicates if the project manager is keeping the project staff allocations in line with the
Metric
effort that is being booked to the project through time entry. This metric looks at the prior three months and
determines how close the allocations match the actuals. The closer the match, the higher the score.
Resource
Resource Allocation
Adoption Metric
This adoption metric indicates if the Resource Manager is keeping their staff’s allocations current. The metric
scores increase as the allocations of the full resource capacity are within a reasonable range for longer future
periods of time. Highest scoring goes to resources that have allocations close to 100% for the next 120 days.
Resource
Resource Data Quality
Metric
This adoption metric indicates if the Resource Manager is keeping specific fields in their resource’s data
populated and demonstrates active management of the resource attributes by the Resource Manager role
Resource
Resource Timesheet Entry
Metric
This adoption metric indicates if the Time-entry Resource is entering their time on a timely basis. The metric
looks back 4 weeks and determines how many timesheets were entered and how many are missing. The
Highest scoring goes to timesheets submitted on time and with total hours greater than 10.
Resource
Resource Commitment
Metric
This adoption metric compares the resource Soft Booked and Hard Booked allocations within a 4 week view.
Hard Booked allocation hours receive a higher range score while Soft Booked allocations are a lower score.
Project
Project Baseline Adoption
Metric
This adoption metric indicates the level of adoption in using baselines. The metric indicates if any baseline
exists against the project. Scoring the metric on a project is either a zero (no baseline) or a 5 (has a baseline).
Project
Project Status Reporting
Adoption Metric
Project
Project Risk Management
Adoption Metric
This adoption metric indicates if the project manager is creating and publishing a project status on a biweekly basis. The metric looks back over a 6 week period and increases the score based on increasing
numbers of published status reports.
This adoption metric indicates if the project manager is managing risks against the project within CA Clarity.
The score indicates whether the risks are being used at all and increases the score value based on the
completeness of the Risk data and for the higher scores the active management of risk resolutions.
Project
Project Scheduling
Adoption Metric
This adoption metric indicates if the project manager is managing more than summary time entry tasks
within CA Clarity. Scores increased based on having 3 tasks or more and having the tasks closed when they
were scheduled to be complete. Highest scores on the metric require active management of 10 or more tasks.
28
www.regoconsulting.com
Phone: 1-888-813-0444
Rego Adoption Metrics
29
www.regoconsulting.com
Phone: 1-888-813-0444
Rego Adoption Metrics
30
www.regoconsulting.com
Phone: 1-888-813-0444
Technical Best Practices
● If the metrics you are looking for do not exist –
○ Make sure the requirements for the Metric are well defined
○ Architect the Metrics with keeping potential
OBS/Department/Resources changes in perspective (e.g. Re-Org)
○ Capture positive metrics
● Development approach
○
○
○
○
Automate data collection overnight. Real time metrics may be overkill.
Collect Data within Clarity
Capture Actuals as well as Baseline (or Budget) value for comparison
Capture the values at different levels of organization
(OBS/Departments/Roles)
31
www.regoconsulting.com
Phone: 1-888-813-0444
Technical Best Practices
● Reporting
○ Should be in visual format (Clarity Dashboards /Webi/Crystal or
Xcelsius Dashboards, not data extracts).
○ Dashboards should tell a story and highlight issues. Use 5 minute rule.
○ Show Trends
○ Drill down capability to show the details
32
www.regoconsulting.com
Phone: 1-888-813-0444
CIO / Senior Executive View
33
www.regoconsulting.com
Phone: 1-888-813-0444
Metrics Malfunctions
You know you have a metric malfunction if you hear:
●
●
●
●
●
I didn’t know this was being measured.
We measure this because the boss says so.
I have no idea how I can affect this measure.
I have no idea what this number means.
I do not worry about that metric I measure my own.
34
www.regoconsulting.com
Phone: 1-888-813-0444
Summary
●
●
●
●
People-Process-Technology
Alignment Goals-KPIs-Pis-Data
Automate metric collection
Crawl-Walk-Run
35
www.regoconsulting.com
Phone: 1-888-813-0444
Metric Examples
Metric Focus
Project
Metric
Project Allocation Adoption
Metric
Resource
Resource Allocation Adoption
Metric
Resource
Resource Data Quality Metric
Resource
Resource Timesheet Entry
Metric
Resource
Resource Commitment Metric
Project
Project Baseline Adoption
Metric
Project
Project Status Reporting
Adoption Metric
Project
Project Risk Management
Adoption Metric
Project
Project Scheduling Adoption
Metric
Description
This adoption metric indicates if the project manager is keeping the project staff allocations in line
with the effort that is being booked to the project through time entry. This metric looks at the
prior three months and determines how close the allocations match the actuals. The closer the
match, the higher the score.
This adoption metric indicates if the Resource Manager is keeping their staff’s allocations current.
The metric scores increase as the allocations of the full resource capacity are within a reasonable
range for longer future periods of time. Highest scoring goes to resources that have allocations
close to 100% for the next 120 days.
This adoption metric indicates if the Resource Manager is keeping specific fields in their resource’s
data populated and demonstrates active management of the resource attributes by the Resource
Manager role
This adoption metric indicates if the Time-entry Resource is entering their time on a timely basis.
The metric looks back 4 weeks and determines how many timesheets were entered and how many
are missing. The Highest scoring goes to timesheets submitted on time and with total hours
greater than 10.
This adoption metric compares the resource Soft Booked and Hard Booked allocations within a 4
week view. Hard Booked allocation hours receive a higher range score while Soft Booked
allocations are a lower score.
This adoption metric indicates the level of adoption in using baselines. The metric indicates if any
baseline exists against the project. Scoring the metric on a project is either a zero (no baseline) or
a 5 (has a baseline).
This adoption metric indicates if the project manager is creating and publishing a project status on
a bi-weekly basis. The metric looks back over a 6 week period and increases the score based on
increasing numbers of published status reports.
This adoption metric indicates if the project manager is managing risks against the project within
CA Clarity. The score indicates whether the risks are being used at all and increases the score value
based on the completeness of the Risk data and for the higher scores the active management of
risk resolutions.
This adoption metric indicates if the project manager is managing more than summary time entry
tasks within CA Clarity. Scores increased based on having 3 tasks or more and having the tasks
closed when they were scheduled to be complete. Highest scores on the metric require active
management of 10 or more tasks.
36
www.regoconsulting.com
Phone: 1-888-813-0444
Questions
Contact US
888.813.0444
"What gets measured gets done,
what gets measured and fed back
gets done well, what gets
rewarded gets repeated.“
John E. Jones
Email Contact
info@regoconsulting.com
atul.kunkulol@regoconsulting.com
clayton.Reynolds@regoconsulting.com
don.dickson@regoconsulting.com
Thank you.
Web Site
www.regoconsulting.com
37
www.regoconsulting.com
Phone: 1-888-813-0444