eHealth Benefits Management Toolkit
Session 3:
Benefits Definition
and Measurement
v4.0
May 2009
1
Session 3 Outline
Benefits
measurem
ent
categorisat
ion
2
Benefits
profile
template
Benefits
Realisation
Plan
Transition
to
business
as usual
Benefits Management
Session 3 purpose
At the end of the session you will have an understanding of:
• what methods can be used to define quantifiable benefits and
estimate their value;
• how to finalise detailed benefit definitions using SMARTT and
ensure the benefits are measurable;
• how to pull all the outputs of the previous sessions together to create
the Benefits Realisation Plan; and
• what further activities need to take place at the end of the project.
3
Benefits Management
Benefits Measurement Categorisation
Benefits
measurem
ent
categorisat
ion
4
Benefits
profile
template
Benefits
Realisation
Plan
Transition
to
business
as usual
Benefits Management
Specific Definitions of Benefits
Benefits should be:
• clearly and succinctly defined
• aligned with the projects objectives and the overall NHS strategy
• focused on improving service and patient care, not focused on IT
Effective measurement and monitoring will:
• indicate the extent to which benefits are being realised;
• give early warning of potential problems;
• create the opportunity to adapt the benefits or changes to enable the
overall objectives; and
• ensure that achieved benefits are measured, reported and
communicated.
5
Benefits Management
Measurable Outcomes
The next step is to:
• devise measures that will show when the benefits have been delivered
and to what degree;
• determine which benefits can be quantified;
• determine the financial value of benefits where possible;
• investigate additional data capture activities required; and
• establish baseline, targets and benchmarking figures.
6
Benefits Management
Defining Measures
• Use existing measures where possible
• Agree all new measures with those concerned
• Ask yourself:
– Is it achievable?
– Is it an appropriate and valid indication that the benefit has been
delivered?
– Is it directly attributable to a particular change in the project?
• Consider the effort and cost of capturing and collating the data
• Decide who will be responsible for ensuring that the data is available
• Ensure the measures encourage the right type of behaviour
• Define when and how often the measurements should be taken
7
Benefits Management
Benefits Categorisation
(JW)
Degree of explicitness of measures
Financial
By applying a cost, price or other valid financial formula to
a quantifiable benefit, a financial value can be calculated.
Quantifiable
Sufficient evidence exists to forecast how much
improvement / benefit should result from the changes.
Measurable
This aspect of performance is currently being measured
or an appropriate measure could be implemented. But it
is not possible to estimate by how much performance will
improve.
By use of agreed criteria, specific individuals or groups
will decide to what extent the benefits has been realised,
based on their experience and judgement.
Observable
8
Benefits Management
Non Quantifiable Benefits
Observable benefits
• lowest category of benefits in term of robust measuring
• often the softer / people benefits
• can help get buy-in essential to the implementation of changes and the
realisation of their resulting benefits
• agree realisation criteria and who will say whether they are met
Measurable benefits
• performance is easily measured; but
• potential improvement won’t be known until after the event
• take baseline measurements before any changes are made for
comparison
9
Benefits Management
Quantifiable Benefits
Quantifiable benefits
• aspect of performance easily and consistently measured
• improvement can be predicted and targets defined
• usually applicable to embedded processes where performance is well
documented and predictable
Financial benefits
• highest category of benefits in term of robust measuring
• focus on benefits that can easily be translated in terms of financial gain
or reduced cost: if it is too complicated or not robust don’t assign a
monetary value to it
• don’t confuse value with savings
10
Benefits Management
Progressing Non Quantifiable Measures to Quantifiable
Measures
The following methods can be used to get from measurable to quantifiable
measures:
• Modelling or simulation: extrapolate expected benefits from existing
data. It may be necessary to establish trends/times series.
• Benchmarking: evaluate changes in relation to “best practices” in the
comparable organisations.
• References sites: can also be used to establish a comparative base but
assess relevance and feasibility in your own organisation.
• Pilots: test the new way of working and system on a small scale.
11
Benefits Management
Examples for Benefits Template
Category
Benefit Descriptions and Numbers
Measure
Financial
e.g. “increase utilisation of theatres resources by
allowing patient-cancelled appointments to be
filled at short notice from reserve list” 
expected increase in utilisation = 1 in 3
cancellations filled = 200 slots = £1000 per slot =
£200,000
cancellations £200,000
filled x
resource
value
(£1,000)
Quantifiable e.g. “elimination of referrals from SCI gateway
Value /
Improvem
ent Level
% of referrals 100%
processed
being printed onto paper for transfer to
secondary care system”  number of referrals, through new
and therefore those being printed, known before system
implementation; integration will eliminate these
number of x- N/A
Measurable e.g. “reduce unnecessary x-rays by avoiding
ray requests
repeat request”  not known how many x-ray
requests are repeats, but assumption is that
these will be eliminated, reducing overall number
of x-ray requests
% of satisfied N/A
Observable e.g. “reduce staff frustration due to need for
or very
double data entry”  staff satisfaction survey
satisfied
with regard to duplicate data entry
Benefits Management
12
Benefits Profile Template
Benefits
measurem
ent
categorisat
ion
13
Benefits
profile
template
Benefits
Realisation
Plan
Transition
to
business
as usual
Benefits Management
SMARTT Benefits
Detailed benefit descriptions contain the following categories:
Benefit
Characteristic
Specific
Measureable
Achieveable
Realistic
Time-bound
Targeted
14
Definition
Example
Description of the benefits
Number of re-xray forms completed.
Quality dimensions enabled
Safety, efficiency, timeliness
Variable to be evaluated
Number of forms filled in per month
How high a priority is the
benefit?
Scored on a scale of 1-5 (where 5 is the highest priority)
How likely is the benefit to
be realised?
Scored on a scale of 1-5 (where 5 is the highest
likelihood)
What is the benefit
dependent on?
Strategic direction, process, information management,
skills/ behaviours, organisation/ roles, infrastructure/
physical environment, technology integration
Realisation timeframe
E.G. Measure benefit from 3 months pre-go-live until 1
year post go-live
Measurement interval
recommendation
E.G. Per minute, per hour, per week, etc.
Frequency of
measurements
E.G. One week out of each month, one month out of
each year, etc.
Benchmark
E.G. 20 forms/month
Target
E.G. 20% reduction
Benefits Management
Benefit ID
Enter ID number from BDN
Benefit description
Enter benefit description
Benefit owner
Enter the name of the person or group who
will receive the benefit
Benefit sponsor
Enter name and role of the person who will
ensure that the benefit is achieved and
instigate remedial actions if required
Safe
Tick if relevant
Effective
Tick if relevant
Timely
Tick if relevant
Efficient
Tick if relevant
Patient-centred
Tick if relevant
Equitable
Tick if relevant
•Utilise measurements that are
already being used by the Health
Board where possible
•Take into account pre-existing
government targets of efficiency
estimates when setting
benchmarks and targets for
benefits
•Ensure that dependencies are
taken into account when setting
realisation timelines
Measure/Variable to be evaluated
Enter description of measure to be collected
Benefit explicitness
Enter category: financial (high), quantifiable,
measurable, observable (low)
Value/Level of improvement
Only applicable for quantifiable and financial
benefits
Impact
How important is the benefit? Scale of 1 (low)
to 5 (high)
Likelihood
How likely is the benefit to be realised? 1
(low) to 5 (high)
Strategic direction
Enter ID and description of objective(s) the
benefit contributes to
Changes required
Enter ID and owner of required changes (see
change profile for details)
IM&T required
Enter ID and description of required IM&T
capabilities
Benefit realisation date
Enter date the benefit will be realised (or
dates and degree of realisation for staged
roll-out)
Measurement timeframe
Enter start and end dates of monitoring and
reporting
Measure interval
Enter description of the period of time the
measurement covers
Baseline
Enter value, source and date if applicable
15
Benchmark or target
Filling in the Benefit Profiles
ACTION:
Expand on the benefits
information from the previous
workshops in the template for
each priority benefit identified:
Benefits
Management
Enter value,
source and
date if applicable
Preparing to Measure
• Are there pre-set requirements for benefits realisation timelines?
• Who is going to be collecting the measurements?
• How will the measurement be collected? (data sources)
• Who will oversee the benefits collection?
• Where will the results be reported to?
• Who will conduct benefits reviews?
• Who will be responsible for communicating about the benefits side of the
project?
16
Benefits Management
Preparing for Benefits Monitoring
In order that the benefits monitoring is as productive as possible:
• define a robust mechanism to monitor progress and ensure that reports
are distributed to the appropriate people;
• ensure that the Benefits Management templates are up to date;
• hold regular benefits reviews to track the progress of benefits realisation
against the project and programme objectives;
• use benefits reviews as an opportunity to highlight areas where benefits
are not being achieved, and identify the barriers to achieving them;
• for those benefits that are under-realised, refer to the dependencies
shown in the BDN for potential root causes;
• assign responsibilities for taking and coordinating corrective actions
where benefits are not being met; and
• identify and communicate successes too!
17
Benefits Management
Benefits Realisation Plan
Benefits
measurem
ent
categorisat
ion
18
Benefits
profile
template
Benefits
Realisation
Plan
Transition
to
business
as usual
Benefits Management
Formalising it
The Benefit Realisation Plan provides evidence to:
•
•
•
•
•
Justify investment decisions
Prioritise competing work in context of finite resources
Establish clear ownership / governance arrangements
Realise benefits
Demonstrate value and effective use of resources
Store the Benefit Realisation Plan with the other key project management
work.
19
Benefits Management
Benefits Realisation Plan (BRP)
When drafting the Benefit Realisation Plan remember to:
• Consider the dependencies identified between benefits, changes and
IS&T capabilities;
• Identify timeline and milestones that take those dependencies into
account;
• Work with stakeholders towards minimising the negative impact of
identified disbenefits; and
• Extend the plan beyond the system implementation date (often viewed as
the end of the project) to schedule regular reviews of benefits realised
and consider potential further benefits.
20
Benefits Management
BRP Content
The Benefit Realisation Plan should include:
• the completed Benefits Dependency Network (BDN)
• the completed benefit profiles, including dates for capture of baseline
figures at an early stage
• the completed change templates
• the completed stakeholders analysis
Prioritise the benefits so that the most important always has the most
focus. This ensures that the project makes the greatest impact.
Identify dates for expected delivery of the benefits.
21
Benefits Management
Benefits Strategy
A benefits strategy should also be prepared for the business case,
including:
•
•
•
•
•
22
the rationale linking the drivers, objectives and benefits
defined roles and responsibilities
a mechanism for regular measurements and reporting
a mechanism for regular reviews and corrective action
arrangements for the transfer of responsibilities at close of project
Benefits Management
Linking Business Case & Benefits Management
Providing all this, the business case will demonstrate that:
• benefits outweigh disbenefits;
• a shared vision is held which is strategically aligned;
• the BDN clearly shows how the shared vision will be achieved;
• the change required, when considered in relation to the overall schedule
of change planned, is not overwhelming for any stakeholders; and,
• the benefits realised will be worth the effort required to achieve them.
It will also show that:
• there is a clear process for identifying, monitoring and realising the
benefits; and
• the baseline benefits position has been recorded to enable comparison
with projected targets for monitoring the achievement of benefits.
23
Benefits Management
Transition to Business as Usual
Benefits
measurem
ent
categorisat
ion
24
Benefits
profile
template
Benefits
Realisation
Plan
Transition
to
business
as usual
Benefits Management
Close of project activities
Following full implementation of information systems, the achievement of
the business case should be reviewed.
The review will produce 3 reports:
• formal end-of-project review
• lessons learned
• potential further benefits
25
Benefits Management
End of Project Review
The end of project review will:
• assess to what extend the project was successful in delivering the
benefits;
• decide what further actions are required, by whom and when they should
be reviewed; and
• detail handover arrangements for any tasks outstanding (including
benefits monitoring and reporting).
The Benefits Plan should be updated as relevant to ensure a reference
document is available to handover to the “business as usual” team.
26
Benefits Management
Lessons Learned
The purpose of the “lessons learned report” is to:
• reflect on the lessons learned for future investments;
• disseminate useful lessons learned during the project for the benefit of
other projects, internal and external;
• provide a useful control as part of the functions of an independent quality
assurance group;
• consider how much effort was required to complete the benefits
management documentation and monitor the benefits; and
• make recommendations for future enhancement or modification of the
benefits management method.
27
Benefits Management
Further Benefits
The Review Team should also consider and communicate:
• what opportunities exist for the identification and realisation of additional
benefits; and
• who will take the business case for them forward.
Additional benefits should be defined, planned and implemented following
the same methodology as the initial project benefits.
28
Benefits Management
Transition to Business As Usual
• Leave no loose ends: each outstanding activity needs to be allocated to a
named individual
• Have end dates for decommissioning of legacy systems to ensure the
new IM&T are used – this will ensure new processes are used too and
related benefits are delivered
• Monitor the implementation of change over a period:
– people may revert to the “old ways” once the initial enthusiasm has
worn off
– people may find work-arounds where the technology or new
processes are not working instead of reporting the problems
– address teething problems
29
Benefits Management
Summary of Benefits Management Lifecycle
Project lifecycle
Benefits management lifecycle
Outputs
Identify and structure benefits
Project
initiation
Benefits Dependency
Network (BDN)
Identify change activities required to realize the
benefits
Prepare benefit plan and business case
Run first benefits analysis
Delivery
management
Make adjustment to system capabilities,
business changes or benefit definitions
as required
Business
as usual
30
Baseline data
Use benefit plan to inform project management,
system development and change management
activities
Regularly review project progress
against benefits analysis
Close
Benefit profiles
Change profiles
Stakeholder analysis
Run end of project benefits analysis (formal
review)
Updated figures
Amended
templates and BDN
Formal report to
Board
Identify need for further reviews and
potential for further benefits
Benefits Management
References and Further Reading
• The Clinical Indicators Team has produced a presentation on statistical
process control (SPC):
http://www.indicators.scot.nhs.uk/Workshops/SPC.html
• HM Treasury’s “Green Book” provides examples of methods used to
estimate the financial value of activities or resources:
http://www.hm-treasury.gov.uk/d/green_book_complete.pdf
• No Delays Scotland have devised a methodology for measuring benefits:
http://www.nodelaysscotland.scot.nhs.uk/ServiceImprovement/Tools/Pages
/IT211_Methodology_for_measuring_benefits.aspx
• Statistics for NHS England and Wales for benchmarking on:
http://www.ic.nhs.uk/statistics-and-data-collections
31
(cont’d)
Benefits Management
References and Further Reading
(cont’d)
• Benchmark statistics (Scotland) are available from ISD:
http://www.isdscotland.org/isd/782.html
• Quarterly and annual figures on patient safety are published by NPSA:
http://www.npsa.nhs.uk/nrls/patient-safety-incident-data/quarterly-datareports/
• The Department of Health publishes annual and quarterly healthcare
performance statistics:
http://www.dh.gov.uk/en/Publicationsandstatistics/Statistics/Performanceda
taandstatistics/index.htm
• Population numbers by NHS Boards – births, deaths, totals, by age, by
gender – are available from GRO:
http://www.gro-scotland.gov.uk/statistics/publications-and-data/populationestimates/index.html
32
Benefits Management
Congratulations on completing the benefits
management programme and on all your efforts
to ensure that maximum benefit is gained from
eHealth Systems.
33
Benefits Management