Managing for Results
Programme Cycle Management
Trine Eriksen and Inger K. Stoll
Financial Mechanism Office
November 2012
1
RESULTS BASED MANAGEMENT
What is Results Based
Management (RBM)
RBM is a management strategy
by which all actors,
contributing directly or
indirectly to achieving a set of
results, ensure that their
processes, products and
services contribute to the
achievement of desired results
(outputs, outcomes and
impact)
What is a result and a
risk?
In the context of the Grants:
A result is the output, outcome
or impact of a development
intervention
A risk is an event that may
occur and impede the
objective
2
KEY CONCEPTS
Results achievement
Beneficiary oriented
Managing for results and reduced risks
3
RESULTS BASED MANAGEMENT
IN PRACTICE
1. Clearly identifying program beneficiaries and designing
programs to meet their needs.
2. Defining realistic expected results, based on appropriate
analysis.
3. Identifying and managing risks.
4. Monitoring progress towards results with the use of
appropriate indicators.
5. Increasing knowledge by learning lessons and integrating
them into decisions, and
6. Reporting on results achieved and the resources involved.
4
Part I
Quality at entry Basis for good management
5
PROGRAMMING CYCLE
The Programming Cycle and Continuous Feedback
Planning
Reporting
Programming
Budgeting
Evaluation
Monitoring
Implementation
6
The Programme and the Context
National/
sector area
Impact
--------------------------------
National/
sector area
Outcome
Programme
area
-------------------------------Inputs
Outputs
Programme
area
Activities
Present situation
Baseline
Future situation
Targets
RESULTS CHAIN
Impact
Widespread improvement in society/sector Long term
Outcomes Intermediate effects on target groups and systems Medium term
Outputs
Products and services delivered
Short term
Activities
Tasks transforming inputs to outputs
Ongoing
Inputs
Financial, human & material resources
Ongoing
8
Identify main programme
elements of EEA/N Grants
Objective (impact)
32 predefined objectives
Widespread improvements
in societ or a sector
Outcome(s)
Intermediate effects on
target groups and systems
160 predefined outcomes
In DoRIS
data base
Outputs
Products and services
delivered
Self-defined (at least two
outputs per outcome)
In DoRIS
data base
Activities
Tasks transforming inputs
to outputs
Self-defined
Not in DoRIS
Inputs
Financial, human and
material resources
Budget +
Budget in DoRIS
9
WHAT IS AN OUTCOME?
• Outcomes can occur on multiple levels:
•
•
•
•
•
•
Individual level;
Family or household;
Community or population group;
Organization;
System (involving multiple institutions or agencies); or
State.
10
DIFFERENT TYPES
OF OUTCOMES
• The nature of the changes can vary as well. Outcomes can involve
different types of changes:
•
•
•
•
•
•
Changes in perceptions, attitudes, feelings, beliefs, or values;
Knowledge;
Skills or level of functioning;
Behaviours or practices;
Conditions;
Status (such as socio-economic status, and status can reflect formal or
informal status); or
• Policies.
11
Defining outcomes and
outputs
• Express an outcome and an output
as a positive condition.
• A measurable, desired result: Wait list for a
cancer operation has been reduced from 16
moths to 10 months at hospital x in y province
• Avoid «double»objectives as these blur what
will be measured.
in order to
through
12
Cross-Cutting issues as
outputs and targets
• Good governance
• Gender equality
• Sustainable development: Environmental, Economic
and Social
Cross-cutting issues are either reflected in the outputs
or the indicators (targets), or assumptions.
If the programme substantially addresses cross-cutting
issues, it is advisable to define outputs and targets that
reflect the cross-cutting issues. This strengthens followup.
13
Programme logic
Contribute to
Contribute to
Likely to
happen
Should be
able to
guarantee
that this
will happen
Overall
objectives
Objective
Expected
outcomes
Outputs
which will contribute to the two
overall objectives
which will contribute to the fulfilment of the
objective
if the outputs are produced, then the
expected outcome is likely to be
achieved
if the activities take place, then
the outputs will be produced
Activities
Inputs
if the inputs are available, then
the activities will take place
14
ASSUMPTIONS
• The intervention logic is an aim-means chain and a
cause-effect chain.
• It is assumed that “certain events” need to occur at
one level in order to achieve the desired effect at the
next level.
• It is not certain that these events happen; they
represent uncertainties.
15
Impact/Objective
Outcome
There are
Assumptions
between each
level:
Outputs (deliverables)
Activities
Inputs
Identify
Assess
Monitor
Manage and
Mitigate
16
Identify important
assumptions
• Assumptions are worded as positive conditions (see objectives)
• Assumptions are linked to the different levels of the
Programme Matrix. Each level must contain the necessary and
sufficient conditions for the next level above.
• Start from bottom of matrix and work upwards
• Are weighted according to importance and probability
Example: Scholarship recipients return to assigned jobs
17
Check the significance
of the assumptions
1. Eliminate the obvious factors which are:
•
•
Not important for the outcome
Very likely to occur
2. Assess the probability of occurrence
for the remaining factors:
a) Quite likely but
not certain:
Include the factor
into the PM and
make sure to:
• Monitor it
• Report
changes
• If possible
influence it
b) Not likely to
occur (killing factor!)
Redesign the
programme
c) If this is not
possible:
Reject the
programme
proposal
RISKS
• If important assumptions do no occur, they might
represent a risk.
• A risk is any event that could occur and adversely
impact the achievement of results at all levels
• Risks are an expression of uncertainty
• Assumptions are expressed as a positive condition (a
necessary event), whilst risks are expressed as
negative conditions (un-wanted events)
19
Identify main programme
elements
Objective (impact)
Widespread improvements
in societ or a sector
Assumptions and risks
Outcome(s)
Intermediate effects on
target groups and systems
Assumptions and risks
Outputs
Products and services
Delivered
Assumptions and risks
Activities
Tasks transforming inputs
to outputs
Assumptions and risks
Inputs
Financial, human and
material resources
Assumptions and risks
20
Cross-Cutting issues as
assumptions
• Good governance
• Gender equality
• Sustainable development: Environmental, Economic
and Social
Cross-cutting issues are either reflected in the outputs
or the indicators (targets), or assumptions. If not taken
care of they can represent a risk to the sustainability of
the programme results. Therefore cross-cutting issues
should be monitored in the same way as assumptions
and risks.
21
The basic elements of the
Programme Matrix
Impact (Objective)
Outcome
Outputs (deliverables)
Activities
Assumptions describe the
necessary conditions that
must exist for the cause and
effect relationships between
the levels to occur.
If an assumption does not
occur, it might represent a
risk
Inputs
22
The Matrix is the point of reference
throughout the life of the programme
National/
sector area
National/
sector area
Programme implementation
Programme
area
Programme
area
Programme
Present situation
Baseline
Future situation
PART II
Risk Management
24
PART III
Programmes and
projects
25
Programmes and
projects
Pre-defined projects
Strategic
approach
Programme
proposal
Projects
Pre-defined projects
26
Projects
Programme
Programme design
OBJECTIVE
(INDICATORS)
(ASSUMPTION
S)
OUTCOME
INDICATORS
ASSUMPTION
S
OUTPUTS
INDICATORS
ASSUMPTION
S
PROGRAMME OUTCOME
PROGRAMME OUTPUTS
OBJECTIVE
(INDICATOR
S)
(ASSUMPTIO
NS)
OBJECTIVE
(INDICATOR
S)
(ASSUMPTI
ONS)
OBJECTIVE
(INDICATOR
S)
(ASSUMPTI
ONS)
OUTCOME
INDICATORS
ASSUMPTION
S
OUTCOME
INDICATORS
ASSUMPTIO
NS
OUTCOME
INDICATORS
ASSUMPTIO
NS
OUTPUTS
INDICATORS
ASSUMPTION
S
OUTPUTS
INDICATORS
ASSUMPTIO
NS
OUTPUTS
INDICATORS
ASSUMPTIO
NS
ACTIVITIES
INPUTS
ASSUMPTION
S
ACTIVITIES
INPUTS
ASSUMPTIO
NS
ACTIVITIES
INPUTS
ASSUMPTIO
NS
27
Example – Renewable
Energy
Increased share of
renewable energy in
energy use
Objective (pre-defined)
Expected outcome(s)
(pre-defined)
Outputs
More energy
efficient RES in
place
Types of
projects
•Modernised RES
infrastructure
•R&D on RES
•Feasibility of RES
mapped out in
relation to local
conditions
Increased awareness of and
education in renewable
energy solutions
Increased renewable
energy production
X policies at local
and regional level
to stimulate RES
developed
•Training in RES
planning
competence
•Plans/policy
development
X awareness
raising
programmes at
local level
carried through
X training courses in
RES provided to
officials at local and
regional level
•Awareness raising
campaigns at local
level
•Training courses
for officials at
regional level
•Train the trainers
•Training courses
for officials at
regional level
100 MW capacity
RES constructed
and in operation
•Windmills
•Solar systems
•Hydropower
•Bioenergy
•
•
•
•
•
•
•
•
•
•
28
EXAMPLE Gender-based
violence
Gender-based
violence prevented
and tackles
Objective (pre-defined)
Expected outcome(s)
(pre-defined)
Services for
women at risk of
violence designed
Outputs
Types of
projects
•
Support to
existing centers
•
Support to start
new centers
•
•
Support to
provide services
for women at
risk such as
counseling,
phone-lines
Training
Domestic violence reduced
Gender-based
violence reduced
A national
coordination centre
for service provision
established
•
•
New
coordination
center
Support for
coordination
activities
•
Training
•
Methodologies
and guidelines
development
•
Studies
Procedures for work
in crisis center
standardised
Services for
victims of
domestic violence
improved
Awareness around
gender based
violence increased
•
Media campaign
•
•
Education
activities
Training for
counselors
•
Support to start
new counseling
centers
•
Curriculum
development
•
Training of
police and
medical
personnel at
local and
regional levels
29
•
Support to
develop new
standards and
methodologies
•
Training
•
Manual
developed
Call for proposals
Be clear about which outcomes and outputs you
want the projects to contribute to.
Which projects are necessary and sufficient?
Project template in DoRIS:
http://www.eeagrants.org/id/3192
30
Logical relationship
31
PART IV
Basis for results
measurement
32
BASIS FOR RESULTS
MEASUREMENT
Result based management
33
"The starting point for a results-oriented approach is ex
ante setting of clear and measurable targets and
outcome indicators. Indicators must be clearly
interpretable, statistically validated, truly responsive
and directly linked to policy intervention, and promptly
collected and publicised."
From the conclusions of the 5th Cohesion Report, adopted by the
European Commission in November 2010
QUALITY AT ENTRY!
30
INDICATOR
• A means to measure
achievement, or to help assess a
development
35
HOW SHOULD INDICATORS
BE FORMULATED?
• Once indicators have been identified, they should be
formulated to reflect measurable change. Indicators
frequently utilize the following metrics to depict measurable
amounts:
• Number;
LESS IS MORE
• Ratio;
• Percentage; or
• Rate;
• Time
Indicators shall whenever relevant, be broken down per gender
In DoRIS, always specify unit of measurement!
36
EXAMPLES OF INDICATORS
Example 1:
Outcome: Improved health status of young children in region/district X
Indicator: Percentage of children under 6 years who
are immunized against x y z diseases in district A
Example 2:
Outcome: Increased participation of worker’s
organisations in policy development
Indicator: Number of cases in which the submitted
comments of worker’s organisations on draft legislation
are reflected in final legislation
37
INDICATORS
To describe how results will be measured, indicators
must be specific, observable and come from reliable
data.
• Does the indicator provide the most direct evidence
of the condition or result to be measured?
• Is the indicator important, understood, and accepted
by important stakeholders?
• Is the indicator defined in the same way over time?
38
• Are data for the indicator collected in the same way
over time?
• Will data be available for the indicator?
• Will data be collected frequently enough?
• Are data currently being collected?
• If data are not currently collected, can cost effective
instruments for data collection be developed
quickly?
39
Indicators related to
Bilateral Relations
• All programmes should contribute to
strengthened Bilateral Relations
• Use the Bilateral guideline.
Four types of bilateral results expected:
Extent of cooperation
Shared results
Wider effects
Knowledge and mutual understanding
40
Indicators related to
«Horizontal issues»
• A human-rights based approach and concerns for
horizontal issues like fighting discriminiation, racism,
hate speech etc. should guide defining the outputs
and the indicators.
Example: X% of members of Parliament are minorities.
Example: Incidences of hate speech has decreased in Y
school
41
Programme matrix indicators
Sources of Verification
• When defining the indicators, consider the data sources.
Where can you easily get data that are reliable and timely?
• Specify your sources of information
• Assess where you can get easy access to data
Example
Ministry of Health’s statistics on cancer (health status of
population) and various statistics on cancer detection and
treatment services
Programme matrix –
Sources of Verification
WHAT ARE BASELINES?
• Baselines provide context for the setting of
targets and capture the situation before a
development intervention begins, or at the
beginning of a time period that will be
monitored and assessed.
• Baselines describe the conditions prior to
programming efforts. The present situation!
• Essential to make credible and meaningful
assessments on progress towards outcomes.
45
Programme matrix –
Baseline data
WHAT ARE TARGETS?
• Targets set the level for the magnitude of change
expected by the end of a predetermined time period.
They reflect what organizations commit themselves to
achieve by the end of some duration of time.
• Targets provide tangible and meaningful points of
discussion with beneficiaries, stakeholders, and
partners, and allow us to add further specificity to the
outcomes from the results logic.
47
TARGETS
• When you have defined the indicators and know the baseline
situation, you set the targets.
• What do you want the situation for the beneficiaries to be in 3 or 5
years?
• Examples:
• 1. 60% of W 40+ called in for cancer test in Southern Region by year
2016.
We want to increase the % of W 40+ called in from 30% in 2010 to
60%
in 2016.
• 2. Time lag between detection of cancer to operation is not more
than 3
months in 2016.
48
BASELINES AND TARGETS
SUMMARY
• A baseline is the value of the indicator before the
implementation of the project starts
• Targets orients stakeholders to the tasks to be
accomplished
• Targets help establish clear expectations
• Targets serve as the guideposts for monitoring
whether progress is being made
• Targets promote transparency
49
Programme matrix Targets
43
Experience: The EU
In 2010 the EU evaluated the 4,131 impact indicators under the ERDF
programmes. Among the programmes;
• 94% had final achievements;
• 58% had targets;
• 6% had baselines;
• 55% had targets and achievements;
• 5% had baselines, targets and achievement; and finally
• 0.5% had baselines but no targets and no achievements.
How can we assess impacts if we have no idea of the starting point?
Conclusion: Much of the data collected are not meaningful and do not
represent the impact of the Structural Funds.
51
Part IV Quality at implementation
• Monitoring
• Reporting on results and risks
• Evaluation
52
PROGRAMMING CYCLE
The Programming Cycle and Continuous Feedback
Planning
Reporting
Programming
Budgeting
Evaluation
Monitoring
Implementation
53
PERFORMANCE
MONITORING
•
•
•
•
•
•
Define processes and systems for performance
measurement;
Decide the intervals at which measurement will occur;
Assign roles and responsibilities;
Set standard requirements for project promoters
Build in adequate time and resources for analysis of
performance measurement data so that progress is reviewed
and assessed.
Use web site for communication with project promoters and
other stakeholders
54
Performance monitoring
• Status regarding the output and outcome
indicators is measured and reported
towards the baseline situation (“where
we were”), and also towards the future,
desired situation (“the targets”).
The Matrix is the point of reference
throughout the life of the programme
National/
sector area
National/
sector area
Programme implementation
Programme
area
Programme
area
Programme
Present situation
Baseline
Future situation
Programme performance
framework
Reporting requirements in annual
report from Programme Operators
Programme outcome
• Report and analyse status of the selected indicators for the expected programme outcome(s)
• Analyse how the projects and the programme outputs have contributed to the expected
programme outcomes.
Programme outputs
• Report and analyse status of the selected indicators
• Summerize and analyse how projects have contributed to the planned programme outputs
Outcome and outputs of bilateral relations
• Report and assess achievemenet of outcome and outputs of bilateral relations
Assess risks and risk mitigation, cross cutting issues, sustainability, and outreach to target groups
58
EVALUATIONS
• Evaluations have three key functions:
(1) Learning: As an input to provide decision-makers
with knowledge and evidence about performance
and good practices;
(2) Accountability: To donors, funders, political
authorities, stakeholders and the general public, and
(3) Communication: Results for the beneficiaries
THANK YOU!
59