Monitoring, Evaluation, Reporting and Improvement Strategy

advertisement
Monitoring, Evaluation, Reporting and
Improvement Strategy –
Caring for our Country and the
Biodiversity Fund
April 2013
Contents
1 Introduction
4
2 Purpose of this strategy
6
3 MERI principles and Key Evaluation Questions
7
4 Implementing MERI — funding recipients (project level MERI)
8
5 Implementing MERI — the Australian Government (program level MERI)
13
6 MERI Toolkit
15
7 Terminology
15
1 Introduction
To help understand how investments to improve our natural resource management (NRM) and
biodiversity conservation are making a difference, information about where investments are made
and what they are achieving is needed. This information can also be used to help understand how we
can improve the design and delivery of projects and programs.
Monitoring, Evaluation, Reporting and Improvement (MERI) are simple concepts that, when applied,
help us understand what is being achieved and help identify possible improvements, for projects
and programs. The figure below provides a simple depiction of the MERI cycle.
Figure 1: The cycle on monitoring, evaluation, reporting and improvement
Funding recipients and the Australian
Government have a role to play in ensuring that
Caring for our Country and the Biodiversity
Fund are adequately monitored, evaluated and
reported to allow identification of achievements
and integration of improvements.
Investments in NRM and biodiversity
conservation are made through a number of
different funding streams. Each of these
funding streams uses specific documentation
and processes. However, information from
each of these streams will be used to
understand progress towards the broader
Caring for our Country and Biodiversity Fund
strategic objectives. Data and information from
the following will be collected and reported as
part of this MERI Strategy:

The Biodiversity Fund1

Caring for our Country, including:
 Regional Delivery
 Sustainable Environment stream
 Sustainable Agriculture stream
Implementation of this MERI Strategy will be
supported by a range of other documentation
that sets specific requirements for recipients
under the relevant funding streams, including
program guidelines, funding agreements,
and planning and reporting templates
(provided as part of the MERI Toolkit
(see section 6). Further information on Caring
for our Country and the Biodiversity Fund can
be found at the Caring for our Country and
Biodiversity Fund web pages.
1
Biodiversity is one component of the broader Land Sector Package,
which also includes Regional Natural Resource Management
Planning for Climate Change Fund and the Indigenous Carbon
Farming Fund. These funds will implement separate though
complimentary MERI processes to this strategy.
This MERI Strategy differs from the previous
Caring for our Country MERI strategy in that it:

Provides a consistent and integrated
approach to monitoring evaluating,
reporting and improvement for both Caring
for our Country and the Biodiversity Fund.

Clarifies the role of funding recipients in
providing project level data, and
strengthens the processes which
enable this.

Strengthens the Australian Government
commitment to making as much
information as possible public, to help
maximise learning about how best to
invest to address the natural resource
management and biodiversity conservation
challenges that Australia faces.
2 Purpose of this strategy
This document outlines the approach needed
from funding recipients and the Australian
Government, to ensure:

Funding recipients can track and report on
project progress and achievements.

The Australian Government can track
program progress and report on program
achievements, including their impacts on
the environment, natural resource base
and community.

Information collected along the way is used
effectively, including by funding recipients
and the Australian Government to inform
ongoing and future design and delivery of
projects and programs.

Accountability of public expenditure.
To build a picture of how our NRM and
biodiversity conservation investments are
making a difference, information from a range
of sources is needed. This MERI Strategy aims to
support the collection and communication of
information at three levels, as depicted in
Figure 2.
Figure 2: Information from three sources will be used to
build a comprehensive picture of program achievements
3 MERI principles and Key
Evaluation Questions
The following principles should be applied
when designing and implementing MERI at
both project and program level:


Fit for purpose: Tailored to the design,
purpose and objectives of Caring for our
Country and the Biodiversity Fund projects
and programs.
Credible: Based on scientifically and
administratively defensible methods
and approaches.

Transparent: Clearly demonstrate how
public money has been spent, the resulting
outputs, and where possible outcomes.

Cost effective: The cost of MERI activities
is within allowable limits, and where
possible draws on and links with existing
processes and information.
The following Key Evaluation Questions should
help frame the broad approach to MERI at the
project and program level. Based on the general
Key Evaluation Questions below, project and
programs may develop more specific project or
program evaluation questions.

Effectiveness: Were the planned activities
performed, and to the standard required?

Impact: What is the outcome, or result of
having done the planned activities i.e. what changes in the condition of
environmental assets, changes in
management practices and/or changes
in institutions can be observed?

Appropriateness: Was this the best
way to have addressed the problem?
Did the activities and the way they were
undertaken align with stakeholder needs
and expectations. Were they consistent
with relevant NRM policy and priorities?

Efficiency: Did the project / program
achieve the desired results within budget?
To what extent did the project / program
attain the highest value from available
resources – could the same outcomes be
achieved more simply and for less cost
next time?

Legacy: Will the project / program
continue to have an impact after
its completion?
4 Implementing MERI – funding
recipients (project level MERI)
Funding recipients can allocate a portion of
their total project budget to MERI activities,
in accordance with the relevant funding
stream guidelines.
In general, Caring for our Country and
Biodiversity Fund funding recipients will
be required to:
1. Develop a Project MERI Plan (see 4.1 below).
4.1 Project (MERI) Plans
Caring for our Country and Biodiversity Fund
recipients may be required to develop a plan
(using the funding stream specific templates),
at the commencement of their project, which in
general aims to:

Document the detail and structure of the
project. This helps identify which aspects
or stages of the project should be
monitored (some project templates will call
for the development of a program logic —
see text box on page 9)

Monitor and evaluate progress and collect
evidence of achievements (see 4.2)

Provide detail of communication activities.
Communication of project activities,
experiences, and achievements amongst
funding recipients, the community and
the broader public is an important aspect
of MERI.
2. Undertake monitoring and evaluation
activities to collect information as evidence
of project progress and achievements
(see 4.2 below).
3. Provide progress reports throughout the life
of the project, and a final report upon
completion of the project (see 4.3 below).
MERI requirements differ for the Biodiversity
Fund and Caring for our Country. Biodiversity
Fund projects are relatively similar and so
monitoring and evaluation activities are largely
predetermined. Caring for our Country will
invest in a wide range of projects and so
individual MERI approaches will need to
be developed.
NOTE: Not all Caring for our Country funding
programs refer to ‘Project MERI Plans’
specifically -specific MERI related requirements
are detailed in relevant funding stream
specific documentation, including guidelines,
funding agreements and templates.
The type and complexity of monitoring and
evaluation required for Caring for our Country
funded projects will depend on the type and
size of the project. This is reflected in relevant
templates, with smaller grant recipients
required to develop a simple project MERI
plan, while larger projects will be required
to develop a more comprehensive plan.
Further information on the MERI requirements
for Caring for our Country is described in
section 4.2.
What is program logic?
Program logic is an approach to project or program planning that illustrates simply, the logic behind
how activities lead to outcomes. Program logic is usually represented as a diagram or matrix,
which shows a series of expected consequences eventually resulting in the final outcome.
This structure helps identify where monitoring activities should be integrated throughout the
project, to show whether the project is progressing as planned and demonstrate achievements.
The diagram below illustrates the types of ‘outcomes’ that occur at the different stages of a project
or program. Example program logics are provided in the MERI Toolkit (see section 6).
Figure 3: Basic structure of a NRM program logic
4.2 Project monitoring
and evaluation
Funding recipients are responsible for
collecting and reporting project level data that
demonstrates progress and achievement of
project outcomes. The type of information that
is collected will vary depending on the type and
size of the project. It will also depend on the
particular outcomes that are being monitored
or evaluated. Generally speaking, funding
recipients have less control and influence on
longer term outcomes (see Figure 3, above) and
so these outcomes can be harder and more
costly to monitor and evaluate.
4.2.1
Caring for our Country
The way monitoring and evaluation is
undertaken at the project level will vary
across the broad variety of Caring for our
Country projects.
When thinking about the type and purpose of
monitoring and evaluation at the project level,
funding recipients should consider:

The type of monitoring and evaluation
activities that are feasible given the
available MERI budget (set out in the
funding agreement) and project
team capacity.

Key elements (activities or outcomes) of
the project that should be monitored and
evaluated to provide evidence of progress
and /or achievements — what is key in
understanding the progress and success of
the project?

Program outcomes and the types of
monitoring at the project level that would
help demonstrate the project’s contribution
to the program outcomes.

Existing monitoring and evaluation
approaches that can be used to ensure
consistency across projects, within regions,
across the program etc.
This section of the strategy provides an outline
for funding recipients of both Caring for our
Country and the Biodiversity Fund to guide
monitoring and evaluation activities.
As part of its commitment to MERI,
the Australian Government will advise on
possible monitoring and evaluation
approaches (including standard indicators)
to promote a consistent approach to
collecting and assessing project level data.
This advice will be made available on the
Caring for our Country website.
To fulfil monitoring and evaluation activities
funding recipients can use their MERI budget
to purchase third party specialised services.
For example a project may choose to engage a
third party to provide specialised monitoring,
or to assess monitored data and information
and provide an evaluation of their project.
Intentions to engage a third party evaluator
should be detailed in the Project MERI Plan.
Table 1 (below) indicates the types of monitoring approaches at different project levels
and for different project capacities, and provides some example evaluation questions.
Project
outcome level
Purpose of
monitoring
Evidence need
Monitoring approach
(e.g)
Evaluation
Questions (e.g)
Longer term
project outcome
To help
understand the
NRM or
biodiversity
outcome
(or impact) of
the project.
Evidence of asset
condition change
(e.g. enhanced
ecosystem health)
• (existing or new) scientifically
robust methodology applied.
• Use (existing or new)
indicators of change
• What was the impact
of the project?
• Were there
unintended impacts
• Were there
external factors
which impacted
on the project.
Intermediate
project outcome
To help
understand
whether the
project
(effectively)
achieved the
management or
conservation
activities it set
out to.
Evidence of
project deliverable
(e.g. reduction in
density of
weeds / pests)
Evidence of better
practice adopted.
• Baseline and post activity pest
numbers on project site
• Photo point (GPS linked)
documentation
• Follow-up survey with
participants
• Recognised ‘management’
actions recorded
• Did the project
achieve what it set
out to and to the
standard expected,
and within budget?
• What could have
been done
differently / better?
Immediate project
activities and
outcomes
To understand
whether a
specific project
activity
occurred and
had the
intended result
(was
appropriate) in
order for the
project to
progress.
Evidence of
revegetation
activities completed
Evidence of workshop
attendance
• Photographic documentation
• Registration of attendees
and attendee satisfaction
with workshop
• Was the activity
completed and
did it have the
intended result?
• Was the activity
appropriate, or
would a different
approach be better?
4.2.2
Biodiversity Fund
All Biodiversity Fund projects, with the
exception of Round 1 projects, with approved
Funds greater than or equal to $500,000 (GST
exclusive) are required to collect ecological
monitoring data and report the data collected to
the department. This data can be collected using
one of the following recognised methods, the
department may add additional methodologies
to this list:






Habitat Hectares
BioMetric: Terrestrial Biodiversity
Tool
BioCondition
TasVeg: Tasmanian Vegetation
Condition Assessment Method
Bushland Condition Monitoring
Native Vegetation Condition
Assessment & Monitoring for WA
If you do not intend to use one of these listed
monitoring methods, you must then collect and
report ecological monitoring data to the
department in accordance with the
department’s Biodiversity Fund Ecological
Monitoring Guide.
All Biodiversity Fund funding recipients will be
required to answer the evaluations questions
provided by the Australian Government within
the relevant reporting templates. This will
require providing simple data about the project
and some observations about how the project is
progressing. This data and information will be
used to help understand how Biodiversity
Fund projects are contributing towards the
program objectives.
The reporting templates and the Biodiversity
Fund Ecological Monitoring Guide are available
at the Biodiversity Fund website.
4.3 Project reporting
Funding recipients will be required to report
periodically to the Australian Government on
the progress of their project, and provide a final
report on project completion, using the relevant
reporting templates. Information from these
reports will be used by the Australian
Government to communicate publicly the
location, objectives and other details of projects.
The Australian Government will also
consolidate project level reports to track
progress and communicate program outcomes
to stakeholders and the broader community.
Periodic and final reporting will
generally involve:

Summarising delivery of project activities
in the reporting period including a
summary of monitoring results.

Reporting progress towards project
outcomes, including using available
standard measures (provided in
reporting templates).

Identifying lessons learnt and necessary
changes to the project design or delivery

Providing any other relevant information
including maps, photos, case studies,
raw data etc.

Financial reporting.
5 Implementing MERI –
the Australian Government
(program level MERI)
The Australian Government is committed
to accounting for, and demonstrating
outcomes from, its NRM and biodiversity
conservation investments.
The Australian Government will be responsible
for program level MERI for Caring for our
Country and the Biodiversity Fund, including:

Providing advice and developing processes,
systems and MERI documentation to
support funding recipients undertake
their MERI responsibilities.

Collating, analysing and reporting on
consolidated project data.

Storing and making publicly available,
relevant project and program information.

Supporting program level monitoring,
research and evaluation activities across
themes, landscapes or regions.

Providing timely updates on program
progress and achievements.
Separate program level MERI plans for the
Caring for our Country Sustainable
Environment and Sustainable Agriculture
streams, and Biodiversity Fund have been
developed to outline the strategic objectives,
outcomes and investment approaches for
monitoring and evaluation at the program level.
These will be available on the Caring for
our Country and Biodiversity Fund websites.
Where applicable, Program MERI Plans will also
outline relevant project level monitoring
approaches (including standard indicators),
which align with the program monitoring
approach. These plans will be updated from
time to time, and will be available on the Caring
for our Country and Biodiversity Fund websites.
5.1 Program monitoring
The Caring for our Country and Biodiversity
Fund program monitoring will be underpinned
by three types of monitoring data (also see
Figure 2):
1. Project information — information
provided by funding recipients to the
Australian Government will be collated,
analysed and reported as it becomes
available. Project information will be
presented for individual projects and
also help provide snap shots on
program progress.
2. Targeted monitoring — The Australian
Government will support monitoring
activities, which complement information
provided at the project level, to enable an
understanding of how the program is
progressing towards its objectives and
impacting on NRM and biodiversity assets.
Monitoring activities may focus on particular
geographic areas (including project sites),
themes, and assets. Results from monitoring
activities will inform evaluation processes
(see 5.2).
3. Landscape scale, research and other data
— Information from existing sources will be
drawn on to provide further evidence of
changes to the natural resource base, and as
context for understanding the programs
impact. This may include remotely sensed
data (such as extent of native vegetation),
or supporting relevant scientific research.
5.2 Program evaluation
5.3 Program reporting
Specific theme or asset based evaluations may
be undertaken, for Caring for our Country and
the Biodiversity Fund to provide further depth
and analysis of the achievement and / or to
inform future program design and delivery.
As a way of sharing information and
demonstrating results as they occur,
the Australian Government will make
project and program information publicly
available. To date this has occurred through
annual departmental and program progress
reports. The Australian Government intends to
move towards a more detailed and interactive
reporting approach, publicly reporting project
and program level data, as it becomes available.
Information will continue to be communicated
through the following regular reports:
Evaluations for Caring for our Country and the
Biodiversity Fund will aim to assess progress
towards program objectives, as well program
administration and delivery. They will take into
account input from funding recipients and other
stakeholders, and will assist with program
improvements and any future program design.
They will also focus on the achievements and
NRM and biodiversity conservation outcomes
from Caring for our Country and the
Biodiversity Fund.
Annual departmental reports to demonstrate
accountability and report on progress to
Parliament and the public. For the Biodiversity
Fund, this will also include the annual reports of
the Land Sector Carbon and Biodiversity Board.
Program progress updates will provide a
platform to provide updates on program
progress, highlight particular achievements and
share information and case studies from the
project level.
Monitoring and evaluation reports generated
through the Australian Governments targeted
scientific monitoring approach will be
published and made publicly available
upon completion.
6 MERI Toolkit
The following documents will directly support
the implementation of this MERI Strategy,
and may be updated from time to time:
Available on the Caring for
our Country website
http://www.nrm.gov.au/funding/meri

Caring for our Country Project MERI Plan
template (under $80,000)

Caring for our Country Project MERI Plan
template (over $80,000)

MERI FAQ

Example Project MERI Plan for Caring for
our Country

Example Program Logic

Sustainable Environment Stream Program
MERI Plan

Sustainable Agriculture Program MERI Plan
The following are useful reference materials for
the development and implementation of MERI:

Australian Government Natural Resource
Management Monitoring Evaluation
Reporting and Improvement Framework.

Developing and Using Program Logic in
Natural Resource Management
Available on the Department of
Sustainability, Environment, Water,
Population and Communities, Land Sector
Package, Biodiversity Fund Website
http://www.environment.gov.au/
cleanenergyfuture/biodiversity-fund/meri

Biodiversity Fund Project MERI
Plan template

Biodiversity Fund Reporting Templates

Biodiversity Fund Ecological Monitoring
Guide and guidance video

Example project MERI plan for
Biodiversity Fund
7 Terminology
Achievement
For the purpose of this MERI Strategy ‘achievement’ is a general term
referring to the realisation of particular activities and outcomes.
Baseline
The state or condition of an asset (environmental, social etc) prior to an intervention
(project or program)
Evaluation
A systematic process of information collation and analysis to assess the impact,
appropriateness, effectiveness, efficiency or legacy of a project or program.
Monitoring
The (preferably regular) collection of data / information, usually to assist timely
decision making, and demonstrate changes over time.
Objective
A desired result of an activity, project or program
Program level
Refers to Australian Government MERI responsibilities and activities at whole of
Caring for our Country program and the Biodiversity Fund program.
Program Outcome
The Strategic Objectives or 5 Year outcomes of Caring for our Country.
Project Outcomes
The results of project activities. These may be immediate, intermediate or longer term
outcomes.
Project level
Refers to funding recipient MERI responsibilities and activities
Funding recipient
Person or organisation that receives funding from the Caring for our Country or
Biodiversity Fund program(s).
Australian Government
The Australian Government as represented by the Department of Agriculture, Fisheries
and Forestry and /or the Department of Sustainability, Environment, Water, Population
and Communities.
Download