Performance Measurement
in the Conservation Community
Status, progress, barriers, and next steps
Background material for the
Measuring Conservation Effectiveness Summit
May 5-6th, Palo Alto, CA
INTRODUCTION
To prepare for the May 5-6th summit on Measuring
Conservation Effectiveness, 29 conservaiton implementers
and funders were surveyed to ask about their Systematic
Performance Measurement (SPM) practice, which we
defined as:
The regular monitoring, evaluation, and adaptation of
conservation actions based on clearly stated goals,
objectives, and assumptions so as to assess
effectiveness, promote learning, and report achievements.
INTRODUCTION
Respondents included:
• 15 conservation implementers, representing more than 2 billion
US$ in conservation spending and an effort equivalent to 16,000
staff person years.
• 14 conservation funders, representing 1.1 billion US$ in annual
support spread across about 1,000 grants.
• Respondents varied in size,
but most had budgets
between 11 and 100 M USD.
12
Implementers
10
Funders
8
6
# of
respondents
4
2
0
0-1
1-10
11-100 101-1000 >1000
$ Millions
INTRODUCTION
Respondents included:
• Conservation Implementers
• Conservation Funders
Marisla
Foundation
Mulago
Foundation
INTRODUCTION
In practice, implementers say SPM takes many forms…
“annual plans and
performance reviews”
“OECD-DAC principles for
evaluation”
“project reviews against
set criteria”
“peer-review audits”
“information flow up to
organization level”
“implementation of the Open
Standards”
“reporting of key deliverables
against expected results”
INTRODUCTION
…as do funders.
“communication tool for
our board”
“venture capital type
investments with a clear
theory of victory”
“rigorous post-grant monitoring”
“requirement for grantees”
“tool to understand why a
funded project fails”
“constant and direct
assessments of progress”
“qualitative approach to evaluating
grantee results and sustained change”
INTRODUCTION
We note that small organizations seem to do SPM to a
greater extent than large organizations and this affected
our analysis.
We show results herein both by % of responding organizations and % of total
combined conservation spending, and graphics are coded accordingly.
For example, we asked: At the scale of projects & programs, is SPM practiced
well at your organization?
8
2000
By Org
6
By $$$
1500
# of
respondent
s
4
$ Million
1000
2
500
0
Almost never Occasionally
(<10%)
(10-49%)
Often (5090%)
Almost always
(>90%)
0
Almost never Occasionally
(<10%)
(10-49%)
Often (5090%)
Almost always
(>90%)
INTRODUCTION
Here‘s what we learned
1.
SPM is widely acknowledged as important...
2.
...but is not done to a great extent across our community…
3.
…however, we can improve by leveraging the progress
we have made…
4. and overcoming several critical obstacles.
SPM IS IMPORTANT IN CONSERVATION
1. SPM is widely acknowledged as
important in conservation.
SPM IS IMPORTANT IN CONSERVATION
SPM is generally viewed favorably and as a priority.
Survey Question: In general for each audience, how strongly do you agree/disagree
with the following statements:
Strongly
disagree
Has a positive/favorable attitude
towards implementing SPM
within the organization
Upper
Management
Program Directors
Program Directors
Project Managers
Project Managers
20%
40%
60%
80%
Strongly
agree
100%
By Org
Board
Upper
Management
0%
Moderately
agree
Views the implementation of SPM
in the near future as a high priority
By Org
Board
Moderately
disagree
0%
20%
40%
60%
80%
100%
SPM IS IMPORTANT IN CONSERVATION
When SPM is implemented, the primary driver is to
improve effectiveness.
Survey Question: If your organization does or attempts SPM, how important are
the following reasons behind that effort?
Not at all important
Minimally important
Somewhat important
Very important
To improve organizational effectiveness and/or efficiency
By Org
To enable project managers to evaluate effectiveness
To satisfy requests from board and/or upper management
To improve strategic fundraising
To facilitate adaptive learning of what works in conservation
To improve coordination and/or integration across programs
To improve evidence-based communication
To satisfy pressure from donors and/or supporters to show results
To guide funding decisions by board and/or upper management
To guide capacity building decisions by board and/or upper…
To satisfy requests from project managers
0%
20%
40%
60%
80%
100%
SPM IS IMPORTANT IN CONSERVATION
To assess and improve our effectiveness, there are
many important questions we want to answer....
Survey Question to IMPLEMENTERS. How important is answering the following
questions within your organization?
Not at all important
Minimally important
Somewhat important
Very important
Are our projects having their intended impacts?
Are our programs having their intended impacts?
By Org
Can credible results be demonstrated to our board, donors,
and supporters?
Are our actions being adapted and improved?
What can be learned to improve our organization's work?
Are our actions cost-effective?
Do we understand why a project/program fails when it does
so?
Can our actions be better coordinated across the org?
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
SPM IS IMPORTANT IN CONSERVATION
Survey Question to FUNDERS: How important is answering the following
questions within your foundation?
Not at all important
Minimally important
Somewhat important
Are our grantee's projects having their intended impacts?
Very important
By Funders
Are our funding programs having their intended impacts?
Are our grantee's actions being adapted and improved?
What can be learned to improve our foundation's work?
Do we understand why a grantee's project fails when it does so?
Can our funding be better coordinated with other foundations?
Are our foundation strategies regularly being adapted and improved?
Can credible results be demonstrated to our board?
Are our grantee's actions cost-effective?
Can our funding be better coordinated across the foundation?
0%
20%
40%
60%
80%
100%
SPM IS IMPORTANT IN CONSERVATION
While funders are fairly confident they can answer key
performance questions regarding their grantees…
Survey Question: Using your foundation's current SPM system(s), how well can your
foundation answer the following questions?
Not at all
Minimally well
Somewhat well
Can credible results be demonstrated to our board?
Very well
By Funders
Are our grantee's projects having their intended impacts?
What can be learned to improve our foundation's work?
Are our foundation strategies regularly being adapted and improved?
Can our funding be better coordinated with other foundations?
Do we understand why a grantee's project fails when it does so?
Can our funding be better coordinated across the foundation?
Are our funding programs having their intended impacts?
Are our grantee's actions being adapted and improved?
Are our grantee's actions cost-effective?
0%
20%
40%
60%
80%
100%
SPM IS IMPORTANT IN CONSERVATION
…many of the implementing organizations feel less
able to do so.
Survey Question: Using your organization's current SPM system(s), how well can your
organization answer the following questions?
Not at all Minimally well Somewhat well
Can credible results be demonstrated to our board,
donors, and supporters?
By Org
Very well
By $
Can our actions be better coordinated across the org?
What can be learned to improve our organization's
work?
Are our actions being adapted and improved?
Are our programs having their intended impacts?
Are our projects having their intended impacts?
Are our actions cost-effective?
Do we understand why a project/program fails when it
does so?
0%
50%
100% 0%
50%
100%
SPM IS NOT DONE WIDELY
2. That we often cannot answer key performance
questions with confidence is likely because...
...SPM is not done to a great extent across our
community.
SPM IS NOT DONE WIDELY
For every conservation dollar spent, we say that only
10-30 cents worth is guided by SPM.
1800
1600
Survey Question: Of the total budget
for your organization’s conservation
efforts, what % is guided by SPM?
1400
conservation 1200
spend
1000
$ Millions
800
600
400
200
0
0-20
21-40
41-60
61-80
% of budget guided by SPM
81-100
SPM IS NOT DONE WIDELY
Despite this estimate, we generally think we think we
are doing SPM pretty well...
8
6
My foundation is effective at
helping our grantees do SPM
well.
My organization does SPM well.
8
4
4
2
2
0
0
Strongly Moderately Moderately
disagree
disagree
agree
Strongly
disagree
Strongly
agree
By Funders
Our grantees do SPM well.
8
2000
6
1500
4
$ M 1000
2
500
0
0
Strongly Moderately Moderately
disagree
disagree
agree
Strongly
agree
By Org
6
Moderately Moderately
disagree
agree
Strongly
agree
My organization does SPM well.
By $$$
Strongly Moderately Moderately
disagree disagree
agree
Strongly
agree
SPM IS NOT DONE WIDELY
…but when we are pushed, we admit that we rarely put our
projects through a full SPM cycle.
Of ~7000
projects
currently undertaken by
implementing organizations,
only ~2500
have good conservation plans
in place, but…
only
~350-500
have completed the cycle of designimplement-monitor-evaluate-adapt.
SPM IS NOT DONE WIDELY
We generally do well at basic design and
implementation of projects,
but things break down when it comes to rigorous
design, and monitoring, evaluation, and adaptation.
SEE NEXT 2 SLIDES
Survey Question: For your projects, indicate the extent to which your organization does each of
the following SPM practices well.
None
Few
Some
Almost All
A well-defined scope of work
Identification & outreach to key stakeholders in planning process
By Org
Identification & outreach to key organizational partners in planning process
Identification of conservation targets
DESIGN
Identification of threats
Prioritization of threats to be addressed
Situation analysis
Identification of conservation actions
Prioritization of conservation actions to be implemented
Articulation of logic models for conservation actions
IMPLEMENT
Development of conservation action plan
Development of operational plan
Implementation of operational plan
MONITOR
Development of monitoring plan
Implementation of monitoring plan
EVALUATE
& ADAPT
Assessment of conservation status
Assessment of performance of conservation action
Use of data from monitoring and assessment to adapt future conservation action
SHARE
Sharing of lessons learned internally
Sharing of lessons learned outside the organization
0%
20%
40%
60%
80%
100%
These patterns seem even more apparent in
None
larger implementing organizations.
Few
Some
Almost All
A well-defined scope of work
Identification & outreach to key stakeholders in planning process
By $$$
Identification & outreach to key organizational partners in planning process
Identification of conservation targets
DESIGN
Identification of threats
Prioritization of threats to be addressed
Situation analysis
Identification of conservation actions
Prioritization of conservation actions to be implemented
Articulation of logic models for conservation actions
Development of conservation action plan
IMPLEMENT
Development of operational plan
Implementation of operational plan
Development of monitoring plan
MONITOR
EVALUATE
& ADAPT
SHARE
Implementation of monitoring plan
Assessment of conservation status
Assessment of performance of conservation action
Use of data from monitoring and assessment to adapt future conservation action
Sharing of lessons learned internally
Sharing of lessons learned outside the organization
0%
20%
40%
60%
80%
100%
KEY INGREDIENTS TO PROGRESS TO DATE
3. To improve our ability to measure
effectiveness, we can leverage several
ingredients key to our progress to date.
KEY INGREDIENTS TO PROGRESS TO DATE
Institutional mandate and SPM champions have been
key ingredients to the progress we’ve made to date.
Survey Question: Where you do see SPM happening in your organization, how important were the
following 'key ingredients' or 'catalysts' to SPM adoption?
Not necessary
Moderately useful
Very important
Absolutely essential
Institutional mandate
Presence of a champion within organization
By Org
Evidence that SPM led to increased effectiveness and/or efficiency
A vision for what could be accomplished with SPM
Dedicated funding for SPM
Dedicated SPM program with staff supporting implementation
A comprehensive plan that integrated SPM
Donor requirement to adopt SPM
Donor reporting requirement
Software tools that support SPM collection, management, &…
Seeing it being successfully implemented by other conservation or…
0%
20%
40%
60%
80%
100%
CRITICAL OBSTACLES TO PROGRESS
Institutional mandate is a key driver of SPM adoption.
Survey Question: Is SPM mandated at your organization? Is it practiced well at your
organization?
100%
By Org
Almost always
practiced well
80%
Often practiced well
60%
Occasionally
practiced well
40%
20%
Almost never
practiced well
0%
Informally
Promoted
Formally
Promoted
Mandated
KEY INGREDIENTS TO PROGRESS TO DATE
The 4 implementers that say that 80-100% of spending is guided by SPM
reinforce the importance of institutional mandate, evidence that SPM
helps, and champions.
Survey Question: Where you do see SPM happening in your organization, how important were the following
'key ingredients' or 'catalysts' to SPM adoption?
Not necessary
Moderately useful
Very important
Institutional mandate
Absolutely essential
By Top Implementers
Evidence that SPM led to increased effectiveness and/or efficiency
Presence of a champion within organization
A comprehensive plan that integrated SPM
A vision for what could be accomplished with SPM
Dedicated funding for SPM
Donor requirement to adopt SPM
Donor reporting requirement
Dedicated SPM program with staff supporting implementation
Software tools that support SPM collection, management, &…
Seeing it being successfully implemented by other conservation or…
0%
20%
40%
60%
80%
100%
KEY INGREDIENTS TO PROGRESS TO DATE
Funders also emphasize the importance of a mandate, evidence that
SPM helps, having a workplan that includes SPM, SPM being integral
to an org’s mission and goals, and dedicated SPM funding.
Not necessary
Moderately useful
Very important
Institutional mandate
Absolutely essential
By Funders:
Evidence that SPM led to increased effectiveness and/or efficiency
Perception of
Grantees
A workplan of activities that integrated SPM
SPM as an integral part of organization's mission & goals
Donor reporting requirement
Dedicated funding for SPM
Donor requirement to adopt SPM
Presence of a champion within organization
Seeing it being successfully implemented by other conservation or
development NGO
Dedicated SPM program with staff supporting implementation
Software tools that support SPM collection, management, & reporting
0%
20%
40%
60%
80%
100%
CRITICAL OBSTACLES TO PROGRESS
4. To improve our ability to measure
effectiveness, we also must overcome
several critical obstacles.
For all organizations, these primarily include:
• Lack of time
• Lack of money
• Lack of staff dedicated to SPM
• Perception that SPM is too complex
In addition to these, obstacles particularly important
in larger organziations include:
• Lack of donor pressure
• Lack of board pressure
• Lack of demand from upper management
CRITICAL OBSTACLES TO PROGRESS
Funding dedicated to SPM represents 1-5% of total
conservation spending by implementers.
2500
2000
$ Millions
By $$$
1500
1000
500
0
1-5%
5-20%
>20%
% conservation spend spent on SPM
10
8
6
# of
respondents 4
2
0
By Org
1-5%
5-20%
>20%
CRITICAL OBSTACLES TO PROGRESS
For every 100 projects, there is ~1 staff member
dedicated to SPM at implementing organizations.
Other staff data:
• Collectively, there’s about 1 conservation
coach for every 60 projects and 150
organizational staff.
• There’s about 1 conservation trainer for
every 140 projects.
By Org
16,400
organizational
staff
71 dedicated
SPM staff
• There’s about 1 auditor for every 500
projects.
• At organizations, there is a ratio of 230 : 1
organizational staff to SPM staff
• At foundations, the ratio is 30 : 1
• There’s about 1 foundation staff dedicated
to SPM for every 90 grants
• At foundations, about 20% of
conservation program staff’s time is
dedicated to SPM
By Funders
578
foundation
staff
19 dedicated
SPM staff
CRITICAL OBSTACLES TO PROGRESS
Understanding of what SPM is and how it might benefit
an organization seems weak.
Survey Question: In general for each audience, how strongly do you agree/
disagree with the following statements?
Moderately
Strongly
Moderately
disagree
By Org $$$
Has a thorough understanding of what SPM
is and how it is implemented at the
organization
Board
Upper Management
Upper Management
Program Directors
Program Directors
Project Managers
Project Managers
20%
40%
60%
80%
100%
Strongly
agree
Has a thorough understanding of what the
possible benefits of SPM are to the
organization
Board
0%
disagree
disagree
0%
20%
40%
60%
80% 100%
CRITICAL OBSTACLES TO PROGRESS
The following four slides show responses to the
survey question:
In your experience, how important are the
following factors in impediting adoption of SPM?
The data is presented four ways:
• Slide 33: By Implementing Organization responses
• Slide 34: By Implementing Dollars spent
• Slide 35: By Implementing Organizations that report less
than 20% of their budget is guided by SPM
• Slide 36: By Funder responses: perception of grantees
Not an issue
Minor hurdle
Major barrier
SPM: Obstacles to SPMLack of time
SPM
Extreme Survey
obstaclePage <#>
By Org
Lack of money
Lack of staff dedicated to SPM
Perception that SPM is too complex
Lack of training
Money,
time,
and
trainers
the most important barriers.
Lack of an overall
culture of
accountability
to our bottomare
line
Lack of quality trainers
Lack of good examples of SPM helping to achieve conservation…
Lack of incentives to change the status quo
Lack of support and/or interest from project managers
Lack of donor pressure
Lack of board pressure
Lack of demand from upper management
Lack of support from upper management
Lack of database exchange to share practices and learning
Perception that SPM is too simplistic
Perception that SPM is unnecessary to doing effective conservation
Lack of peer pressure
Lack of good software tools to implement SPM
Reporting guidelines of major donors discourage or inhibit SPM
Perception that SPM has become a meaningless buzzword
0%
20%
40%
60%
80%
100%
Not an issue
Minor hurdle
Major barrier
SPM
Extreme obstacle
Survey Page <#>
Lack of money
By $$$
Perception that SPM is too complex
Lack of donor pressure
Lack of board pressure
Lack of time
Lack of demand from upper management
Lack of staff dedicated to SPM
Lack of an overall culture of accountability to our bottom line
Lack of incentives to change the status quo
Lack of support from upper management
Lack of good examples of SPM helping to achieve conservation…
Lack of quality trainers
Lack of training
Lack of support and/or interest from project managers
Lack of peer pressure
Perception that SPM is unnecessary to doing effective conservation
Perception that SPM has become a meaningless buzzword
Reporting guidelines of major donors discourage or inhibit SPM
Lack of good software tools to implement SPM
Lack of database exchange to share practices and learning
Perception that SPM is too simplistic
0%
20%
40%
60%
80%
100%
By the 5 implementers
that say <20% of budget is
guided by SPM
Not an issue
Minor hurdle
Major barrier
Extreme obstacle
Lack of money
Lack of time
Lack of staff dedicated to SPM
Lack of donor pressure
Lack of quality trainers
Perception that SPM is too complex
Lack of board pressure
Lack of support from upper management
Lack of training
Lack of an overall culture of accountability to our bottom line
Lack of incentives to change the status quo
Lack of demand from upper management
Reporting guidelines of major donors discourage or inhibit SPM
Lack of good examples of SPM helping to achieve conservation goals
Lack of good software tools to implement SPM
Lack of database exchange to share practices and learning
Perception that SPM has become a meaningless buzzword
Lack of support and/or interest from project managers
Perception that SPM is unnecessary to doing effective conservation
Lack of peer pressure
Perception that SPM is too simplistic
0%
20%
40%
60%
80%
100%
By Funders:
Perception of
Grantees
Not an issue
Minor hurdle
Major barrier
Extreme obstacle
Lack of time
Lack of money
Perception that SPM is too complex
Lack of staff dedicated to SPM
Lack of an overall culture of accountability to our bottom line
Lack of demand from upper management
Lack of support from upper management
Lack of support and/or interest from project managers
Lack of good examples of SPM helping to achieve conservation goals
Lack of quality trainers
Lack of training
Lack of incentives to change the status quo
Perception that SPM is unnecessary to doing effective conservation
Lack of board pressure
Lack of donor pressure
Lack of peer pressure
Perception that SPM has become a meaningless buzzword
Lack of good software tools to implement SPM
Perception that SPM is too simplistic
Reporting guidelines of major donors discourage or inhibit SPM
Lack of database exchange to share practices and learning
0%
20%
40%
60%
80%
100%
CONCLUSIONS
Performance Measurement in the Conservation Community
1. SPM is important in conservation
 We say it is important and a top priority for the near future.
 We believe answering key questions regarding our effectiveness is very important.
 We generally do not believe we can answer these questions, however.
2. SPM is not done widely
 Only 10-30% of current conservation spending is guided by SPM.
 Very few projects do more than the initial steps of SPM
 Basic design and implementation of projects is generally done well, but things break down
when it comes to rigorous design, monitoring, evaluation, and adaptation.
CONCLUSIONS
Performance Measurement in the Conservation Community
3. We can improve by leveraging key factors to progress to date...
 Institutional mandate
 Champions for SPM within organizations
 A vision for what can be accoplished with SPM
 Evidence of SPM leading to increased effectiveness of conservation action
4. ...and by overcoming several critical obstacles.
For all organizations, these primarily include:
• Lack of time
• Lack of money
• Lack of staff dedicated to SPM
• Perception that SPM is too complex
In addition to these, obstacles particularly important in larger organziations include:
• Lack of donor pressure
• Lack of board pressure
• Lack of demand from upper management
THANK YOU!!
Thank you to all survey respondents!
Marisla
Foundation
Mulago
Foundation
Summit Research Committee
Sheila O'Connor (chair)
Bernd Cordes
William Crosse
Brett Jenks
Richard Margoluis
Matthew Muir
Elizabeth O'Neill
Nick Salafsky
Kristin Sherwood
Frog illustrations © Ted Kahn 2010
Original artwork used with permission
Neotropical Conservation Foundation
www.neotropicalconservancy.org