The Story of Washington’s Murder
(according to Bill Sutherland)
12th Dec 1799 Inspects plantation in snow
13th Sore throat
14th Dec 2:00 AM Breathing difficulties
• Mr Rawlins, Estate overseer, provided mixture molasses, vinegar
• Mr Rawlins ordered by Washington to remove 0.75 pint blood
10:00 AM Dr James Craik, personal physician, arrives
•
•
•
•
•
Preparation dried beetles applied to throat
1 pint blood removed
No improvement so another 1 pint blood removed
Vinegar in water gargle – near suffocation
2 more pints blood removed
3:00 PM Dr Elisha Dick, prominent physician arrives
• 1.8 pints blood removed
• Pulse low
10:10pm Died
Total > 6.5 pints removed
Pierre Charles
Alexandre Louis
(1787-1872)
• Introduced numerical methods to
examine the effectiveness of medical
interventions
• Showed blood letting ineffective
How Practitioners Make Decisions
Myth  Anecdote  Observation  Comparison  Replicated  RRC
Documented
Now
Academic
Practitioners
Future
Academic
Practitioners
Source: Sutherland 2007
Measuring Effectiveness
An Overture
Measuring Conservation Effectiveness Summit
May 5-6, Palo Alto, California
Research On Over 220 Measures
Systems in Different Fields
Approaches Reviewed
60
50
40
30
20
56
55
35
42
30
10
0
Business &
Mngmt
Education & International Public Health
Envt &
Soc Serv
Dvlpmt
& Pop
Conservation
Business & Management
Family Tree
ACCOUNTING &
CERTIFICATION
1500
1900
1970
Double-Entry
Bookkeeping
Auditing by
Accountants
Cost
Accounting
Accounting
Standards
Engineering
Standards
2000
EFFECTIVENESS
Stock Market
Index
Scientific
Mngmt
Economic
Production
Total Quality
Mngmt
Econ Welfare
Indicators
Managing for
Results
Outcome
Evaluation
Process
Performance
Reengineering Benchmarking
1980
1990
STATUS
ASSESSMENT
Process
Certification
Activity Based
Costing
Six Sigma
Balanced
Scorecard
Operations
Research
Social
Learning
Reflective
Practice
Learning
Benchmarking Organization
Chaordic
Systems
Outcome
Community
Evaluation
of Practice
Q1. What Is Measuring Effectiveness?
ACCOUNTING &
CERTIFICATION
1500
1900
1970
1980
1990
2000
STATUS
ASSESSMENT
EFFECTIVENESS
Status Question:
How are Species and Ecosystems Doing?
?
Effectiveness Question
Are Our Actions Leading to Desired Results?
?
Certification Question:
Do We Meet External Standards?
?
Why We Are Focused on Effectiveness
ACCOUNTING &
CERTIFICATION
STATUS
ASSESSMENT
EFFECTIVENESS
??
The
Focus for
This
Summit
Conservation Target
Wildlife and Habitat
Impacts
Direct
Threats
Indirect
Outcomes
Actions
Outputs
Costs of measuring change
Time to see an impact
Level of confidence
Q2. What is the Unit for Measuring Effectiveness?
Q3. How Do You Measure Effectiveness?
ACCOUNTING &
CERTIFICATION
STATUS
ASSESSMENT
EFFECTIVENESS
1500
1900
1970
1980
1990
2000
External
Summative
Evaluations
Participatory
Formative
Evaluations
Project Cycle
Based
Monitoring
Need to Integrate Measuring Effectiveness
into an Iterative Project Cycle
C
Develop a
Monitoring
Plan
D
B
Implement
Management &
Monitoring
Plans
Develop a
Management Plan:
Goals, Objectives,
& Activities
The AM
Project
Cycle
E
A
Analyze
Data and
Communicate
Results
Develop
Conceptual
Model Based
on Local Site
Conditions
Start
Iterate
Clarify Group's
Mission
Use Results to
Adapt & Learn
The Basic Systematic Performance
Management (SPM) Cycle
Plan
Adapt
Do
Check
Biodiversity Conservation
Projects Come In All Shapes and Sizes
1. A community fishing ground in the Pacific
2. A TNC preserve or provincial park
3. Coordinated efforts to manage state and federal
funding for Puget Sound or Lake Ontario
4. The WWF Markets Initiative
5. A funder’s grantmaking strategy for the California
Coastal waters, or the Coral Triangle
Similar Concepts, Different Terms
An Excerpt from the CMP’s Rosetta Stone Analysis
CMP
CI
TNC
Biodiversity Focal
Targets
Targets
Conservation
Outcomes
Focal
Conservation
Targets
Landscape
Species
LongTerm
Goals
Threats
Pressures
Threats
Threats
Threats
Milestones
Objectives
Targets
Project
Targets
Objectives
AWF
Threats
WCS
WWF
Conservation Measures Partnership’s
Open Standards Are One Flavor of SPM
• Developed by leading orgs &
agencies
• Draws on many fields
• Open source &
common language
• Used around the world
Q4. How Much Should We Invest in
Measuring Effectiveness?
SPM does not specify
M&E Design !!
The “Burden of Proof” Depends on Risks,
Costs, and Timeframe for Decision Making
Marisla
Foundation
Packard
Foundation
Generally Invest More in Measures When:
• Stakes are high (high cost of error or inaction)
• Potential to leverage learning
• Costs of measures are low relative to actions
Q5. Should We Mandate Measuring Effectiveness?
The Bell Curve: Treating Cystic Fibrosis
Distribution of Cystic Fibrosis
Treatment Center Success Rates
Poor
Below Above
Avg Avg
Best
A Surprise – The Best Get Better
It’s the centers in the top quartile that are improving fastest….they are
at risk of breaking away. What the best may have, above all, is a
capacity to learn and adapt – and to do so faster than everyone else.
The Bell Curve Leads to Uncomfortable Questions
Will being in the bottom half be used against doctors in lawsuits? Will
we be expected to tell our patients how we score? Will our patients
leave us? Will those at the bottom be paid less than those at the top?
The answer to all these questions is likely yes.
Ultimately We Are Not Competing Within
Conservation, But Against Other Social Causes
Worthy Cause A
Conservation
Worthy Cause B
Q7. How Do We Roll Out Measuring
Effectiveness Both Internally….
After a guy gave an ‘air ball’
Six Sigma presentation…the standard joke was
that the guy “decided to leave” before his
elevator reached the ground floor.
Q7. How Do We Roll Out Measuring
Effectiveness Both Internally….and Externally?
If We Create a Culture of Data Sharing…

?
Q8. How Do We Continue this Conversation
Going Forward?
Our Fundamental Hypothesis
Resources for
Cons Projects
100 %
0%
+ Success
-
+
- Failure
Our Fundamental Hypothesis
Resources for
Cons Projects
100 %
0%
+ Success
-
-
- Failure
+
+
Investment in
SPM
Measuring Effectiveness Can Transform
the Practice of Conservation
• Practitioners in field using measures results to plan
and implement effective projects
• Managers and donors getting better information to
improve work
• All stakeholders learning from each other
• Increased public support for conservation
• Ultimately, better conservation outcomes