Measuring Impact of Certification:
Towards a Focus for Future Research
ISEAL Conference, June 8, 2011
©2009 Rainforest Alliance
Demonstrating Impact: Now and Tomorrow
MYTH  ANECDOTE  OBSERVATION  COMPARISON  REPLICATED  RRC
Documented
Now
Academic
Practitioners
Academic
Future
Standard
Indicators
Research
Strategy
Practitioners
BUSINESS MODEL OF CERTIFICATION
Consumer behavior (values)
Retailers
Brands
Manufacturers
Traders
Processors
Sustainability
feedback:
Demand for
sustainable
markets
Business
Practices
in Value
Chain
Production Unit (Best
Management Practices)
Measureable Ecological, Social,
Economic benefits
Sustainability
feedback:
Evidence of
Impact
WHAT IS THE HYPOTHESIS TO TEST?
All study designs must first start with a question or set of questions to be
answered in the evaluation. For example…..
• What are the monetary costs and benefits of implementing the SAN standard (including
price premiums), and how do they change over the lifetime of a certificate?
• How do farm productivity, crop quality and soil health compare between farms that do
and do not implement best management practices we assume change these conditions?
What are the independent variables (Controlled and varied by study) and
dependent variables (determined by independent variable)?
4
HOW TO DEVELOP EVALUATION AND RESEARCH MODEL
What is the
meaningful
Independent
Variable?
Consumer behavior (values)
Retailers
Brands
Manufacturers
Traders
Processors
Sustainability
feedback:
Demand for
sustainable
markets
Business
Practices
in Value
Chain
What is the
meaningful
Independent
Variable?
Production Unit (Best
Management Practices)
Measureable Ecological, Social,
Economic benefits
Sustainability
feedback:
Evidence of
Impact
EVALUATION OF INTERVENTIONS NOT BUSINESS MODELS
Unbundling certification to measure adoption of best management practices (BMPs).
Operation 1
BMP1
BMP2
BMP3
BMP4
Operation 2
BMP1
BMP2
BMP3
BMP4
Operation 3
BMP1
Operation 4
No BMPs
BMP4
Gains us a more nuanced understanding of what production and social and environmental
management practices drive what economic, social or ecological returns, and under what
conditions.
Informs learning and refinement of standards.
6
Level of confidence
Dependent variables.
Determined by control
and variation of
independent variable
Reduced encroachment
Independent variable.
Controlled and
varied by study
Indirect
Adoption of RA best
management practices
Productivity
Action
Costs of measuring change
Desired
Impact?
Time to see an impact
DETERMINING HYPOTHESIS BASED ON COST, LENGTH OF STUDY AND
LEVEL OF CREDIBILITY NECESSARY
Hypothesis TESTING: Research Strategy
Research that tests
hypothesis by reaching
conclusions about
relationship between
dependent and
independent variables
is
Before and after
implementation of
BMPs/Certification
attempt to discover
new information
Control group
Independent
variable
Exposed to all elements
of the research, except
the intervention
(experimental) variable
are
Determined by
independent
variable
are
is
is
Control
and varied
Dependent
variable
Intervention
(experimental)
group
demonstrate that
what we say we do
is true
examine the validity
of a project’s
hypothesis
Exposed to the
intervention
(experimental)
variable
8
Hypothesis TESTING: Research Strategy
Research that tests
hypothesis by reaching
conclusions about
relationship between
BMPs and farm income
variables
BMP(s)
Farms with no
BMPs
Exposed to elements such
as international prices for
coffee, advantageous
weather conditions or
improvements in processing
and marketing,.
Farms with
BMPs
are
Determined by
control and
variation of
BMPS
attempt to discover
new information
are
is
is
Controlled
and varied
by RA
Real Farm
Income
is
Before and after
implementation of
BMPs/Certification
demonstrate that
what we say we do
is true
examine the validity
of a project’s
hypothesis
Exposed to all
elements +
BMPs
9
CONTRIBUTION VS. ATTRIBUTION
DEMONSTRATING CONTRIBUTION OF RA PRACTICES
• Measure before and after implementation of BMP to see changes in social, economic
and environmental conditions.
• The implicit assumption is that had farms not applied the BMP, their outcomes, on
average, would have stayed the same.
• Results will show impact of BMP but will not isolate the influence of these
practices from the impact of factors unrelated to BMP implementation, e.g.
international prices also influences income, advantageous weather conditions also
influences productivity, a government intervention also improves working conditions.
• BMP implementation must be presented as just ONE of a possible number of factors
influencing desired outcomes.
10
CONTRIBUTION VS. ATTRIBUTION
DEMONSTRATING ATTRIBUTION OF RA PRACTICES
• Measure before and after implementation to see changes in social, economic and
environmental conditions.
• At the same time also measure changes in social, economic and environmental on farms
that DO NOT implement BMP as the counterfactual outcomes (comparison group).
• The implicit assumption is that had farms not implemented BMP , their outcomes would
be the same, on average, as those of farms in the comparison group.
• With a comparison group, the impact of BMP implementation can be defined as the
difference between the observed intervention outcome and the
counterfactual outcome.
• BMPs can be attributed to desired outcomes because counterfactual outcome accounts
for changes as a result of influencing factors, except BMP implementation
11
CONSTRUCTING A CREDIBLE COUNTERFACTUAL
• A credible counterfactual must address selection bias, a problem
that will violate assumptions when farms select themselves – or are
selected by NGOs/traders – into certification.
• 2 approaches to eliminate selection bias
– Experimental
– Quasi-experimental.
• Each varies in feasibility, cost, the degree of clarity and scientific
validity of results.
12
EXPERIMENTAL DESIGN (OPTIMAL APPROACH)
• Involves gathering a set of farmers equally eligible and willing to participate in
certification and randomly dividing them into two groups: those who receive
the technical assistance (treatment group) and those from whom the
intervention is withheld (control / counterfactual group).
Advantages
Challenges
Random assignment of farms serve as a perfect
counterfactual, free from selection bias (assuming
sufficient sample size).
Perhaps unethical and political owing to the denial
of certification.
Simplicity in interpreting results as true impact —
difference between the means of both groups.
Can be expensive and time consuming, particularly
in the collection of longitudinal data points.
Farms in treatment or control groups may change
certain identifying characteristics during the
experiment that could invalidate or contaminate
results.
13
QUASI-EXPERIMENTAL DESIGN
• Consists of constructing a comparison group using matching comparisons. This involves
identifying non-certified farms comparable in essential characteristics to certified farms. Both
groups should be matched on the basis of very similar observable characteristics that plausibly
affect outcomes
• Ideally matched comparison groups should be selected before project implementation, not
afterwards.
Advantages
Can draw on existing data sources and are thus
often quicker and cheaper to implement
Disadvantages
Reliability of the results is often reduced, as the
methodology may not completely solve the
problem of selection bias
Matching methods can be statistically complex, thus
requiring considerable expertise in the design of
the evaluation and in analysis and interpretation of
the results.
14
META ANALYSIS OF LITERATURE
• Looks at set of specific BMPs across standard systems (Ag, Forestry,
Fisheries) as list of independent variables
• Categorize by dependent variables examined
• Method
• Statistical analysis applied
• Result - Statistical significance
• Geographic scope / focus
• Time frame adequate - based on dependent variable (Oak tree versus
Sunflower)
15
The Rainforest Alliance works to conserve biodiversity and ensure sustainable livelihoods
by transforming land-use practices, business practices and consumer behavior.