Case study:
Time-limited vouchers to encourage
fertilizer adoption: Are they effective? 1
Introduction
In 2010, the Ministry of Agriculture in Naguda decided to work with an NGO called “Fertilizer
for All” to pilot a new approach to increase adoption of fertilizer. Fertilizer adoption is limited in
Naguda and the Ministry wants to increase fertilizer use to increase yields. In the month after
harvest, the NGO staff visited 1,000 farmers to offer them time-limited vouchers to adopt
fertilizer in their farming plots, and convey the message below:
“Hello. My name is Marc Oyeye, and I’m from Fertilizer for All. We are an NGO that
promotes the use of fertilizers in farming plots. We’d like to let you know that with this
voucher, you could get fertilizer at a 50% discount. Please also note that it will expire in
two weeks.”
Did the time-limited voucher increase the adoption of fertilizer? How can we find out? This case
study addresses these questions by examining the various methods that can be used to assess the
impact of a program or intervention. While the context of this case study is fertilizer use in
Naguda, the questions raised here are also valid for the assessment of the impact of other public
programs in developing countries.
Background
By some estimates, there are approximately 1.4 billion people living on less than $1.25 a day,
many of whom are farmers. As such, identifying ways to increase agricultural incomes is crucial
in meaningfully alleviating poverty. Such strategies are especially important in Naguda, where
agricultural yields have been low and remained stagnant for many years. The use of fertilizer has
the potential to dramatically increase yields and, if used correctly, is a highly profitable
investment. However, adoption rates remain are low. If fertilizer is so profitable, why do so few
farmers in Naguda use it? Is it lack of information about profitability, lack of money to purchase
it, or an inability to save for the purchase?
In search of an explanation for this low fertilizer adoption rate, and especially in search of a
solution, the Minister of Agriculture hired a consultant, who proposed the following strategy:
Based on Duflo et al. “Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya.”
http://www.povertyactionlab.org/publication/nudging-farmers-use-fertilizer-theory-and-experimental-evidencekenya
1
To: His Excellency the Minister of Agriculture
From: S.Holma, consultant
Date: December 15, 2010
Ref: Fertilizer adoption survey update
Diagnostic
I have implemented a survey of 500 randomly selected farmers in Naguda to find out
why the fertilizer adoption rate is so low. After careful analysis, I conclude that the
following is the most important reason why farmers do not adopt fertilizer:
They have difficulty saving harvest income until the time when inputs are needed.
Proposal
Since the issue is that farmers have difficulty saving, the Ministry of Agriculture should
hire a company to visit the farmers and give them time-limited vouchers to incentivize
them to purchase fertilizer right after the harvest season, when they have cash available.
Since most farmers in Naguda have neither telephones nor cell-phones, visiting farmers
door-to-door is the only viable option.
The Ministry of Agriculture is skeptical about this proposal. Before rolling out the strategy in the
whole country, they decide to hire an NGO, “Fertilizer for All” to run a pilot experiment to test
the efficacy of time-limited fertilizer vouchers. An impact evaluation will be built into the pilot
experiment. The Minister of Agriculture hires you to run the impact evaluation.
Question 1 for discussion – What is the one basic question that your
impact evaluation should be able to answer?
Please complete Question 1 before reading further.
******************************
Did the time-limited fertilizer voucher work?
In December 2010, the NGO “Fertilizer for All” obtained a list of 1,000 farmers in Naguda. The
list of 1,000 farmers was obtained from the archives of the national civil registry of Naguda, in
which all farmers are registered. The archives also contain data on the size of the farmer’s
household, the age of the farmer, where the farmer lives (i.e., Northern Region or Southern
Region), and the level of economic development of the farmer’s district.
In January 2011, interviewers visited all 1,000 farmers, but were only able to speak with 403
people. That is to say, only 403 farmers were at home and offered the voucher. For each of the
1,000 farmers, the volunteers noted whether the farmers are at home for visit or not.
Finally, after 6 months, Fertilizer for All revisited the farmers and determined whether they had
actually used the vouchers in the new farming season or not.
Fertilizer for All has agreed to share with you its data concerning the 1,000 farmers involved in
their fertilizer voucher program. We are asking you to use this data to gauge the impact of the
fertilizer voucher campaign on fertilizer adoption rates, i.e. its impact on the percentage of
farming plots that are fertilized.
Method 1 – Difference in the proportion of fertilizer usage, between farmers that received
fertilizer voucher versus those that did not receive it.
Assume that the 403 farmers that received the fertilizer voucher constitute the ‘treatment’ group
and the remaining 597 farmers (i.e. those that were visited but were not in their home) represent
the ‘comparison’ group. If you want to determine the impact of receiving a voucher on fertilizer
adoption rate, you might check to see whether those who received the voucher were more likely
to fertilize their farming plots than those who did not. Table 1a below compares the proportion of
farmers in the ‘treatment’ group that had their farming plots fertilized with the proportion of such
farmers in the ‘comparison’ group.
Table 1a: Percentage of farmers that had adopted fertilizer in 2011
Method 1:
Simple difference
… among farmers
that got the voucher
… among farmers
Estimated
that did not get the voucher impact
16.5 %
9.6 %
6.9 pp*
Question 2 for discussion – Do you think this method can give you a precise idea of the
actual impact of the voucher on the farmers’ fertilizer adoption rate? Why or why not?
Please complete Question 2 before reading further.
******************************
Method 2 – Use a multiple regression model to determine the differences
between farmers that received the fertilizer voucher and those that did not.
If you believe that the farmers that received the fertilizer voucher may have inherent
characteristics different from those who did not receive them, you can test the difference by
using a multivariate regression, as follows:
The participant group and the comparison group are defined in the same way as in
Method 1. To estimate the impact of the program, one does a regression in which the
‘dependent variable’ indicates whether the farmer used fertilizer or not (i.e., 0 = did not fertilize
plot, 1 = fertilized plots). The ‘key explanatory variable’ is a variable indicating whether the
farmer was offered the voucher or not (i.e., 0 = offered, 1 = was not offered). Potential
differences in other characteristics of the farmers can be disentangled by adding other
‘explanatory variables’ such as the size of farmer’s household, the age of farmer, size of the
property etc. The coefficient of the ‘key explanatory variable’ is the estimated program impact.
Table 1b shows the estimated impact of the fertilizer voucher using the multivariate method
(Method 2) compared to the estimated impact of the simple difference (Method 1)
Table 1b: Percentage of farmers that had adopted fertilizer in 2011
… among farmers
offered voucher
Method 1: Simple Difference 16.5 %
Method 2: Multiple regression
… among farmers Estimated
not offered voucher impact
9.6 %
6.9 pp*
5.1 pp*
pp=percentage points
*: statistical significance = 5 %
Controls include size of farmer’s household, age of the farmer, a variable indicating the level of economic
development in the farmer’s district and a variable indicating whether the farmer lives in the Northern Region.
Table 2 compares the average characteristics of the ‘treated’ groups and ‘comparison’ groups
used in these two methods.
Table 2: Average characteristics of farmers
Farmers
that got
voucher
Size of farmer’s property (Ha)
Average age of farmer
Percentage of farmers that have mobile phone
Percentage in the Northern Region
Sample size
Farmers that
did not get
voucher
Difference
2.8
1.0
1.8*
35.8
31.0
4.8
32.7 %
32.2%
0.5 pp
54.7 %
46.7 %
8.0 pp*
403
597
Question 3 for discussion – Why do you think that the impact estimated using Method 2 is
smaller than the impact estimated using Method 1?
Question 4 for discussion – Do you think the impact estimated with Method 2 represents
the true causal effect of the voucher on farmers’ fertilizer adoption? Why or why not?
Question 5 for discussion – Can you correct the weaknesses of Method 1 by taking a random
sample of farmers got the voucher and a random sample of farmers who did not get the voucher?
Question 6 for discussion – Using the data described above, can you come up with some more
convincing methods for estimating the impact of the voucher? What kind of information would be
helpful?
Please complete questions 3 through 6 before reading further.
******************************
Method 3 – Using Panel Data
If you are still concerned about differences in characteristics between farmers that were offered
the vouchers and those that were not, you might use panel data, i.e. you could follow the
same farmers over time.
As it turns out, the archives of the NGO also had data indicating whether farmers had adopted
fertilizer in the past two years, from 2009 to 2010. The farmers’ past behavior with regard to
fertilizer adoption can be a solid predictor of their future fertilizer adoption behavior. Table 3
shows the past fertilizer adoption behavior for the group of farmers that were offered the
vouchers versus farmers that were not at home during Fertilizer for All’s visit.
Table 3: Percentage of farmers who adopted fertilizer prior to 2011*
… among
farmers
that got
voucher
… among farmers
that did not get
voucher
Difference
Used fertilizer in 2011
16.5 %
9.6 %
6.9 pp
Used fertilizer in 2010
14.4 %
8.6 %
5.8 pp
Used fertilizer in 2009
12.7 %
7.3 %
5.4 pp
Difference between fertilizer
usage in 2011, those in 2010
2.1 %
1.0 %
1.1 pp*
pp=percentage points *:
statistical significance = 5 %
Question 7 for discussion – How could you use this data on behavior regarding fertilizer
adoption in previous years to improve your analysis? What kind of method could you
use? Based on the information in Table 3, what would be your new estimate of the
impact of the fertilizer voucher on fertilizer adoption rates?
Question 8 for discussion – Compare your new estimate to the estimates you obtained with
Method 1 and Method 2. Is the estimated impact lower or higher? Why do you think this is?
Please complete questions 7 through 8 before reading further.
******************************
Randomized Experiment
As it turns out, the 1,000 farmers were chosen at random from the archives of the national civil
registry of Naguda. This is similar to the random drawing done in a clinical trial, where the
treatment/drug is administered randomly so as to be received by one group of patients but not the
other. The complete list includes 7,000 farmers. We can exploit this random drawing of 1,000
farmers to estimate the impact of the fertilizer voucher. The idea is that the 1,000 farmers that
were targeted for receiving vouchers from Fertilizer for All (now referred to as the ‘treatment’
group) should be identical to the 6,000 other Nagudian farmers (now referred to as the ‘control’
group) in terms of observable and non-observable characteristics. The only difference between
the treatment and control groups is that the first group was visited by Fertilizer for All and the
second was not. Table 4 compares the ‘treatment’ group and the ‘control’ group on the basis of
observable characteristics. Table 5 shows the estimated impact of the fertilizer voucher by
comparing fertilizer use in the treatment group with fertilizer use in the control group.
Table 4: Characteristics of treatment and control groups
‘Treatment’ group ‘Control’ group
(Visited)
(Not visited)
Difference
9.5%
10.9%
9.2%
11.3%
0.3 pp
-0.4 pp
4.7
33.1
32.4%
49.9%
1,000
5.1
32.0
31.2%
51.4%
6,000
-0.4
1.1
1.2 pp
-1.5 pp
Adopted fertilizer in 2009
Adopted fertilizer in 2010
Size of farmer’s property (Ha)
Average age of farmer
% of farmers with mobile phone
Percentage in the Northern Region
Size of sample
pp=percentage points
*: statistical significance = 5 %
Table 5: Randomized treatment and control groups
Percentage of farmers that adopted fertilizer in 2011
‘Treatment’ group ‘Control’ group Impact estimate
Method 4a: Random Simple difference
12.4 %
12.2 %
0.2 pp
Method 4b: Random Multiple Regression
0.2 pp
pp=percentage points
*: statistical significance = 5 %
Question 9 for discussion – Notice that the two groups seem very similar in Table 4. Is this what you
were expecting? Why or why not?
Please complete questions 9 and 10 before reading further.
******************************
Technical note: Adjustment for take-up rate
Table 5 shows the simple comparison of treatment and control groups, where the treatment
group consists of all those were visited by Fertilizer for All and the control group consists
of all those who were not visited. This estimated impact does not take into account the
fact that 597 individuals in the “treatment” group were visited but not at home, therefore
were not given vouchers.
If we wish to estimate the impact of actually handing the voucher to the farmers, rather than
just “visiting” the farmers, then we would need to adjust the estimate using the methodology
of Instrumental Variables.
A possible formula for making the adjustment is as follows：
𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 𝑓𝑟𝑜𝑚 𝑚𝑒𝑡ℎ𝑜𝑑 4𝑎
𝐷𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑖𝑛 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑟𝑒𝑐𝑒𝑖𝑣𝑖𝑛𝑔 𝑡ℎ𝑒 𝑣𝑜𝑢𝑐ℎ𝑒𝑟
𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑎𝑛𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑔𝑟𝑜𝑢𝑝𝑠
So:
0.2
400
−0
600
= 0.3
Conclusion
Table 6 shows the estimated impacts of the fertilizer voucher on fertilizer adoption rates using
the various methods discussed in this case study.
Table 6 – Summary of estimated impacts of the fertilizer voucher
Method
Method 1: Simple difference
Method 2: Multiple regression
Method 3: ‘Double difference’ based on panel data
Method 4a and 4b: Randomized experiment
Method 4c: Randomized experiment adjusting for take-up rate
Estimated impact
6.9 pp*
5.1 pp*
1.1 pp*
0.2 pp
0.3 pp
pp=percentage points
*: statistical significance = 5 %
As you can see, not all methods yield the same results. It is therefore critical to choose the
appropriate method. The purpose of this case study was not to assess a specific fertilizer voucher
program, but to test various assessment methods in this particular context.
In the analysis of the fertilizer voucher program, we noticed that those who received voucher
were probably going to adopt fertilizer in their farming plots, but that they were also more likely
to have fertilized their farming plots in previous years. Even when we accounted statistically for
(known!) observable characteristics of farmers, including demographic characteristics and
fertilizer use in previous years, there were still some inherent non-observable differences
between the groups, independent of the fertilizer voucher program. Thus, when our non-random
methods demonstrated a positive and significant impact, this result was attributable to a
‘selection bias’ (in this case, the selection of those who were at home during the visit and
received the voucher) rather than to a successful fertilizer voucher program.
Application to development
Selection bias is a problem that occurs in many program evaluations. Think about some of
the non-random development programs that you have evaluated or seen evaluated. Discuss
how the participant group was chosen and how the ‘selection’ may have affected the
evaluators’ capacity to gauge the true impact of the program.