A stochastic dominance approach
to program evaluation
And an application to child nutritional
status in arid and semi-arid Kenya
Felix Naschold Cornell University & University of Wyoming
Christopher B. Barrett Cornell University
AAEA 27 July 2010
Motivation
1.
Program Evaluation Methods
 By design they focus on mean. Ex: “average treatment effect”
 In practice often interested in distributional impact
 Limited possibility for doing this by splitting sample
2.
Stochastic dominance
 By design look at entire distribution
 Now commonly used in snapshot welfare comparisons
 But not for program evaluation. Ex: “differences-in-differences”
3.
2
This paper merges the two
 Diff-in-Diff (DD) evaluation using stochastic dominance
(SD)
Main Contributions of this paper
1. Proposes DD-based SD method for program evaluation
2. First application to evaluating welfare changes over time
3. Specific application to new dataset on changes in child
nutrition in arid and semi-arid lands (ASAL) of Kenya
3

Unique, large dataset of 600,000+ observations collected by the
Arid Lands Resource Management Project (ALRMP II)

(one of) first to use Z-scores of Mid-upper arm circumference
(MUAC)
Main Results
1.
Methodology
 (relatively) straight-forward extension of SD to dynamic
context: static SD results carry over
 Interpretation differs (as based on cdfs)
 Only up to second order SD
2.
Empirical results
 Child malnutrition in Kenyan ASALs remains dire
 No average treatment effect of ALRMP expenditures
 Differential impact with fewer negative changes in treatment
sublocations
 ALRMP a nutritional safety net?
4
Program evaluation (PE) methods
 Fundamental problem of PE: want to but cannot observe a
person’s outcomes in treatment and control state
i  xiT  xiC
 Solution 1: make treatment and control look the same
(randomization)
 Gives average treatment effect
E    E  xT   E  xC 
 Solution 2: compare changes across treatment and control
(Difference-in-Difference)
 Gives average treatment effect:
5
E     E  xT ,t  xT ,t 1   E  xC ,t  xC ,t 1 
New PE method based on SD
 Objective: to look beyond the ‘average treatment effect’
 Approach: SD compares entire distributions not just their
summary statistics
 Two advantages
Circumvents (highly controversial) cut-off point.
Examples: poverty line, MUAC Z-score cut-off
2. Unifies analysis for broad classes of welfare indicators
1.
6
Definition of Stochastic Dominance
 First order: A FOD B up to z  xmin , xmax  iff
FB  x   FA  x   0  x   xmin , z 
Cumulative %
of population
FB(x)
FA(x)
xmax
 Sth order: A sth order dominates B iff
0
MUAC
score
Z-
FBs  x   FAs  x   0  x  xmin , z 
7
SD and single differences
 These SD dominance criteria
 Apply directly to single difference evaluation (across time OR
across treatment and control groups)
 Do not directly apply to DD
 Literature to date:
 Single paper: Verme (2010) on single differences
 SD entirely absent from PE literature (e.g. Handbook of
Development Economics)
8
Expanding SD to DD estimation Method
 Practical importance: evaluate beyond-mean effect in non



9
experimental data
Let   xt  xt 1 , G denote the set of probability density
functions of Δ. g A () G and g B () G
The respective cdfs of changes are GA(Δ) and GB(Δ)
Then A FOD B iff GB     GA     0  min , max 
A Sth order dominates B iff GBs     GAs     0  min , max 
Expanding SD to DD estimation –
2 differences in interpretation
1. Cut-off point in terms of changes not levels.
 Cdf orders changes from most negative to most positive 
‘poverty blind’ or ‘malnutrition blind’.
 (Partial) remedy: run on subset of ever-poor/always-poor
2. Interpretation of dominance orders
 FOD: differences in distributions of changes between
intervention and control sublocations
 SOD: degree of concentration of these changes at lower end
of distributions
 TOD: additional weight to lower end of distribution. Sense in
doing this for welfare changes irrespective of absolute welfare?
10
Setting and data
 Arid and Semi-arid district in Kenya
 Characterized by pastoralism
 Highest poverty incidences in Kenya, high infant mortality and
malnutrition levels above emergency thresholds
 Data
 From Arid Lands Resource Management Project Phase II
 28 districts, 128 sublocations, June 05- Aug 09, 600,000 obs.
 Welfare Indicator: MUAC Z-scores
 Severe amount of malnutrition:
 10 percent of children have Z-scores below -1.54 and -2.55
 25 percent of children have Z-scores below -1.15 and -2.06
11
The pseudo panel used
 Sublocation-specific pseudo panel 2005/06-2008/09
 Why pseudo-panel?
Inconsistent child identifiers
2. MUAC data not available for all children in all months
3. Graduation out of and birth into the sample
1.
How?





12
14 summary statistics – mean & percentiles and ‘poverty
measures’
Focus on malnourished children
Thus, present analysis median MUAC Z-score of children below 0
Control and intervention according to project investment
Results: DD Regression
 Pseudo panel regression model
 No statistically significant average program impact
13
Results – DD regression panel
VARIABLES
intervention dummy based on
ALRMP investment
change in NDVI 2005/06-08/09
squared change in NDVI
2005/06-08/09
Constant
Observations
R-squared
14
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
(1)
(2)
(3)
(4)
(5)
median of
MUAC Z <0
10th
percentile
25th
percentile
median of
MUAC Z <-1
median of
MUAC Z <-2
0.0735
0.0832
0.0661
0.0793
0.0531
(0.248)
(0.316)
(0.371)
(0.188)
(0.155)
1.308*
2.611***
2.058***
0.927*
0.768*
(0.0545)
(0.00294)
(0.00754)
(0.0997)
(0.0767)
-12.91**
-8.672
-12.70*
-0.954
1.924
(0.0293)
(0.136)
(0.0510)
(0.802)
(0.479)
0.501***
0.892***
0.839***
0.203***
0.120***
(2.99e-07)
(1.40e-08)
(8.70e-09)
(0.000133)
(0.00114)
114
114
114
114
106
0.319
0.299
0.297
0.249
0.280
Stochastic Dominance Results
Three steps:
 Steps 1 & 2: Simple differences
 SD within control and treatment over time:
no difference in trends. Both improved slightly
 SD control vs. treatment at beginning and at end:
control sublocations dominate in most cases, intervention never
 Step 3: SD on DD (results focus for today)
15
FOSD Difference Intervention vs. Difference Control
0
.2
.4
.6
% of sublocations
.8
1
Median MUAC of obs<0. Categorization by Investment
-1
-.4
.2
.8
1.4
2
difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09
Control
intervention
FOSD Difference Intervention vs. Difference Control
0
-.1
-.2
% of sublocations
.1
.2
Median MUAC of obs<0. Categorization by Investment
-1
-.4
.2
.8
1.4
2
difference in median MUAC Z-score with MUAC<0. drought adjusted. 2005/06-2008/09
16
Confidence interval (95 %)
Estimated difference
25th percentile MUAC. Categorization by Investment
0
.2
.4
.6
.8
1
FOSD Difference Intervention vs. Difference Control
-1.5
-.8
-.1
.6
1.3
2
difference in 25th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
Control
17
intervention
10th percentile MUAC. Categorization by Investment
0
.2
.4
.6
.8
1
FOSD Difference Intervention vs. Difference Control
-1.5
-.8
-.1
.6
1.3
2
difference in 10th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
Control
18
intervention
Conclusions
 Existing program evaluation approaches  average
treatment effect
 This paper: new SD-based method to evaluate impact across
entire distribution for non-experimental data
 Results show practical importance of looking beyond
averages
 Standard DD regressions: no impact at the mean
 SD DD: intervention sublocations had fewer negative
observations
 ALRMP II may have functioned as nutritional safety net (though
only correlation, no way to get at causality)
19
Thank you.
20
Expanding SD to DD estimation –
controlling for covariates
 In regression DD: simply add (linear) controls
 In SD-DD need a two step method
Regress outcome variable on covariates
2. Use residuals (the unexplained variation) in SD DD
1.

21
In application below first stage controls for drought
(NDVI)
SD, poverty & social welfare
orderings (1)
1. SD and Poverty orderings
 Let SDs denote stochastic dominance of order s and Pα stand
for poverty ordering (‘has less poverty’)
 Let α=s-1
 Then A Pα B iff A SDs B
 SD and Poverty orderings are nested
 A SD1 B  A SD2 B  A SD3B
 A P1 B  A P 2 B  A P3 B
22
SD, poverty & social welfare
orderings (2)
2. Poverty and Welfare orderings (Foster and Shorrocks 1988)
 Let U(F) be the class of symmetric utilitarian welfare
functions
 Then A Pα B iff A Uα B
 Examples:
 U1 represents the monotonic utilitarian welfare functions such
that u’>0. Less malnutrition is better, regardless for whom.
 U2 represents equality preference welfare functions such that
u’’<0. A mean preserving progressive transfer increases U2.
 U3 represents transfer sensitive social welfare functions such
that u’’’>0. A transfer is valued more lower in the distribution
 Bottomline: For welfare levels tests up to third order make
23
sense
The data (2) – extent of malnutrition
24
Table 3 10th percentile MUAC Z-score – whole sample
Year
Garissa Kajiado Laikipia Mandera Marsabit
Mwingi
Narok
Nyeri
Tharaka
Turkana
2005/06
-2.4
-2.14
-1.75
-2.65
-2.33
-2.36
-2.55
-1.67
-1.87
-2.26
2008/09
-1.88
-2.22
-2.1
-2.13
-2.29
-2.14
-2.35
-1.54
-1.74
-2.25
Table 4 25th percentile MUAC Z-score – whole sample
year
Garissa Kajiado Laikipia Mandera Marsabit
Mwingi
Narok
Nyeri
Tharaka
Turkana
2005/06
-1.97
-1.67
-1.16
-2.06
-1.79
-1.84
-1.96
-1.2
-1.45
-1.85
2008/09
-1.45
-1.76
-1.4
-1.69
-1.69
-1.68
-1.76
-1.15
-1.28
-1.86
DD Regression 2
 Individual MUAC Z-score regression
 To test program impact with much larger data set
 Still no statistically significant average program impact
25
Results – DD regression indiv data
Dependent variable: Individual MUAC Z-score
VARIABLES
time dummy (=1 for 2008/09)
0.0785
(0.290)
control - intervention by investment
-0.0576
(0.425)
Diff in diff
0.0245
(0.782)
Normalized Difference Vegetation Index
1.029***
(6.25e-07)
Constant
-1.391***
(0)
Observations
R-squared
26
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
271061
0.033
Full table of SD results
Sublocation panel
Median MUAC of obs < 0
% below -1 SD
Dominance
I.1 Intervention 05/06-08/09
FOSD
SOSD
TOSD
I.2 Control 05/06-08/09
FOSD
SOSD
TOSD
II.1 Intervention vs. Control 05/06
FOSD
SOSD
TOSD
II.2 Intervention vs. Control 08/09
FOSD
SOSD
TOSD
III. Diff Intervention vs Diff.
Control
FOSD
SOSD
Which*
Signif.
Y
Y
Y
08/09
08/09
08/09
Y
Y
Y
Individual data
MUAC Z-Score
Dominance
Which*
*
Signif.
NS
S
S
Almost
Y
Y
08/09
08/09
08/09
NS
NS
NS
08/09
08/09
08/09
NS
NS
NS
Y
Y
Y
08/09
08/09
08/09
Y (almost)
Y
Y
Control
Control
Control
NS
NS
NS
Almost
Y
Y
#
N
Unclear
Unclear
-
NS
NS
NS
N
Y?
-
NS
NS
Dominance
Which*
Signif.
Y
Y
Y
08/09
08/09
08/09
S
S
S
NS
NS
NS
Y
Y
Y
08/09
08/09
08/09
S
S
S
Control
Control
NS
NS
NS
Y
Y
Y
Control
Control
Control
S
S
S
N
Y
Y
Control
Control
Control
NS
NS
NS
Y
Y
Y
Control
Control
Control
S
S
s
N
Y
Interve
ntion
NS
NS
* Lower curves to the right are dominate for these indicators for which a greater number indicates ‘better’.
** For parts I. and II. higher curves to the left dominate for the proportion of observations below -1SD, as lower
proportions are ‘better’. In contrast, for changes from 2005/06-2008/09 in part III. larger positive changes are better,
so lower curves to the right dominate.
# Control sites dominate up to MUAC Z-score of -0.1. Intervention sites dominate for MUAC Z-score > 0.
27
FOSD Difference Intervention vs. Difference Control
0
.2
.4
.6
.8
1
Median MUAC of obs<0. Categorization by Investment
-1
-.4
.2
.8
1.4
2
difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/
Control
28
intervention