And an application to child nutritional status in arid and semi-arid Kenya
Felix Naschold University of Wyoming
Christopher B. Barrett Cornell University
May 2012 seminar presentation
University of Sydney

2
1.
2.
3.






Program Evaluation Methods
By design they focus on mean
Ex: “average treatment effect” (ATE)
In practice, often interested in broader distributional impact
Limited possibility for doing this by splitting sample
Stochastic dominance
By design, look at entire distribution
Now commonly used in snapshot welfare comparisons
But not for program evaluation. Ex: “differences-in-differences”
This paper merges the two
 Diff-in-Diff (DD) evaluation using stochastic dominance
(SD) to compare changes in distributions over time between intervention and control populations

3
1.
Proposes DD-based SD method for program evaluation
2.
First application to evaluating welfare changes over time
3.


Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya
Unique, large dataset of 600,000+ observations collected by the
Arid Lands Resource Management Project (ALRMP II) in Kenya
(One of) first to use Z-scores of Mid-upper arm circumference
(MUAC)

4
1.
2.



Methodology
(relatively) straight-forward extension of SD to dynamic context: static SD results carry over
Interpretation differs (as based on cdfs)
Only feasible up to second order SD

Empirical results
Child malnutrition in Kenyan ASALs remains dire
 No average treatment effect of ALRMP expenditures


Differential impact with fewer negative changes in treatment sublocations
ALRMP a nutritional safety net?

5
 Fundamental problem of PE: want to but cannot observe a person’s outcomes in treatment and control state
  i x iT
 x iC


Solution 1: make treatment and control look the same
(randomization)
 Gives average treatment effect as E
     
T

E x
C
Solution 2: compare changes across treatment and control
(Difference-in-Difference)

E
Gives average treatment effect as:
 

 x
,

1



 x
,

1

6



Objective: to look beyond the ‘average treatment effect’
Approach: SD compares entire distributions not just their summary statistics
Two advantages
1.
2.
Circumvents (highly controversial) cut-off point
Examples: poverty line, MUAC Z-score cut-off
Unifies analysis for broad classes of welfare indicators

 First order: A FOD B up to z
  x
F
, x min max
B
  
F

A iff
 
Cumulative % of population x
 x min
, z

F
B
(x)
F
A
(x)
7
 x max
S th order: A s th order dominates B iff
0
F
B s
  
F
A s
 
ZMUAC score x
 x min
, z


8
 These SD dominance criteria


Apply directly to single difference evaluation (across time OR across treatment and control groups)
Do not directly apply to DD
 Literature to date:
 Single paper: Verme (2010) on single differences
 SD entirely absent from the program evaluation literature (e.g.,
Handbook of Development Economics)

9





Practical importance: evaluate beyond-mean effect in nonexperimental data
Let x t x t

1
, and G denote the set of probability density functions of Δ , with 𝑔
𝐴
∆ , 𝑔
𝐵
∆ ∈ 𝑮
The respective cdfs of changes are G
Then A FOD B iff G
B
 
G
A
 
0
A
( Δ ) and G

B
(
,
Δ

) min max
A S th order dominates B iff G
B s
 
G s
A
 
0


,
 min max


10
1. Cut-off point in terms of changes not levels.
 Cdf orders change from most negative to most positive  ‘initial poverty blind’ or ‘initial malnutrition blind’.
 (Partial) remedy: run on subset of ever-poor/always-poor
2. Interpretation of dominance orders
 FOD: differences in distributions of changes between intervention and control sublocations


SOD: degree of concentration of these changes at lower end of distributions
TOD: additional weight to lower end of distribution. Is there any value to doing this for welfare changes irrespective of absolute welfare?
Probably not.

11


Arid and Semi-arid districts in Kenya


Characterized by pastoralism
Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds
Data




From Arid Lands Resource Management Project (ALRMP) Phase II
28 districts, 128 sublocations, June 05- Aug 09, 602,000 child obs.
Welfare Indicator: MUAC Z-scores
Severe malnutrition in 2005/6:
 Median child MUAC z-score -1.22/-1.12 (Intervention/Control)


10 percent of children had Z-scores below -2.31/-2.14 (I/C)
25 percent of children had Z-scores below -1.80/-1.67 (I/C)

12



Sublocation-specific pseudo panel 2005/06-2008/09
Why pseudo-panel?
1.
Inconsistent child identifiers
2.
3.
MUAC data not available for all children in all months
Graduation out of and birth into the sample


How?
14 summary statistics for annual mean monthly sublocation specific stats: mean & percentiles and ‘poverty measures’
Focus on malnourished children


Thus, present analysis median MUAC Z-score of children z
≤
0
Control and intervention according to project investment

13
 Pseudo panel regression model where D is the intervention dummy variable of interest
NDVI is a control for agrometeorological conditions

L are District fixed effects to control for unobservables within major jurisdictions
No statistically significant average program impact

VARIABLES intervention dummy change in NDVI 2005/06-08/09
(1) median of
MUAC Z <0
(2)
10th percentile
(3)
25th percentile
(4) median of
MUAC Z <-1
(5) median of
MUAC Z <-2
0.0735
(0.248)
0.0832
(0.316)
0.0661
(0.371)
0.0793
(0.188)
0.0531
(0.155)
1.308*
(0.0545)
2.611***
(0.00294)
2.058***
(0.00754)
0.927*
(0.0997)
0.768*
(0.0767)
14
(change in NDVI) 2 2005/06-08/09 -12.91**
(0.0293)
Constant
Observations
R-squared
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
-8.672
(0.136)
-12.70*
(0.0510)
-0.954
(0.802)
1.924
(0.479)
0.501*** 0.892*** 0.839***
(2.99e-07) (1.40e-08) (8.70e-09)
114
0.319
114
0.299
114
0.297
0.203***
(0.000133)
114
0.249
0.120***
(0.00114)
106
0.280

15
Three steps:
 Steps 1 & 2: Simple differences
 SD within control and treatment over time:
No difference in trends. Both improved slightly.
 SD control vs. treatment at beginning and at end:
Control sublocations dominate in most cases, intervention never dominates.
 Step 3: SD on Diff-in-Diff (results focus for today)

16



In regression Diff-in-Diff: simply add (linear) controls
In SD-DD need a two step method
1.
Regress outcome variable on covariates
2.
Use residuals (the unexplained variation) in SD-DD
In application below, use first stage controls for agrometeorological conditions (as reflected in remotely-sensed vegetation measure, NDVI).

17
FOSD Difference Intervention vs. Difference Control
Median MUAC of obs<0. Categorization by Investment
For (drought-adjusted) median
MUAC z-scores:
 Below z=0.2, intervention sites FOD control sites, although not at 5% statistical significance level.
 ALRMP interventions appear moderately effective in preventing worsening nutritional status among children.
-1 -.4
.2
.8
1.4
2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09
Control intervention
FOSD Difference Intervention vs. Difference Control
Median MUAC of obs<0. Categorization by Investment
-1 -.4
.2
.8
1.4
2 difference in median MUAC Z-score with MUAC<0. drought adjusted. 2005/06-2008/09
Confidence interval (95 %) Estimated difference

FOSD Difference Intervention vs. Difference Control
25th percentile MUAC. Categorization by Investment
18
-1.5
-.8
-.1
.6
1.3
2 difference in 25th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
Control intervention

FOSD Difference Intervention vs. Difference Control
10th percentile MUAC. Categorization by Investment
19
-1.5
-.8
-.1
.6
1.3
2 difference in 10th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
Control intervention

20



Existing program evaluation approaches focus on estimating the average treatment effect. In some cases, that is not really the impact statistic of interest.
This paper introduces a new SD-based method to evaluate impact across entire distribution for non-experimental data
Results show the practical importance of looking beyond averages
 Standard Diff-in-Diff regressions: no impact at the mean


SD DD: intervention locations had fewer negative observations and of smaller magnitude, especially median and below
ALRMP II may have functioned as nutritional safety net (though only correlation, there is no way to establish causality)

21

22
1. SD and Poverty orderings
 Let SD s denote stochastic dominance of order s and P for poverty ordering (‘has less poverty’)
α stand



Let α=s-1
Then A P
α
B iff A SD s
B
SD and Poverty orderings are nested


A SD
1
A P
1
B  A SD
2
B  A P
2
B
B


A P
A SD
3
B
3
B

23
2. Poverty and Welfare orderings (Foster and Shorrocks 1988)
 Let U(F) be the class of symmetric utilitarian welfare functions



Then A P
α
Examples:
B iff A U
α
B



U
1 represents the monotonic utilitarian welfare functions such that
u’>0. Less malnutrition is better, regardless for whom.
U
2 represents equality preference welfare functions such that u’’<0. A mean preserving progressive transfer increases U
2
.
U
3 represents transfer sensitive social welfare functions such that
u’’’>0. A transfer is valued more lower in the distribution
Bottom line: For welfare levels tests up to third order make sense

24
Table 3 10 th
percentile MUAC Z-score – whole sample
Year Garissa Kajiado Laikipia Mandera Marsabit Mwingi Narok Nyeri Tharaka Turkana
2005/06 -2.4 -2.14
2008/09 -1.88 -2.22
-1.75
-2.1
-2.65
-2.13
-2.33
-2.29
-2.36
-2.14
-2.55
-2.35
-1.67
-1.54
-1.87
-1.74
-2.26
-2.25
Table 4 25 th
percentile MUAC Z-score – whole sample year Garissa Kajiado Laikipia Mandera Marsabit Mwingi Narok Nyeri Tharaka Turkana
2005/06 -1.97 -1.67
2008/09 -1.45 -1.76
-1.16
-1.4
-2.06
-1.69
-1.79
-1.69
-1.84
-1.68
-1.96 -1.2 -1.45
-1.76 -1.15 -1.28
-1.85
-1.86

25


Individual MUAC Z-score regression
To test program impact with much larger data set
 Still no statistically significant average program impact

Dependent variable: Individual MUAC Z-score
VARIABLES
time dummy (=1 for 2008/09) control - intervention by investment
Diff in diff
Normalized Difference Vegetation Index
Constant
26
Observations
R-squared
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
0.0785
(0.290)
-0.0576
(0.425)
0.0245
(0.782)
1.029***
(6.25e-07)
-1.391***
(0)
271061
0.033

I.1 Intervention 05/06-08/09
FOSD
SOSD
TOSD
I.2 Control 05/06-08/09
FOSD
SOSD
TOSD
II.1 Intervention vs. Control 05/06
FOSD
SOSD
TOSD
II.2 Intervention vs. Control 08/09
FOSD
SOSD
TOSD
III. Diff Intervention vs Diff.
Control
FOSD
SOSD
Sublocation panel
Median MUAC of obs < 0 % below -1 SD
Dominance Which* Signif. Dominance Which*
*
Y
Y
Y
08/09
08/09
08/09
NS
S
S
Almost
Y
Y
08/09
08/09
08/09
Y
Y
Y
08/09
08/09
08/09
NS
NS
NS
Y (almost) Control NS
Y Control NS
Y
N
Unclear
Unclear
Control NS
-
-
-
NS
NS
NS
Y
Y
Y
Almost
Y
Y
N
Y
Y
08/09
08/09
08/09
Control NS
Control NS
Control NS
Y
Y
Y
Y
Y
Y
Individual data
MUAC Z-Score
Signif. Dominance
NS
NS
NS
NS
NS
NS
# NS
Control NS
Control NS
Y
Y
Y
Y
Y
Y
Which* Signif.
08/09
08/09
08/09
08/09
08/09
08/09
Control
Control
Control
Control
Control
Control
S
S
S
S
S
S
S
S s
S
S
S
N
Y?
-
-
NS
NS
N
Y
-
Interve ntion
NS
NS
* Lower curves to the right are dominate for these indicators for which a greater number indicates ‘better’.
** For parts I. and II. higher curves to the left dominate for the proportion of observations below -1SD, as lower proportions are ‘better’. In contrast, for changes from 2005/06-2008/09 in part III. larger positive changes are better,
27
# Control sites dominate up to MUAC Z-score of -0.1. Intervention sites dominate for MUAC Z-score > 0.

FOSD Difference Intervention vs. Difference Control
Median MUAC of obs<0. Categorization by Investment
-1 -.4
.2
.8
1.4
2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09
Control intervention
28