Impact Evaluation Checklist
Bibliographical Information
1.
2.
3.
4.
Authors
Title
Year of publication
Publication type (Journal name, working paper, dissertation, etc)
Study Summary
5. Country
6. Intervention
7. Outcome—what is it?
a. Construct Validity (Are there inherent measurement problems with the outcome?
Index? Rare? Hard to measure? Variation? Regression to mean?)
b. Estimated effect (coefficient for preferred specification) and plain-English
interpretation
c. Standard Error / Confidence Interval: What can you rule out?
8. Sample
d. Representative?
e. Size? (For T/C)
f. Attrition?
g. Accretion?
Quality/Evaluation Design
9. Evaluation Design. Summary of Main IE Method
10. To what extent have the identification assumptions been satisfied for the main IE method?
11. Robustness: Did the IE use multiple IE estimation methods?
12. Describe other IE methods used and whether assumptions for those have been satisfied
13. Did the IE use multiple specifications (e.g. putting in and taking out potentially endogenous
variables)? If so, briefly describe.
14. Are the outcomes of interest sensitive to changes in method or specification?
15. Is the finding robust in your view? Why or why not? Are there remaining
concerns/challenges to internal validity?
16. What type of treatment effect is being estimated? Consider whether the effects are ATE /
ATT / ITT / LATE, and how this affects the interpretation of the results.
17. For whom are these estimates valid—what are the defining characteristics for inclusion in
the estimation sample? Connect your answer to sampling and the item of treatment
effects. For example, what types of individuals remain in the estimation sample after
imposing the common support condition, or which type of individuals are "compliers" if
the treatment effect estimated is a LATE.
18. Are there any other major concerns in the quality of this study? For example, History,
Maturation, Testing?
19. In your view, how credible are the findings in this study? Overall Quality Rating (High
Medium, Low).
Identifying Assumptions of 5 Major Impact Evaluation Methods
The quality of evaluation design is the most critical determinant of credibility of impact
evaluation estimates as being attributable to the intervention. In addition to the standard
threats to internal validity, following are the “Identifying Assumptions” or conditions under
which each of the main IE estimation methods is likely to be valid:
Assumptions under randomized experiment:
(i) Balanced treatment and control groups (the two groups having no statistically
significant difference in main baseline or time-invariant characteristics); and
(ii) No/Low noncompliance or handled sufficiently well (e.g., ITT). Minimal incidence
of beneficiaries not receiving treatment or spillover to controls.
Assumptions under double difference:
(i) Parallel trending (the treatment and control groups progress similarly in terms of
the outcomes of interests); and
(ii) No time-varying confounders (no time-variant characteristics that are not included
in the estimation specifications that may affect the progress of the outcomes beyond
the intervention).
Assumptions under matching:
(i) Common support (the overlap in terms of propensity scores or matching variables
between the treatment and control group);
(ii) Balancing checks (the treatment and control groups having no statistically
significant difference in main observable characteristics);
(iii) Matching on outcomes or covariates (the variables used to match are not affected
by the intervention); and
(iv) Correct specification of the participation decision: No selection on unobservables
(there are no factors except those explicitly included in the matching function which
matter for decision making. There should be a discussion of potential selection bias
due to unobservable differences between the treatment and control).
Assumptions under instrumental variables:
(i) First stage tested (the relationship between the intervention and the instrument is
statistically significant; F-test or Wald test > 10); and
(ii) Exclusion restriction (the instrument affects the outcome only via the intervention).
Assumptions under regression discontinuity:
(i) Selection around the assignment rule is effectively random (No sorting-beneficiaries tricking the rule to be eligible for the treatment); and
(ii) Balanced covariates at discontinuity (the two subgroups above and below the
eligibility cutoff have statistically similar characteristics).