Analyzing data: Causation
Review
What are evaluation criteria?
 What are step3 and step 4?
 What are the step3 and step4 output report?

S519
Step5: Analyzing data

Dealing with the causation issue, basically be able to
answers following questions:




How certain does the client need us to say that the
evaluand „caused“ a certain change?
What are the basic principles for inferring causation?
What types of evidence do we have available to help us
identify or rule out possible causal links?
How should we decide what blend of evidence will
generate the level of certainty needed most costeffectively?
S519
Certainty about causation (D-ch5)
Each decision-making context requires a
different level of certainty
 Quantitative or qualitative analysis


All-quantitative or all-qualitative
Sample choosing
 Sample size


Mix of them
S519
Inferring causation: basic principles

Two basic principles:



Look for evidence for and against the suspected main cause (i.e.,
evaluand)
Look for evidence for and against any important alternative causes (i.e.,
rival explanations)
Too many evidences or causes, which are the primary causes


All based on the level of certainty you need for your evaluation
Stepwise process

Put yourself in the hardest critics, gather enough evidence to support your
explanation


Repeat it until all remaining alternative explanations are ruled out.
Critical multiplism: triangle
The harder people attack, the more solid your answers need to be.
S519
Inferring causation: 8 strategies

1: ask observers

Two possibilities



Ask actual or potential impactees
Ask indirect impactees (i.e., co-worker, parents...)
Design your interview questions to include causation questions



E.g., how much has your knowledge increased as a result of attending this
program? – get primary cause
E.g., did anything else besides the program increase your knowledge in
this area over the same period of time? – get other causes
E.g., please describe anything else that has happened to you or someone
you know as a result of participating in this program? – get the causes
which people know or believe were caused by the program.
S519
Inferring causation: 8 strategies

1: ask observers



Causation-rich questions tend to be leading (direct the
respondent to answer in a particular way). Be careful about
the wording when designing interview questions
The causation question is not just whether the program
produced the effect but also what other factors enabled or
inhibited the effect.
Individual might not be a reliable witness to answer the
causation question, other evidence will be required to
make justifiable causal inferences.
S519
Inferring causation: 8 strategies

1: ask observers

Methods

Questionnaires to identify the targeted groups (people
who experienced substantial changes)


Using open-end to get more opinions
In-depth interview with the targeted groups

To find out causation.
S519
Inferring causation: 8 strategies

2. Check whether the content of the evaluand
matches the outcome
Alcoholics treatment program  alcoholics avoid
relapses
 Check whether the strategies which alcoholics use
to avoid relapses after the program, are the same
as the strategies taught in the program

S519
Inferring causation: 8 strategies

3. Look for other telltale patterns that suggest
one cause or another

Modus operandi method – look for evidence -detective metaphor to describe the way in which
potential causal explanations are identified and
tested.
S519
Inferring causation: 8 strategies

4. Check whether the timing of outcomes makes sense

Common sense:



an outcome should appear only at the same time as or after whatever
caused it – a considerable delay.
A further downstream the outcomes, the longer they should take to
appear
Using timing to confirm or disconfirm causal links:



Is the outcome happened before the evaluation? Or Other downstream
outcomes too early?
Is the timing of the outcomes logical to possible causes?
Do the further downstream outcomes in the logic model occur out of
sequence?
More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for
experimental research. Newbury Park, CA: Sage.
S519
Inferring causation: 8 strategies


4. Check whether the timing of outcomes makes sense
Example – a community health education on diet and exercise




Fairly immediate knowledge and skill gain: during or immediately after
the intervention
A short delay (days or weeks) before the knowledge and skills are
transformed into changing behavior
A moderate delay (weeks or months) before we see changes in
individual health indicators (weight, cholesterol, blood pressure, etc.)
A long delay (months or years) to see changes on improvement on
diabetes and heart diseases
S519
Exercise

Take grantsmanship workshop as one example


List the timeline potential outcomes (fairly immediate, a short delay, a
moderate delay, a long delay)
Using timing strategies to confirm or disconfirm the cause links, state
one page for how and why:





One month after the workshop, 3 proposals got grants
One year after the workshop, 3 proposals got grants
Three months after the workshop, some people write good proposals, but
some are not.
Think about your own solution
Form a group and discuss
S519
Inferring causation: 8 strategies

5. Check whether the „dose“ is related
logically to the „response“.

The dose-response idea
The more dose of drug, the better response later on
 If more A (dose), then better B (response)
 Compare the less dose with more dose (not overdose)
to confirm or disconfirm the cause links


E.g., for performance evaluation project, if we found that
performance had been improved dramatically in the unit
where the system has been poorly implemented,  this
system is not the cause of the performance improvement.
S519
Inferring causation: 8 strategies

6. Make comparisons with a „control“ or
„comparison“ group

Divide the participants into different groups


control group (receive the evaluand) vs. Comparison
group (receive no evaluand)
Sampling should be done carefully to make sure
no systematic differences between groups
Sample size
 randomization

S519
Inferring causation: 8 strategies

7. Control statistically for extraneous variables

When using control and comparison groups, try
to exclude external variables and make two
groups no systematic differences:

Statistical methods


Regression analysis
Try to identify other potential systematic differences

E.g., math improvement for students, how to sample students
and think about other potential existing difference. Is the
random sampling enough?
S519
Inferring causation: 8 strategies

8. Identify and check the underlying causal
mechanism(s)

Try to look for an underlying mechanism to make
the case for causation more or less convincing

Cigarette smoking  lung cancer



Correlation studies
Carcinogenic in cigarette causes cancer
Normally coming from literature.
S519
Put them together


Do we need all the evidences we collect from 8
strategies?
How to select them?



Put yourself in the shoes of a tough critic, identify the most
potential threatening rival explanation, then chose the
types of evidence that will most quickly and cost-effectively
confirm or dispel that rival explanation.
Go to next less tough rival explanation, ...
Continue, until you have amassed a body of evidence to
provide you enough certainty to draw causal inferences
S519
Exercise

Grantsmanship workshop

Grantsmanship workshop strengthen local communities






For (evidences)
Against (evidences)
Other alternative causes (i.e., rival explanations)
Using strategies to confirm or disconfirm these evidences
or causes
Putting them together
Form a group to discuss
S519