S519: Evaluation of
Information Systems
Analyzing data:
Causation
Ch5
Step5: Analyzing data

Dealing with the causation issue, basically be
able to answers following questions:




How certain does the client need us to say that the
evaluand „caused“ a certain change?
What are the basic principles for inferring causation?
What types of evidence do we have available to help
us identify or rule out possible causal links?
How should we decide what blend of evidence will
generate the level of certainty needed most costeffectively?
Certainty about causation (Dch5)


Each decision-making context requires a
different level of certainty
Quantitative or qualitative analysis

All-quantitative or all-qualitative



Sample choosing
Sample size
Mix of them
Inferring causation: basic
principles

Two basic principles:



Look for evidence for and against the suspected main cause (i.e.,
evaluand)
Look for evidence for and against any important alternative
causes (i.e., rival explanations)
Too many evidences or causes, which are the primary
causes


All based on the level of certainty you need for your evaluation
Stepwise process

Put yourself in the hardest critics, gather enough evidence to support
your explanation


Repeat it until all remaining alternative explanations are ruled out.
Critical multiplism: triangle
The harder people attack, the more solid your answers need to be.
Inferring causation: 8
strategies

1: ask observers

Two possibilities



Ask actual or potential impactees
Ask indirect impactees (i.e., co-worker, parents...)
Design your interview questions to include causation questions



E.g., how much has your knowledge increased as a result of
attending this program? – get primary cause
E.g., did anything else besides the program increase your
knowledge in this area over the same period of time? – get other
causes
E.g., please describe anything else that has happened to you or
someone you know as a result of participating in this program? – get
the causes which people know or believe were caused by the
program.
Inferring causation: 8
strategies

1: ask observers



Causation-rich questions tend to be leading (direct the
respondent to answer in a particular way). Be careful
about the wording when designing interview questions
The causation question is not just whether the
program produced the effect but also what other
factors enabled or inhibited the effect.
Individual might not be a reliable witness to answer
the causation question, other evidence will be
required to make justifiable causal inferences.
Inferring causation: 8
strategies

1: ask observers

Methods

Questionnaires to identify the targeted groups (people who
experienced substantial changes)


Using open-end to get more opinions
In-depth interview with the targeted groups

To find out causation.
Inferring causation: 8
strategies

2. Check whether the content of the evaluand
matches the outcome


Alcoholics treatment program  alcoholics avoid
relapses
Check whether the strategies which alcoholics
use to avoid relapses after the program, are the
same as the strategies taught in the program
Inferring causation: 8
strategies

3. Look for other telltale patterns that suggest
one cause or another


Modus operandi method – look for evidence -detective metaphor to describe the way in which
potential causal explanations are identified and tested.
A silly example


Evidences: a naked man, dead; in the middle of the desert;
personal belongings near by; half match at hand
Cause of his death
Inferring causation: 8
strategies

4. Check whether the timing of outcomes makes sense

Common sense:



an outcome should appear only at the same time as or after whatever
caused it – a considerable delay.
A further downstream the outcomes, the longer they should take to
appear
Using timing to confirm or disconfirm causal links:



Is the outcome happened before the evaluation? Or Other
downstream outcomes too early?
Is the timing of the outcomes logical to possible causes?
Do the further downstream outcomes in the logic model occur out of
sequence?
More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for
experimental research. Newbury Park, CA: Sage.
Inferring causation: 8
strategies


4. Check whether the timing of outcomes makes sense
Example – a community health education on diet and
exercise




Fairly immediate knowledge and skill gain: during or immediately
after the intervention
A short delay (days or weeks) before the knowledge and skills
are transformed into changing behavior
A moderate delay (weeks or months) before we see changes in
individual health indicators (weight, cholesterol, blood pressure,
etc.)
A long delay (months or years) to see changes on improvement
on diabetes and heart diseases
Lab
Exercise

Take grantsmanship workshop as one example


List the timeline potential outcomes (fairly immediate, a short
delay, a moderate delay, a long delay)
Using timing strategies to confirm or disconfirm the cause links,
state one page for how and why:





One month after the workshop, 3 proposals got grants
One year after the workshop, 3 proposals got grants
Three months after the workshop, some people write good proposals,
but some are not.
Think about your own solution
Form a group and discuss
Inferring causation: 8
strategies

5. Check whether the „dose“ is related
logically to the „response“.

The dose-response idea



The more dose of drug, the better response later on
If more A (dose), then better B (response)
Compare the less dose with more dose (not overdose)
to confirm or disconfirm the cause links

E.g., for performance evaluation project, if we found that
performance had been improved dramatically in the unit
where the system has been poorly implemented,  this
system is not the cause of the performance improvement.
Inferring causation: 8
strategies

6. Make comparisons with a „control“ or
„comparison“ group

Divide the participants into different groups


control group (receive the evaluand) vs. Comparison
group (receive no evaluand)
Sampling should be done carefully to make sure
no systematic differences between groups


Sample size
randomization
Inferring causation: 8
strategies

7. Control statistically for extraneous variables

When using control and comparison groups, try to
exclude external variables and make two groups
no systematic differences:

Statistical methods


Regression analysis
Try to identify other potential systematic differences

E.g., math improvement for students, how to sample
students and think about other potential existing difference.
Is the random sampling enough?
Inferring causation: 8
strategies

8. Identify and check the underlying causal
mechanism(s)

Try to look for an underlying mechanism to make
the case for causation more or less convincing

Cigarette smoking  lung cancer



Correlation studies
Carcinogenic in cigarette causes cancer
Normally coming from literature.
Put them together


Do we need all the evidences we collect from 8
strategies?
How to select them?



Put yourself in the shoes of a tough critic, identify the
most potential threatening rival explanation, then
chose the types of evidence that will most quickly and
cost-effectively confirm or dispel that rival explanation.
Go to next less tough rival explanation, ...
Continue, until you have amassed a body of evidence
to provide you enough certainty to draw causal
inferences
Lab
Exercise

Grantsmanship workshop (p57)

Grantsmanship workshop strengthen local
communities






For (evidences)
Against (evidences)
Other alternative causes (i.e., rival explanations)
Using strategies to confirm or disconfirm these
evidences or causes
Putting them together
Form a group to discuss