Introducing Informal Inference
Using Data-Centric Lab
Exercises
Rakhee Patel, Rob Gould and Gretchen Davis
UCLA Department of Statistics
CAUSE Teaching and Learning Webinar - January 11, 2011
Outline



Why teach informal inference early?
Informal inference using randomization
Introductory computer lab assignments





Implementation
Goals
Lab example 1: tuberculosis data
Lab example 2: births and smoking data
Discussion
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 2 of 26
Informal Inference


Students often encounter difficulties with formal
inference methods later in introductory courses
Particularly, abstract concepts are difficult




What is a null distribution/hypothesis?
How do we know when to reject the null?
What is a p-value?
Introducing informal inference early in a course may
help with these struggles

Can do so without using complex vocabulary and
mathematical machinery
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 3 of 26
Randomization Tests

Randomization tests are a good way to illustrate
these concepts in a way that students can see with
their own eyes




Also a good way to illustrate power of computers
Can use many measures
In some cases, they more closely match the design of
experiment
See Cobb (2007), The Introductory Statistics Course:
A Ptolemaic Curriculum?
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 4 of 26
Computer Lab Assignments


Students complete eight weekly lab assignments
Each lab uses real data to answer a particular
investigative question


Lab also includes intermediate questions that help
guide students to answer the investigative question
Students ultimately turn in set of summary questions
regarding:




ultimate goals of lab
methods used
conclusions about research question
applications of concepts to real world
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 5 of 26
Implementation Goals


To understand how concepts in lecture are applied
to understanding real data
Design labs so that learning the software does not
get in the way of doing statistical analysis



First lab assignment introduces Fathom software
Subsequent labs provide additional instructions for any
new methods used
TAs will talk less, assist more.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 6 of 26
Informal Inference Goals


To understand why permuting observations
simulates a null distribution
To understand that null sampling distribution is used
to make decisions about rejecting null hypothesis


Develop intuition behind chance models
To understand how to use the null distribution to
estimate the p-value & make decisions

Use chance model to make inference about actual data
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 7 of 26
Example Lab 1: TB or Not TB?

Investigative Question


Objectives


Is Streptomycin an effective treatment for tuberculosis?
Using two-way tables, we can see how to make
conclusions about cause-and-effect by comparing
actual results to a chance model.
Data

Outcomes from Austin Bradford Hill’s first randomized
study in 1948 examining the effects of the antibiotic
Streptomycin vs. a control on 107 tuberculosis patients
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 8 of 26
Example Lab 1: Intermediate Question
1

Based on Hill’s study, does it seem that treatment
and outcome are independent or dependent?
Treatment and outcome
appear to be dependent.
26% died in control group
while only 7% died in
Streptomycin group.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 9 of 26
EXAMPLE LAB 1: INTERMEDIATE QUESTION 2

If treatment and outcome were independent, how
many patients in the Streptomycin group, according
to Hill’s study, would we expect to die if we
replicated the study again?
We would expect a proportion
of 18/107 = 0.168 to die
overall. So of the 55 patients
in the Streptomycin group, we
expect 0.168 of them, or 9.25
patients (roughly 9), to die.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 10 of 26
EXAMPLE LAB 1: INTERMEDIATE QUESTION 3

Design and execute an appropriate simulation in Fathom
using a chance model to replicate Hill’s study under the
assumption that treatment and outcome are independent.
Repeat the simulation 100 times and then use the results
from the chance model to determine whether (a) or (b) is the
most reasonable explanation for the actual data in Hill’s
study.


(a) Streptomycin is a significantly more effective treatment for
tuberculosis than bed rest. Thus, treatment and outcome are
dependent.
(b) The actual difference between treatments is due to chance
variation and Streptomycin may have no effect on tuberculosis.
Thus, it is possible that treatment and outcome are independent.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 11 of 26
EXAMPLE LAB 1: INTERMEDIATE QUESTION 3

Student discussions are crucial in determining how to
set up the simulations, as there are many ways

Question 2 might guide them to look at the number of
deaths in the Streptomycin group for each simulation
As expected, the chance model
shows an average of about 9 to 10
deaths in the Streptomycin group.
Only 4 died in the actual data which
never occurs in these simulations of
the model, so the chance model does
not fit well and (a) is more
reasonable.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 12 of 26
EXAMPLE LAB 1: COMMON MISCONCEPTIONS

Students confuse randomized data and expectations
with actual data:
If outcome is independent of treatment, we expect
9.25 deaths in the Streptomycin group. The chance
model also shows an average of between 9 and 10
deaths, so we cannot reject chance.

Students look at the distance between the actual
and the chance model rather than how often the
actual occurs under the chance model
Since the actual value of 4 is far from the expected
value of 9.25 then we reject the chance model.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 13 of 26
Example Lab 1: Intermediate Question
4

Is Hill’s study an example of an observational study or
an experiment? If we had seen a real difference
between the Streptomycin group and the control group
and concluded that Streptomycin was effective (and
maybe you did in Question 3), can we conclude that the
cause is the antibiotic? Why?
Hill’s study is an experiment because he imposed the
Streptomycin and control treatments on the patients, while
keeping all other factors constant. If we found
Streptomycin to be effective (which we did), we can
conclude that the antibiotic is the cause of the lower
number of deaths because Hill could completely control
the study and treatments.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 14 of 26
Example Lab 1: Observations



Without using the terms null hypothesis or p-value,
students should be able to find a significant difference
between the groups and a causal relationship
Students may have trouble with the concepts and
software used for this lab, but subsequent labs repeat
the same techniques to illustrate the same general
inference steps for various other types of data
It is important to show students several realizations from
the chance model to illustrate what happens under the
“no difference” hypothesis

Any differences that occur are strictly by chance
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 15 of 26
Example Lab 2: Compared to
What?

Investigative Question


Objectives


How does birth weight differ between smoking and
non-smoking mothers?
Using statistical summaries and the chance model, we
can explore whether mothers who smoke have babies
with quantifiably different weights than mothers who
do not.
Data

Characteristics of mothers and their babies for a
sample of 1,000 births in North Carolina in 2004
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 16 of 26
Example Lab 2: Intermediate Question
1

If a physician told you that a mother's smoking
affected the birth of her baby, how much, typically,
would you say the mother's smoking changes a
baby's weight at birth, based on the actual data?
I would guess (based on the sample
we have) that smoking during
pregnancy typically lowers the
baby’s weight by around 0.25 lbs,
the difference in median birth
weights.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 17 of 26
Example Lab 2: Intermediate Question
2



We will work with a chance model that assumes that
smoking has NO effect on the babies' weights.
Using this chance model, simulate a set of data and
calculate the difference in median birth weights
between the smoking and non-smoking groups. Repeat
this 100 times.
According to the chance model (your 100 simulations),
what would you estimate is the typical difference in
medians between the two groups? What would you
consider to be an unusually large difference, according
to the chance model?
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 18 of 26
Example Lab 2: Intermediate Question
2

Again, it is important for students to work together
to determine how to set up the simulation
I would expect the typical difference
in median birth weights to be close
to zero, which is reiterated by the
average difference of -0.011 shown
in the summary table. The plot shows
that an unusually large difference
seems to be around 0.2 lbs.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 19 of 26
Example Lab 2: Intermediate Question
3

In the actual data, the true difference in median birth
weights between the smoking mothers and non-smoking
mothers is 0.25 pounds. How often did the chance
model produce a difference this large or larger?
In the 100 chance model simulations,
I rarely saw a difference in medians
as large as 0.25 lbs (the actual
difference) as indicated by the dot
plot. In fact, only 2 out of the 100
simulations resulted in a difference
as large as the actual result.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 20 of 26
Example Lab 2: Intermediate Question
4

We have mentioned two possible reasons for
differences in the weights of babies from smoking
mothers versus non-smoking mothers in the actual data:



Smoking mothers really do have lighter babies.
The difference between birth weights from smokers and nonsmokers is due to chance.
Use your answer to Question 3 to justify which of these
explanations seems most reasonable
Since the actual difference in birth weights between the smoker and
non-smoker groups rarely occurs using the 100 chance model
simulations, the chance model does not seem to fit the data very
well and therefore it seems that smoking mothers actually do have
lighter babies.
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 21 of 26
Example Lab 2: Intermediate Question
5
Is this study an example of an observational study or an
experiment? Justify your answer. If we had seen a real
difference between birth weights from smokers and nonsmokers and concluded that smoking and birth weight are
associated (and maybe you did in Question 4), can we
conclude that smoking causes low birth weight? Why?
This is an observational study because the smoking was not imposed
on the mothers. If we had found a real difference in birth weights
between smokers and non-smokers (which we did), we still cannot
conclude that smoking is the cause of the lighter babies. This due to
the fact that with an observational study, we cannot control all factors
of the study (such as other bad habits, for example) and so these
other outside factors could affect the weight of the babies as well.

CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 22 of 26
Example Lab 2: Observations



Randomizing allows students to examine any
measure on a data set (such as the median)
Again, students should see several iterations from
the chance model to illustrate the “no difference”
hypothesis
Once students practice using randomization to make
inference, the intuition should become more and
more clear


Does chance model fit the data well?
If not, what does that mean?
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 23 of 26
Discussion



Repetition is key
Seeing is believing
Some future labs follow same structure




Fit model to data
Make inference about validity of model
Normal models can be thought of as nice shortcut to use
when assumptions are met
Students generally show less frustration once the formal
methods are taught

Performance on inference related problems has improved
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 24 of 26
Final Impressions and Future Work

Based on graded lab examples and evaluations:



Need more work to integrate the lessons learned in
lab into the lecture


Students seem to understand why permuting simulates a
null distribution, but are not sure what the null
distribution itself represents
Students don't know how to use the null
Perhaps more support materials are necessary
Integrating randomization-based testing with
parametric testing is more difficult than anticipated
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 25 of 26
Thank You


This work has been supported by grant
NSF DUE‐0737126
Contact information:


Rakhee Patel: rakhee@stat.ucla.edu
Rob Gould: rgould@stat.ucla.edu
CAUSE Teaching and Learning Webinar - January 11, 2011
Slide 26 of 26