How to Implement a
Randomization-Based Introductory
Statistics Course:
The CATALST Curriculum
Bob delMas, Laura Le, Nicola Parker
and Laura Ziegler
Funded by NSF DUE-0814433
Overview of Workshop
•
•
•
•
Overview of CATALST Course
TinkerPlots Introduction
UNIT 1: Chance Models and Simulation
UNIT 2: Models for Comparing Groups –
Randomization Methods
• UNIT 3: Estimating Models Using Data –
Bootstrap Methods
• Assessment Results
University of Minnesota
Project Team
FACULTY
• Joan Garfield
• Andy Zieffler
• Bob delMas
GRADUATE STUDENTS
• Rebekah Isaak
• Laura Le
• Laura Ziegler
Our Collaborators
•
•
•
•
•
•
Allan Rossman, Cal Poly State U
Beth Chance, Cal Poly State U
John Holcomb, Cleveland State U
George Cobb, Mt. Holyoke Coll.
Herle McGowan, NCSU
Instructors at different institutions
who have implemented CATALST
CATALST Course:
Teaching Students to Really Cook
• Metaphor from Alan Schoenfeld (1998)
• Many intro stats classes teach how to follow
“recipes” but not how to really “cook.”
 Able to perform routine procedures and tests
 Don't have the big picture that allows them to solve
unfamiliar problems and to articulate and apply their
understanding.
• Someone who knows how to cook knows the
essential things to look for and focus on, and how
to make adjustments on the fly.
CATALST: The Inference Model
The core logic of inference as the
foundation (Cobb, 2007)
• Model: Specify a model to reasonably
approximate the variation in outcomes
attributable to the random process
• Randomize & Repeat: Use the model to
generate simulated data and collect a
summary measure
• Evaluate: Examine the distribution of the
resulting summary measures
Radical Content
• No t-tests; Use of probability for simulation and
modeling (TinkerPlots )
• Coherent curriculum that builds ideas of models,
chance, simulated data, inference from first day
• Immersion in statistical thinking
• Activities based on real problems, real data
• Course materials contain all classroom activities
and homework assignments
• Lesson plans posted at CATALST website
Radical Pedagogy
• Student-centered approach based on research
in cognition & learning, instructional design
principles
• Minimal lectures, just-in-time as needed
• Cooperative groups to solve problems
• “Invention to learn” and "test and conjecture"
activities [develop reasoning & promote transfer]
• Writing & whole class discussion (wrap-up)
Why TinkerPlots ?
• Fathom® is a viable option for building models and
simulation, but also challenging to students (Maxara &
Biehler, 2006, 2007; Biehler & Prommel , 2010)
• TinkerPlots™ was chosen because of unique visual
(graphical interface) capabilities
 Allows students to see the devices they select (e.g.,
sampler, spinner)
 Easily use these models to simulate and collect data
 Allows students to visually examine and evaluate
distributions of statistics
3 CATALST Units
(14 Week Semester)
• Chance Models and Simulation
 Learning to use the core logic of inference (George Cobb)
• Specify a chance model (sampling or random assignment)
• Generate a trial, Collect measure, Repeat many times
• Evaluate fit of chance model to the observed data
• Models for Comparing Groups
 Randomization Tests
• Random Assignment Studies
• Observed Data Studies
 Design: Random Assignment and Random Sampling
 Drawing valid conclusions using logic of inference
• Estimating Models Using Data
 Bootstrap Method
 Standard Error of a Sample Statistic
 Confidence Intervals
Workshop Overview Summary
You will work through shortened versions of
three activities from the CATALST course.
You will do one activity from:
•Unit 1 - Chance Models and Simulation
•Unit 2 - Models for Comparing Groups
•Unit 3 - Estimating Models Using Data
Preliminary Assessment Results
Agenda for Session 1
• An introduction to TinkerPlots
• One activity from Unit 1 on Chance
Models and Simulation
 A modified version of the Matching
Dogs to Owners
An introduction to TinkerPlots :
Modeling Dice
Matching Dogs to Owners:
Learning Goals
• Develop the reasoning of statistical
significance
• Understand how observed result
can be judged unlikely under a
particular model,
• Begin the process of statistical
thinking
Matching Dogs to Owners:
Student Preparation
At this point students have….
•Modeled random behavior in TinkerPlots
 Coins, dice
 Colors of M&M candies
 Effects of “One Son” or “One of Each” child
policies (using a stopping criterion)
…and Introductory readings on hypothesis testing
have been assigned
Matching Dogs to Owners:
Research Question
http://media-cachelt0.pinterest.com/192x/97/b8/56/97b856be7d0a265348a6f2ea71531ce0.jpg
Matching Dogs to Owners:
Research Question
• Are humans able to match dogs to
owners better than blind luck?
Matching Dogs to Owners:
Simulation
• For each numbered person in the
following photos, write down the letter
for the dog that you think is owned by
that person.
1. _____
A
2. _____
B
3. _____
C
D
4. _____
E
5. _____
6. _____
F
1. _A__
A
2. _C__
B
3. _F__
C
D
4. __E__
E
5. __B__
6. __D__
F
Matching Dogs to Owners:
Research Question
• Are humans able to match dogs to
owners better than blind luck?
Matching Dogs to Owners:
Directions
• Let’s do the modified activity
• Again, put on your “student hat”!
• Afterward, you will get to put your
“teacher hat” back on
Matching Dogs to Owners:
Wrap-up
• What does the “blind guessing” model
mean?
Matching Dogs to Owners:
Wrap-up
• Why do we use the “blind guessing”
model to simulate our data?
Matching Dogs to Owners:
Wrap-up
• What does it mean to have evidence that
supports or does not support the “blind
guessing” model?
Matching Dogs to Owners:
Wrap-up
• Tinkerplots steps
Matching Dogs to Owners:
Reflection
• What about this activity (content, format…) do you
think might help maximize student learning in your
classroom?
• What are your hesitations / what do you think
might hinder student learning in your classroom?
• What questions do you still have about the
implementation of such a course?
• Presuming that you wanted to implement these
activities in your courses, how comfortable would
you feel doing so? Why or why not?
Matching Dogs to Owners:
Reflection
• What about this activity (content,
format…) do you think might help
maximize student learning in your
classroom?
Matching Dogs to Owners:
Reflection
•What are your hesitations about this
activity / what do you think might hinder
student learning if you were to use it in
your classroom?
Matching Dogs to Owners:
Reflection
• What questions do you still have about
the implementation of this type of
activity?
Matching Dogs to Owners:
Reflection
• Presuming that you wanted to
implement these activities in your
courses, how comfortable would you
feel doing so?
Why or why not?
Matching Dogs to Owners:
Building the Model & Simulation
Matching Dogs to Owners:
Building the Model & Simulation
Matching Dogs to Owners:
Building the Model & Simulation
Matching Dogs to Owners:
Building the Model & Simulation
Matching Dogs to Owners:
Building the Model & Simulation
Agenda for Session 2
• Two activities from Unit 2 on Comparing
Groups
 A modified version of the Sleep
Deprivation Activity
 A modified version of the Contagious
Yawns Study Homework
Sleep Deprivation:
Learning Goals
• Develop the need for a summary
measure to compare two groups with
quantitative data (e.g., different
summary measures provide different
information regarding characteristics of
the data; some more relevant than
others)
Sleep Deprivation:
Learning Goals
• Learn how to determine if a single observed
difference is real and important or just due to
chance (the need to consider the observed
result in the distribution of results that are
possible under the null model)
• Find the approximate p-value from simulated
data and draw a conclusion
Sleep Deprivation:
Prior Knowledge
• Informal idea of p-value, but the term is
introduced in this activity
• Basic idea of comparing groups
• Randomization test (by hand)
Sleep Deprivation:
Student Preparation
• Students read the abstract from Stickgold,
R., James, L., & Hobson, J. A. (2000). Visual
discrimination learning requires sleep after
training. Nature Neuroscience, 3(12), 12371238.
• The reading is a scientific article abstract that
introduces the context of and provides
motivation for the Sleep Deprivation activity.
Sleep Deprivation:
Preliminary Discussion
• Begin with a discussion on how to
measure whether or not the amount of
sleep effects test performance.
 Have students come up with different
methods.
 Describe how that relates to the activity.
Sleep Deprivation:
Research Question
• Does the effect of sleep deprivation last,
or can a person “make up” for sleep
deprivation by getting a full night’s
sleep in subsequent nights?
Stickgold, R., James, L., & Hobson, J. A. (2000). Visual
discrimination learning requires sleep after training. Nature
Neuroscience, 3(12), 1237-1238.
Sleep Deprivation:
Study Design
11 Sleep
Deprived
21 Human Subjects
10
Unrestricted
Sleep
Sleep Deprivation:
Directions
• Let’s do the modified activity
• Again, put on your “student hat”!
• Afterward, you will get to put your
“teacher hat” back on
Sleep Deprivation:
Sample Wrap-Up Questions
• What was our null model?
• Why do we need to conduct a test, why can't
we just look at the observed difference?
• What is the purpose of random assignment?
• Where was the plot centered? Why does that
make sense?
• What is a p-value?
• What conclusion did you come to for the sleep
study?
Sleep Deprivation:
Building the Model & Simulation
Options
Fastest
Repeat Improvement
21
Draw
2
Condition
-10.7
-10.7
-14.7
45.6
Mixer
11
total
10 21
Depriv... Unrest...
Stacks
Spinner
Bars
Curve
Counter
Sleep Deprivation:
Building the Model & Simulation
Options
Fastest
Repeat Improvement
21
Draw
2
Join
Condition
-10.7
-10.7
-14.7
45.6
Mixer
Results of Sampler 1
11
total
10 21
Depriv... Unrest...
Stacks
Spinner
Bars
Curve
Counter
Options
Improv... Condition <new>
1
-14.7,De...
-14.7 Deprived
2
-10.7,Un...
-10.7 Unrestric...
3
-10.7,De...
-10.7 Deprived
4
2.2,Depr...
2.2 Deprived
5
2.4,Unre...
2.4 Unrestric...
6
4.5,Unre...
4.5 Unrestric...
Sleep Deprivation:
Building the Model & Simulation
Options
Fastest
Results of Sampler 1
Join
Draw
2
Options
Condition
-10.7
-10.7
-14.7
45.6
Mixer
Results of Sampler 1
Improv... Condition <new>
11
total
10 21
Depriv... Unrest...
Stacks
Spinner
Bars
Curve
Counter
1
-14.7,De...
-14.7 Deprived
2
-10.7,Un...
-10.7 Unrestric...
3
-10.7,De...
-10.7 Deprived
4
2.2,Depr...
2.2 Deprived
5
2.4,Unre...
2.4 Unrestric...
6
4.5,Unre...
4.5 Unrestric...
Condition
Repeat Improvement
21
Options
Unrestricted
7
15.875
Deprived
-20
0
8.77692
20
40
Improvement
15.875 - 8.77692 = 7.09808
Circle Icon
60
Sleep Deprivation:
Building the Model & Simulation
Options
Fastest
Results of Sampler 1
Join
11
total
10 21
Depriv... Unrest...
Mixer
Options
Condition
-10.7
-10.7
-14.7
45.6
Draw
2
Results of Sampler 1
Improv... Condition <new>
Stacks
Spinner
Bars
Curve
Counter
1
-14.7,De...
-14.7 Deprived
2
-10.7,Un...
-10.7 Unrestric...
3
-10.7,De...
-10.7 Deprived
4
2.2,Depr...
2.2 Deprived
5
2.4,Unre...
2.4 Unrestric...
6
4.5,Unre...
4.5 Unrestric...
Condition
Repeat Improvement
21
Options
Unrestricted
7
15.875
Deprived
-20
0
8.77692
20
40
Improvement
15.875 - 8.77692 = 7.09808
History of Results of Sampler 1
Diff_Imp... <new>
494
-0.09166...
495
-8.86
496
2.26944
497
4.42778
498
9.04722
499
-11.0306
500
0.733333
Collect
499
Options
Circle Icon
60
Sleep Deprivation:
Building the Model & Simulation
Options
Fastest
Results of Sampler 1
Join
Draw
2
Options
Improv... Condition <new>
Condition
-10.7
-10.7
-14.7
45.6
Mixer
Results of Sampler 1
11
total
10 21
Depriv... Unrest...
Stacks
Spinner
Bars
Curve
Counter
1
-14.7,De...
-14.7 Deprived
2
-10.7,Un...
-10.7 Unrestric...
3
-10.7,De...
-10.7 Deprived
4
2.2,Depr...
2.2 Deprived
5
2.4,Unre...
2.4 Unrestric...
6
4.5,Unre...
4.5 Unrestric...
Condition
Repeat Improvement
21
Options
Unrestricted
7
15.875
Deprived
-20
0
8.77692
20
40
Improvement
15.875 - 8.77692 = 7.09808
Circle Icon
History of Results of Sampler 1
Options
100%
History of Results of Sampler 1
Collect
499
0% 0%
Options
Diff_Imp... <new>
494
-0.09166...
495
-8.86
496
2.26944
497
4.42778
498
9.04722
499
-11.0306
500
0.733333
-25
-20
Circle Icon
-15
-10
-5
0
Diff_Improvement
5
10
15
20
60
Contagious Yawns Study:
Homework Assignment
• Research Question:
 Are yawns contagious?
Contagious Yawns Study:
Study Design
34 Yawn
Seed Planted
16 No Yawn
Seed Planted
50 Human Subjects
MythBusters [Photograph]. Retrieved April 11, 2013, from:
http://store.discovery.com/?v=discovery_shows_mythbusters
Contagious Yawns Study:
Data
Subject
Yawned
Subject
Did Not
Yawn
Total
Yawn Seed
Planted
10
24
34
Yawn Seed
Not Planted
4
12
16
14
36
50
Total
Contagious Yawns Study:
Questions
• Describe how you would create a model
to answer the research question.
• Describe the simulation process you
would use to answer the research
question.
Contagious Yawns Study:
Building the Model & Simulation
Contagious Yawns Study:
Homework Questions
• Quantify the strength of evidence/p-value for
the observed result. Note that the difference
in the proportion that yawned was 0.044.
• In light of your previous answer, would you
say that the results that the researchers
obtained provide strong evidence that
yawning is contagious? Explain your
reasoning based on your simulation results.
Contagious Yawns Study:
Reflection
• What was the purpose of this
homework assignment?
• How did it differ from the Sleep
Deprivation Activity?
Agenda for Session 3
• Overview of Unit 3 on Estimating
Models Using Data
• A demonstration of the Kissing the Right
Way Activity
Unit 3 Overview:
Estimating Models Using Data
• Sampling
 Bias, precision, and size of population
• Comparing Hand Spans
 Formalizes summary measure of variation
(standard deviation)
• Kissing The Right Way
 Intro to confidence intervals
• Memorizing Letters Part II
 Confidence interval for effect size
• Why 2 Standard Errors?
Kissing the Right Way
Kissing the Right Way:
Learning Goals
• Understand what standard error is
• Understand the difference between standard
deviation and standard error
• Understand what margin of error is
• Understand what an interval estimate is
• Understand the purpose of an interval
estimate
Kissing Study Activity
What percentage of couples
lean their heads to the right
when kissing?
How can we find
out?
Collect data!
Kissing Study Activity
80 Couples
Lean Right
44 Couples
Lean Left
124 Couples
Kissing Activity
Sixty-five percent of the
couples observed leaned to
the right. What percentage of
all couples lean to the right?
How much variation is
there in the estimate from
sample-to-sample?
Use sample as a
substitute for the
population
Kissing Study:
Building the Model & Simulation
Kissing Study:
Building the Model & Simulation
Kissing Study:
Building the Model & Simulation
Kissing Study:
Building the Model & Simulation
Kissing Study:
Building the Model & Simulation
Kissing the Right Way:
Sample Wrap-Up Questions
•
•
•
•
•
•
•
•
What is standard error?
What is margin of error?
When would we need to use margin of error?
How would you interpret the margin of error?
What is an interval estimate?
What is the purpose of an interval estimate?
How would you interpret your interval estimate?
What do you think the purpose was of this
activity?
Agenda for Session 4
• Summary of Assessment Results
• Overview of Assessment Instruments
• Comparison of CATALST and NonCATALST students
CATALST: Assessment
• GOALS: 27 forced-choice items






Study design
Reasoning about variability
Sampling and sampling variability
Interpreting confidence intervals and p-values
Statistical inference
Modeling and simulation
• MOST: Measure of statistical thinking
 4 real-world contexts
 open-ended and forced-choice items
• AFFECT: attitudes and perceptions
Performance of the CATALST (n = 289) &
non-CATALST (n = 440) groups on the GOALS test
Bootstrapped Confidence Interval Limits for Each Item
(CATALST: n = 289; non-CATALST: n = 440)
GOALS Item 1
Design & Conclusions
A recent research study randomly assigned participants into groups that were
given different levels of Vitamin E to take daily. One group received only a placebo
pill. The research study followed the participants for eight years to see which ones
developed a particular type of cancer during that time period.
What is the primary purpose of the use of random assignment for making
inferences based on this study?
CATALST
17.0%
66.7%
16.3%
NON-CATALST Response Options
a. To ensure that a person doesn’t know whether or not they are
38.2%
getting the placebo.
b. To ensure that the groups are similar in all respects except for
29.1%
the level of VitamE.
c. To ensure that the study participants are representative of the larger
32.7%
population.
GOALS Item 3
Design & Conclusions
A local television station for a city with a population of 500,000 recently conducted
a poll where they invited viewers to call in and voice their support or opposition to
a controversial referendum that was to be voted on in an upcoming election. Over
5,000 people responded, with 67% opposed to the referendum. The TV station
announced that the referendum would most likely be defeated in the election.
Select the best answer below for why you think the TV station's announcement is
valid or invalid.
CATALST
5.2%
4.8%
7.2%
82.7%
NON-CATALST Response Options
a. Valid, because the sample size is large enough to represent the
22.3%
population.
b. Valid, because 67% is far enough above 50% to predict a majority
17.5%
vote
16.8%
c. Invalid, because the sample is too small given the size of the city
43.4%
d. Invalid, because the sample is not likely to be representative of
the population
GOALS Items 23, 24 & 25
Understanding p-Values
For questions 23-25, indicate whether the interpretation of the p-value provided is
valid or invalid.
CORRECT
RESPONSE
CATALST
NonCATALST
The p-value is the probability that the $5
incentive group would have the same or lower
success rate than the “do your best” rate.
INVALID
82.3%
58.0%
The p-value is the probability that the $5
incentive group would have a higher success
rate than the “do your best” group.
INVALID
58.4%
48.9%
The p-value is the probability of obtaining a
result as extreme as was actually found, if the
$5 incentive is really not helpful.
VALID
70.0%
51.4%
39.5%
19.3%
STATEMENT
ANSWER ALL THREE ITEMS CORRECTLY
MOST Test: Statistical Thinking
Example of an Open-Ended Item: Context
Consider an experiment where a researcher wants to study
the effects of two different exam preparation strategies on
exam scores. Twenty students volunteered to be in the
study, and were randomly assigned to one of two different
exam preparation strategies, 10 students per strategy.
After the preparation, all students were given the same
exam (which is scored from 0 to 100). The researcher
calculated the mean exam score for each group of
students. The mean exam score for the students assigned
to preparation strategy A was 5 points higher than the
mean exam score for the students assigned to preparation
strategy B.
MOST Test: Statistical Thinking
Open-ended Question
7. Explain how the researcher could determine whether
the difference in means of 5 points is large enough to
claim that one preparation strategy is better than the
other. (Be sure to give enough detail that someone
else could easily follow your explanation in order to
implement your proposed analysis and draw an
appropriate inference (conclusion).)
MOST Test: Sample Scoring Rubric
Component
Randomization-Based
Parametric-Procedures
Modeling:
Hypothesis
Describes the null model of no Describes the Null and
difference
Alternative hypotheses
Context:
Hypothesis
Identifies the context in
definition of null model
Identifies the context in definition
of null and alternative
hypotheses
Modeling: Test
Describes the simulation:
statistic to collect & repetition
States a z-test & describes
checking assumptions
Conclusions
Description identifies how conclusion can be drawn from the test
results
Context:
Conclusion
Conclusion presented in the context of the problem: Refers to the
percentage of breakups for Monday or percentage of breakups for
each day of the week
MOST Test: Preliminary Findings
• CATALST students include a higher percentage of
procedural components
• For all students, responses are weakest on making
conclusions and including context
• CATALST students more likely to describe the use of
technology in their responses
• CATALST students tend to describe steps in more detail
• CATALST students more likely to map a problem to a
previously solved problem
• Non-CATALST students more likely to present a nonstatistical explanation for outcomes
Affect Survey:
Percent Agree or Strongly Agree
Non-CATALST
(n = 453)
CATALST
(n = 283)
Learning to use (software/TinkerPlotsTM) was an
important part of learning statistics.
68.7%
81.5%
I would be comfortable using
(software/TinkerPlotsTM) to test for a difference
between groups after completing this class.
66.0%
91.0%
I would be comfortable using
(software/TinkerPlotsTM) to compute an interval
estimate for a population parameter after
completing this class.
61.7%
83.7%
Learning to (use software/create models in
TinkerPlotsTM) helped me learn to think
statistically.
61.6%
84.5%
CATALST: Interview Study
• Examined statistical reasoning of 5 tertiary students after
Chance Models and Simulation unit
• Each of two novel problems asked students to reason
about the likelihood of a surprising result
• After five weeks, students’ demonstrated consideration of
sampling variability when drawing inferences
• Students’ solutions typically:
 Considered the value most likely to occur under the
chance model (expected value)
 Demonstrated an awareness of sampling variability
 Quantified the degree of “unusualness”
CATALST publications
Garfield, J., delMas, R. & Zieffler, A. (2012). Developing statistical
modelers and thinkers in an introductory, tertiary-level statistics course.
ZDM: The International Journal on Mathematics Education.
Ziegler, L. and Garfield, J. (2013) Exploring students' intuitive ideas of
randomness using an iPod shuffle activity. Teaching Statistics, 35(1), 27.
Isaak, R., Garfield, J. and Zieffler, A. (in press). The Course as Textbook.
Technology Innovations in Statistics Education.
Garfield, J., Zieffler, A., delMas, R. & Ziegler, L. (under review). A New
Role for Probability in the Introductory College Statistics Course.
Journal of Statistics Education.
delMas, R. , Zieffler, A. & Garfield, J. (under review). Tertiary Students'
Reasoning about Samples and Sampling Variation in the Context of a
Modeling and Simulation Approach to Inference. Educational Studies in
Mathematics.
Thank You for your Participation
Catalysts for Change (2012). Statistical thinking: A
simulation approach to modeling uncertainty.
Minneapolis, MN: CATALST Press. (Purchase at
amazon.com)
If you have any questions about the CATALST course,
please contact anyone of the Pis:
Joan Garfield: jbg@umn.edu
Robert delMas: delma001@umn.edu
Andy Zieffler: zief0002@umn.edu
We appreciate your feedback – please fill out the
workshop evaluation