Ask Good Questions
Allan J. Rossman
Dept of Statistics
Cal Poly – San Luis Obispo (USA)
8th International Conference on Teaching Statistics
Frank Sinatra

What’s the secret to
success as a singer?
 “Sing Good Songs”
Rossman
Ask Good Questions (ICOTS-8)
2
My similarly succinct suggestion

What’s the secret to success at
teaching statistics?

“Ask Good Questions”
Rossman
Ask Good Questions (ICOTS-8)
3
No hidden message

My take-home message
 Ask Good Questions
Quiz at end of presentation
 Q: What was my point?
 A: Ask Good Questions

I may email you in 10 years



Rossman
Q: What do you remember?
A: Ask Good Questions
Ask Good Questions (ICOTS-8)
4
George Cobb

“Judge a textbook by its
exercises, and you
cannot go far wrong.”
Rossman
Ask Good Questions (ICOTS-8)
5
My corollary

“Judge a teacher by the questions that he/she asks
of students, and you cannot go far wrong.”
Rossman
Ask Good Questions (ICOTS-8)
6
What kinds/purposes of questions?

Guide students to develop their understanding
and skills


Learning activities
Assess how well students have learned

Rossman
Assessment questions
Ask Good Questions (ICOTS-8)
7
What makes a question “good”?

I respectfully decline to answer


Instead I’ll present some examples of both
kind of questions (learning activities,
assessment questions)


For now
Taken from “Stat 101” (introductory, algebrabased, service course at tertiary level)
Then I’ll revisit this question
Rossman
Ask Good Questions (ICOTS-8)
8
Four examples (learning activities)
Rossman
Ask Good Questions (ICOTS-8)
9
George Cobb (again)

“Shorn of all subtlety
and led naked out of
the protective fold of
educational research
literature, there comes
a sheepish little fact:
lectures don’t work
nearly as well as many
of us would like to
think.”
Rossman
Ask Good Questions (ICOTS-8)
10
Example 1: TVs and life expectancy

Is there an association between a country’s
life expectancy and its number of televisions
per person?
Rossman
Ask Good Questions (ICOTS-8)
11
Example 1: TVs and life expectancy




Does the scatterplot reveal an association?
Which direction? How strong? Linear?
Based on these data, would you conclude
that sending TVs to Haiti would cause
Haitians to start living longer?
Identify a confounding variable that explains
the association
Does a strong association between variables
imply a cause/effect relationship?
Rossman
Ask Good Questions (ICOTS-8)
12
Example 2: Reading cancer pamphlets

Are pamphlets containing information for
cancer patients written at an appropriate level
that cancer patients can understand?
Rossman
Ask Good Questions (ICOTS-8)
13
Example 2: Reading cancer pamphlets




Explain why the mean cannot be calculated
for patients’ reading levels
Determine median reading level of patients,
median readability level of pamphlets
Are the pamphlets’ readability levels wellmatched to the patients’ reading levels?
What proportion of patients have reading
level below that of simplest pamphlet?
Rossman
Ask Good Questions (ICOTS-8)
14
Example 2: Reading cancer pamphlets
0.3
0.25
pat ient s
pamphlets
proportion
0.2
0.15
0.1
0.05
above 12
12
11
10
9
8
7
6
5
4
3
under 3
0
level
Rossman
Ask Good Questions (ICOTS-8)
15
Example 3: Sleep deprivation

Does sleep deprivation have harmful effects
on cognitive functioning three days later?
21 subjects; random assignment
sleep condition


deprived
unrestricted
-16
-8
0
8
16
24
improvement
32
40
Identify type of study, observational units,
explanatory and response variables
Rossman
Ask Good Questions (ICOTS-8)
16
Example 3: Sleep deprivation

Students use simulation (tactile, then
computer-based) to investigate core question
of statistical inference:

Rossman
Is such an extreme difference unlikely to occur by
chance (random assignment) alone (if there were
no treatment effect)?
Ask Good Questions (ICOTS-8)
17
Example 3: Sleep deprivation

Summarize conclusion, and explain
reasoning process that supports your
conclusion
Rossman
Ask Good Questions (ICOTS-8)
18
Example 4: Which tire?

Legendary campus story


Which tire would you pick?
Research question: Do people tend to pick
right front tire more often than random
chance?

Rossman
Again, students investigate the question of how
surprising the observed class result would be
under the null model with ¼ probability
Ask Good Questions (ICOTS-8)
19
Example 4: Which tire?

What if 32% of a random sample selects right
front? Is this a significant/convincing result?




Or, what additional information would you need?
Make prediction for how p-value will change (if at
all) as sample size increases
Calculate p-value for many different sample sizes
Summarize how sample size affects p-value,
strength of evidence

Rossman
Explain why this makes intuitive sense (in hindsight, if
not in foresight)
Ask Good Questions (ICOTS-8)
20
Five examples (assessment items)
Rossman
Ask Good Questions (ICOTS-8)
21
Joan Garfield, Beth Chance


Because students learn
to value what they
know they will be tested
on, we should assess
what we value.” - JG
“The number one
mantra to remember
when designing
assessment
instruments is: Assess
what you value.” - BC
Rossman
Ask Good Questions (ICOTS-8)
22
Assessment example 1
For each of the following quantities, indicate whether it can
NEVER be negative or can SOMETIMES be negative.
a) Sample size
c) Standard deviation
e) Difference in sample means
g) Total sum of squares
i) Coefficient of determination
k) ANOVA F-test statistic
Rossman
b) Sample proportion
d) Inter-quartile range
f) Odds ratio
h) Slope coefficient
j) Correlation coefficient
l) p-value
Ask Good Questions (ICOTS-8)
23
Assessment example 2
The U.S. has slightly more than 300 million
residents. In order to estimate the proportion of
U.S. residents who have a facebook account to
within about 3 percentage points with 95%
confidence, about how many people should be
randomly sampled?
100
100,000
Rossman
1000
1,000,000
Ask Good Questions (ICOTS-8)
10,000
10,000,000
24
Assessment example 3
Are people more likely to lie with email than with penciland-paper communication? A recent study involved 48
graduate students in business at a particular university
who participated in a “bargaining” game. Researchers
kept track of whether the student lied about the amount
of money involved when negotiating with another player.
Some of the participants were randomly assigned to use
email for their communication, while others used paperand-pencil. It turned out that 24 of 26 who used email
were guilty of lying about the amount of money involved,
compared to 14 of 22 who used paper-and-pencil.
Rossman
Ask Good Questions (ICOTS-8)
25
Assessment example 3 (cont.)
Use a simulation analysis to investigate whether
these data provide strong evidence in support of
the research conjecture in the first sentence
above. Summarize the conclusions that you
draw from your analysis. Be sure to address
issues of statistical significance, causation, and
generalizability. Also explain the reasoning
process and justification for your conclusions.
Rossman
Ask Good Questions (ICOTS-8)
26
Assessment example 4
State a research question, and describe a data
collection plan to address it, for which a twoproportion z-test would be the relevant
inference procedure. Identify the explanatory
and response variables, and also state the
relevant hypotheses, defining the parameter
values appropriately. Be sure to indicate
whether the data collection plan involves
random sampling or random assignment.
Rossman
Ask Good Questions (ICOTS-8)
27
Assessment example 5 (from 2009 AP
Statistics exam)


Consider the statistic mean / median. What
values of this statistic might indicate that the
population distribution is skewed to the right?
Consider simulation results for values of
mean / median, based on a normal
population, and also the observed value of
mean / median for given sample data. Do the
simulation results suggest that the underlying
population is skewed to the right? Explain.
Rossman
Ask Good Questions (ICOTS-8)
28
Assessment example 5 (from AP
Statistics, cont.)

Use only statistics in the five-number
summary (min, Q1, median, Q3, max) to
propose your own measure of skewness.
Indicate values of this statistic that would
suggest skewness to the right. Explain.
Rossman
Ask Good Questions (ICOTS-8)
29
Oh, by the way ….

Notice that this talk is trying to model
question-based pedagogy

Rossman
From these specific questions/examples we can
extract principles of “good” questions
Ask Good Questions (ICOTS-8)
30
So, what makes a question “good”?

Makes students think



Engages students’ interest



Goes beyond their starting point
Challenges without overwhelming students
To put forth effort to learn
To care about the course material being studied
Addresses important ideas

Rossman
Indicates to students what’s valued
Ask Good Questions (ICOTS-8)
31
What else makes a question “good”?

Fits within logical progression




Builds on students’ knowledge
Asks students to make/check predictions
Confronts students’ misconceptions
Inspires students to ask their own questions


Rossman
About course material
About “real-world” phenomena that can be
investigated using intellectual skills being learned
Ask Good Questions (ICOTS-8)
32
One vehicle for students posing and
investigating own questions

Students use gapminder software (Hans Rosling) to



Rossman
Pose research questions about the world
Investigate those questions with animated graphics
Write report of their findings
Ask Good Questions (ICOTS-8)
33
Some questions from students’ projects








Are students more likely to recycle water bottles depending on the
proximity of recycle bin and waste basket?
Can people better answer math problems if they are presented with
Roman letter notation as opposed to Greek letter notation?
Are people who walk into a clothing store more likely to purchase
something when the weather is rainy as opposed to sunny?
Is balsa wood less elastic after it has been immersed in water?
Are students’ reaction times affected by whether they’ve just
completed exercising?
Are faculty more likely to drive a foreign car than students?
Do guests at a dinner respond more positively to spaghetti sauce
that has been sweetened or not sweetened?
Do college students spend more money at a local fast food
restaurant if they are under the influence of alcohol?
Rossman
Ask Good Questions (ICOTS-8)
34
More unsolicited advice

ICOTS: Teaching Statistics


Any advice for teachers of younger students?


Ask Good Questions
Any advice for teachers of graduate students?


My experience at introductory tertiary level
Ask Good Questions
Any advice for statistics education
researchers?

Rossman
Ask Good Questions
Ask Good Questions (ICOTS-8)
35
Examples of interest to me

Does presenting activities/examples based
on real data from genuine studies





Improve student learning of statistical ideas?
Increase student interest in statistics?
Enhance transferability of skills?
Do students learn more if tactile simulations
are presented before technology ones?
Does it matter whether students construct or
simply consume simulation results?
Rossman
Ask Good Questions (ICOTS-8)
36
Some final questions for you

Have you ever attended a presentation titled



Ask Bad Questions, or
Don’t Ask Questions?
Of course not! So, …

Rossman
Why am I wasting your time offering such obvious
advice as “Ask Good Questions”??
Ask Good Questions (ICOTS-8)
37
Yeah, why am I wasting your time?

I respectfully suggest that the next time you
or I feel compelled to invest some of our
valuable time, thought, energy, and creativity
on …




Rossman
Preparing crystal-clear lectures
Writing lucid paragraphs of exposition
Developing software illustrations
Crafting beautiful presentations
Ask Good Questions (ICOTS-8)
38
What should we do?
Rossman
Ask Good Questions (ICOTS-8)
39
What should we do?

Resist this temptation!
Rossman
Ask Good Questions (ICOTS-8)
40
So, what do I suggest instead?

Instead we should invest these precious
commodities (time, thought, energy,
creativity) on





Developing engaging classroom activities
Preparing thought-provoking assignments
Writing lab or project expectations
Crafting effective assessment items
In other words, we should focus more of our
attention on making sure that we …
Rossman
Ask Good Questions (ICOTS-8)
41
What’s my point again?
Ask Good Questions!
Rossman
Ask Good Questions (ICOTS-8)
42
Thanks very much!
arossman@calpoly.edu
 http://statweb.calpoly.edu/arossman/
AskGoodQuestionsICOTS.ppt

Rossman
Ask Good Questions (ICOTS-8)
43