Statistically Grounded Instructor Guide: Understanding power and p-values
Quick Info
Level: Introductory or Intermediate Undergraduate Statistics Courses
Brief Description: Students investigate a research question by conducting a statistical analysis with data
collected in a simulated computer game.
Topics Covered: Model assumptions and analysis of two-sample Hypothesis Tests, Confidence Intervals.
While this lab has many opportunities to discuss statistical power, it is not a required topic.
Learning Goals: Provide a simple model of a research study in order for students to better understand that pvalues are not meaningful when the null hypothesis is not true.
Software Required: Data analysis software such as Excel or Minitab for descriptive statistics and visualization.
Students will also need computer access to collect the game data on the web (this can be done inside or
outside of the regularly scheduled class time).
Prerequisites: Descriptive Statistics, Histogram or Boxplot, Two-Sample t-test, Confidence Intervals
Time: 1.5 to2.5 hours in class + 1 to 3 hours of homework
Instructor Resources: Student Labs and Instructor Guides, and instructor’s website
(http://web.grinnell.edu/individuals/kuipers/stat2labs/CoffeeTruck.html)
When should you use this lab in your course and what are the prerequisites?
Prior to this lab, students should have a working knowledge of descriptive statistics (mean, standard deviation,
etc.) and visualizing data (histograms). Students should also be able to conduct a two-sample hypothesis test as
well as calculate and interpret the corresponding confidence interval.
How should you conduct the lab? How much time should you expect to allocate?
DAY 1:
 Collect the game data (15 Minutes) – Provide students a hard copy or electronic copy of only the first
two pages the student handout. Walk through the first page of the lab with students, if computers are
available in class, have students collect data in class. Otherwise, assign the data collection as part of the
class homework.
 Complete questions 1) and 2) during class.
Homework Assignment:
Have students complete Questions 3) and 4), as a homework assignment (possibly in small groups). We
recommend that students do these as homework because the time it takes students to complete these
questions varies greatly.
Day 2:
a) Hand out page 3 of the student handout. This page has solutions so do not hand it out earlier.
b) At the beginning of Day 2 of this activity, do the following:
 Make a table on the chalk board with three columns: Business District mean, Park mean, p-value,
and their 95% confidence interval. While the mean for business tends to be higher than Park,
students should get vastly different p-values. (if you have a large class, you can assign groups and
have these values posted on a class website before class)
 Call on students/groups with very different p-values to very briefly discuss their suggestions to Joe.
(If you have a large or on-line class you can ask students to post comments to two to three
Statistically Grounded: Understanding power and p-values
2
student/group results). Discuss Question #5 and have students complete the lab (55 Minutes). If
students do not complete the lab during class they can finish the work as a homework assignment
(possibly in small groups).
c) We suggest showing the "Dance of the p-values" YouTube video (https://www.youtube.com/watch?
v=ez4DgdurRPg) during class and spending time discussing this in class before asking students to
answer this question. This video does a nice job in helping us get a better picture of what power,
type I errors and type 2 errors really mean. Note that p-value only has meaning under the
assumption that the null hypothesis is true.
Day 3:
a) Optional: This is a great place to start the introduction of power calculations.
b) Consider playing the game again with larger sample sizes, say n = 30, to determine how different the
results can be with varying sample sizes.
c) You can also play the game using Business District, No Music, $3 a cup and compare the morning and
afternoon times. This is a case where the underlying model has a mean of 100 cups sold for both
times. Thus the null hypothesis is true in this comparison.
d) Other related readings that are useful for students to read and discuss include:
“Trouble at the lab”
http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correctingalarming-degree-it-not-trouble
"Why Most Published Research Findings Are False" (Ioannidas)
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124
A somewhat more complex article: "Most Published Research Findings Are False—But a Little
Replication Goes a Long Way"
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0040028
Here are some helpful hints for instructors using this lab for the first time.
1. You should try to have at least 10 student groups in order to increase the likelihood of observing
dramatically different p-values.
2. Spend time talking about the definition of a p-value before giving them the student handout. Students
should get an appreciation for the variation involved in conducting experiments. We have found that
even students who can state the definition of a p-value are very surprised by these results.
3. Encouraging students to work in pairs or small groups provides a great opportunity for the students to
talk about their understanding and teach each other.
What else is in this Instructor Guide?
In the next section, we provide detailed comments on the student lab. We suggest questions you can ask to
promote class discussion and point out common issues you may run into when using the lab. For more
information and ideas on using Statistically Grounded in your course, go to
http://web.grinnell.edu/individuals/kuipers/stat2labs/CoffeeTruck.html.
Statistically Grounded: Understanding power and p-values
3
Statistically Grounded: Understanding power and p-values
Which location leads to the most profit in a small business?
Your friend, Joe, just inherited a coffee truck from his uncle. He loves coffee and has decided to start his own
coffee company. He was able to get a loan from his bank, and after painting and repairing the truck, he has
$20,000 left for supplies and daily life expenses.
Joe’s truck carries various items, but for the moment he is focusing his attention on his coffee. He has a fantastic
coffee blend that people really seem to like. He can make it for about $0.50 a cup. In addition, after a careful
business plan, Joe estimates his daily business and life expenses at $300 a day.
Estimating Profits and Loss:
1) Assume Joe sold 150 cups of coffee at $2.00 a cup. Calculate his profit or loss for that day.
2) Assume Joe sold 100 cups of coffee at $4.00 a cup. Calculate his profit or loss for that day.
Play the Statistically Grounded game: Joe knows you are taking a great statistics course and wants to
use your knowledge to help him make key business decisions. There are multiple choices that Joe could make
that may influence how much coffee he sells:
Where should Joe
park his truck?
Some locations
may have more
sales than others.
Does playing a
certain type of
music from your
truck influence
sales?
Charging less may
increase the
number of cups
Joe sells, but how
will it affect
profits?
Each day, Joe will work every time that
you select. During this study, Joe will
only work during one four hour time
frame each day. Weather will be
relatively constant for this game.
The on-line game allows you to design a study to evaluate whether these conditions influence sales. Go to the
web site http://statgames.tietronix.com/statisticallygrounded/.
Check Yes in the Save Data box and type in the appropriate information to record your data:
Name:______________ (This will be on the web, do not use a player name that will identify you. However,
make sure you print your player name on this worksheet.)
Group Name:_______________(Provided by the instructor)
Statistically Grounded: Understanding power and p-values
4
In this study, we are assuming that Joe is primarily interested in determining whether he should sell coffee in
either the Business District or the City Park during the lunch hour. Check both locations as shown in the image
above. In addition, make sure to select No Music, $3.00 per cup and the Lunch Time.
Select the design type, Balanced Random and number of simulations to be 10, as shown below. Balanced
means that both conditions (Business District and City Park) are tested the same number of times and
Random means that they will be tested in random order. In this study, both locations will be tested 10 times,
totaling 20 days for the study.
1) Select Start, then click on the Game Data button. Note, some computers may require you to remove the
pop-up blocker to see the data. Find your data by clicking on the “Download CSV” link corresponding to
your Player Name. Copy and paste your data into an appropriate statistical software package and
create one or two graphs that display your results.
2) Analyze the data. Can you conclude that there is a difference between locations? Be sure to include a
hypothesis test and a 95% confidence interval for the difference between locations (Business District City Park). Write a short report (300 words or less) that provide a suggestion to Joe. Did your study
provide evidence that there was a difference in locations? Tell Joe what he should do. Clearly describe
the study, how the data was analyzed, and what conclusions were drawn. Graphs should be included in
this report.
Statistically Grounded: Understanding power and p-values
3) Compare your study results to others in the class. Did every group in the class come to the same
conclusion about which Location is best? Did every group in the class get a p-value less than 0.05?
4) Create two separate histograms of the class data found in question 3, one histogram of the business
district means and a second histogram of the park means.
5) This activity was designed so that the distribution of cups sold is normally distributed with a standard
deviation of 10. However, the mean for the business district is 83 cups sold while the mean for the park
is 74 cups sold (assuming it is during the lunch time frame, with no music, and priced at $3 a cup). Do
the samples collected in your class appear to fit these assumptions?
6) When conducting a two-sample t-test, the p-value is the probability of obtaining a test statistic at least
as extreme as the one that was actually observed, assuming that the null hypothesis is true. Write a
short paragraph to Joe giving a practical explanation of p-values and alpha levels in the context of this
study. More specifically, explain why the p-values tended to be inconsistent even though the business
district is truly better than the park.
7) Compare the student confidence intervals in question 3. How often was the Business District mean
greater than the City Park mean? Explain why many statisticians suggest that using confidence intervals
is preferable to using p-values.
5
Statistically Grounded: Understanding power and p-values
6
The theoretical sampling distributions for the business and park locations are given below:
𝑋𝐵 ~ N(83,
10
)
√10
and
𝑋𝑃 ~ N(74,
10
)
√10
These are the proper theoretical sampling distributions that the game is based on. If students correctly
collected data, you should expect the sample means to follow these distributions.