Active Learning
with Clickers and
Classroom Voting
Kelly Cline
Carroll College
Helena, Montana
Outline:
1. What is classroom voting?
2. What does the research say?
Conceptual questions and discussions are the key!
3. Classroom voting at Carroll College since 2004
4. How do we use classroom voting?
5. Past voting results: A gold mine of information!
6. Classifying Questions by past results:
1. Quick Check Questions
2. Discussion Questions
3. Misconception Magnets
4. Checkpoint Questions
7. A call for collaborators: Please try out our questions
and send us your voting results!
What is classroom voting?
1. The instructor presents a multiple-choice questions
to the class.
2. Students are given a few minutes to form an
opinion and to discuss the question with their
neighbors.
3. Each student votes on the answer thus requiring
every student to form an opinion.
4. The instructor leads a class-wide discussion, asking
individual students to explain and defend their vote.
Voting has often been run with clickers…
Newer technologies are developing…
• http://www.polleverywhere.com
Students register votes with phones, tablets, web
browsers. Votes are displayed on a web page.
(Private company with fees)
• http://ripple-core.uoregon.edu/
Students log in with their phones and tablets.
(Free and open source.)
After voting is closed, the instructor
displays a graph of the results, and
leads a class-wide discussion.
Results of Classroom Voting:
Students Get Engaged
• Each student must take an active role, trying out a
mathematical idea, and discussing it with their peers.
Voting breaks them out of passive-lecture-mode, and
gets them thinking and participating.
• In general, active learning methods tend to produce
deeper conceptual knowledge, which is retained longer.
• Students have fun! They enjoy the class far more,
because they are actively involved. Many report that
attendance improves and attrition drops.
Peer Instruction: 2 Cycles of Voting
One important style of voting is called “Peer Instruction”
originally developed for physics by Eric Mazur at
Harvard.
• First the question is posed, and the students must consider
and vote on the answer individually: No discussion is
allowed. The voting is closed, but the results are not
displayed.
• Then the students discuss their answer with their
neighbors, and they get to vote again.
• After the second round of voting, the results are
displayed, and the instructor guides a discussion.
The Education Research
Teaching with clickers and classroom voting has been
studied extensively at the college/university level, in a wide
range of disciplines, across many types of institutions.
Here’s a quick sampler of a few key studies…
In Physics:
Test Scores
Improve
In one recent study at the University of British Columbia,
two sections of physics enrolling 267 and 271 students,
were taught by two experienced instructors until week 12.
At this point one section was taught by pair of novice
instructors using clickers and voting. An exam was given to
both sections. The test class performed dramatically better,
with a mean 2.5 standard deviations above the control.
Deslauriers et al. Science. 332 862-864 (2011).
In Physics:
Attrition drops
substantially
In introductory physics, it was found that sections taught
with peer instruction had substantially fewer students drop
the course, both at Harvard and at a 2-year college.
Lasry et al. Am. J. Phys. 76 1066-1069 (2008)
In Biology: Students learn from the
small group discussions.
In undergraduate genetics, the
instructor immediately followed up a
2-cycle vote on one question with a
second question on the same topic.
No graphs were displayed, there was
no class-wide discussion, and the
instructor did not teach between the
two questions. Yet, student scores
improved, particularly on difficult
questions.
Smith et al. Science 323 122-124 (2009).
More Results
• When students perceive voting as being used primarily
for the teacher’s benefit (for attendance, to quickly grade
quizzes), then they are more likely to resent clickers.
When students perceive voting as being used primarily
for their own benefit, they enjoy and appreciate clickers.
(Trees & Jackson 2009)
• Students stay more alert and attentive to the lesson when
clickers are used, especially when clicker questions are
scattered throughout the period. They pay more
attention because they never know when a question may
come up! (Hoekstra 2008)
In Math: How do students react?
Post-course surveys were given to 513 students in 26
courses, taught by 14 instructors at 10 different
institutions, all of which used voting and discussion as a
primary teaching method. (Avg. class size = 20)
•93% said that voting makes the class more fun.
•90% said that voting helps them engage in the material.
•84% said that voting helps them learn.
•77% said they would choose a voting section of a
mathematics class over a non-voting section.
Zullo et al. MAA Notes: Teaching Mathematics with
Classroom Voting. 2011.
In Math: The Cornell Study
To try and learn if classroom voting makes a difference,
and if so, what types of voting work best, a study was
conducted at Cornell:
330 students, in 17 sections of Calculus I, taught by 14
different instructors, who were each free to choose if and
how they wanted to use classroom voting.
Common exams were given to all sections.
Miller, Sanatana-Vega, & Terrell
PRIMUS, volume 16, #3
How was voting used?
Classes were grouped into 4 categories:
•Deep: Classes asked lots of deep and probing conceptual
questions (1-4 times per week) with 2-cycle voting,
including regular pre-vote peer discussions.
•Heavy Plus Peer: Voting used 3-4 times per week with 2cycle voting and regular pre-vote peer discussions.
•Heavy Low Peer: Voting used 3-4 times per week but
usually one vote/question with no pre-vote peer discussions.
•Light to Nil: Voting used 1-2 times per week or not at all.
The Results
The final exam was out of 150 points, with 130 being an A,
and 116 being a B-.
Classes which asked lots of deep and probing conceptual
questions to stimulate peer discussions saw the best results,
with statistically significant differences. Voting without
discussion had no significant effect.
Who benefited?
• All demographic groups showed statistically significant
differences on the final exam between classes with
voting and pre-vote peer discussion and those without
the discussions.
• The most dramatic difference was on the score of under
represented minorities.
• With voting and discussion: Mean score = 117, B-.
• Without: Mean score = 106, C.
Classroom Voting at Carroll College
•
•
•
•
•
•
•
•
A liberal arts college in Helena, Montana,1400 students
Math class sizes are usually 15 to 35 students
Graduate 8 to 12 math majors each year
Began voting in Fall 2004, in calculus, using questions
from the Cornell GoodQuestions project, a collection
from Mark Schlatter (Centenary College), and others.
Very positive student reactions
Students actively worked to figure out key ideas during
class through discussion: What more could we want?
NSF Funded: Project MathQuest (2006-2009)
NSF Funded: Project MathVote (2010-2013)
Discussions are the key!
• There are many different things you can do with clickers.
The research and our experience indicate that deep
questions and small group discussions produce excellent
results.
• Learning to “talk math” means learning to “think math.”
Students learn to logically defend mathematical ideas!
• We don’t give points or penalties for right or wrong
answers. In our small classes, we don’t even have to give
participation points. The purpose of the questions is to
teach, not to assess and evaluate.
How do we use voting?
On the first day:
• Explain to the class that we’ll be voting on multiplechoice questions several times in most class periods.
• We emphasize that voting is for the students benefit.
They will learn math better through voting and
discussion than they would just by passively watching
the teacher do examples on the board.
• Clearly state a set of rules for voting…
Rules for Voting
1. Everyone must vote.
2. No one is allowed to vote until they have discussed
the question with at least one other person.
3. After the vote several students will be called on to
explain their votes.
4. It doesn’t matter whether you vote right or wrong, as
long as you can explain your thinking. The only
unacceptable response is “I just guessed.”
Let the votes trickle in…
It is hard to predict how much time the class will need with
any given question.
Tell the students to vote when they’re ready, after some
discussion.
The software counts up votes as they are received. When
about 60 – 75% of the class has voted, we call for the rest
to finish up.
Make a headcount, and require everyone to click in before
closing the vote.
Conducting the Post-Vote
Class-Wide Discussion
1. Ask a student: “What did you vote for, and why?”
2. Initially, give no feedback as to whether this was
right or wrong, and ask the same question of another
student, encouraging the students to figure out the
correct answer for themselves.
3. If students express contradictory arguments, ask them
to respond to each other.
4. If possible, confirm the correct answer after most
have already figured it out.
Getting student buy-in
• We vote several questions in almost every class
period. Questions are usually interspersed throughout
the class.
• Students must see that questions form a vital part of
the lesson, rather than being an optional add-on to be
squeezed in if time permits.
• Students must see that essential ideas are taught
through the questions and resulting discussion.
• The questions must be directly relevant and help
prepare students for the assignments and tests which
follow.
http://mathquest.carroll.edu
• Approximately 2,300 multiple-choice questions
designed for classroom voting in university
mathematics
• Questions are freely available for download in pdf
format, either in:
• Large font, for cut/pasting into PowerPoint
• Small font, for printing & handing out a
numbered list to students
• Teacher’s edition and LaTeX files available with an
e-mail request to me: kcline@carroll.edu
Subjects and Numbers
Algebra: 200 questions
Statistics: 107 questions
Precalculus: 226 questions
Differential Calculus: 192 questions
Integral Calculus: 151 questions
Multivariable Calculus: 317 questions
Linear Algebra: 311 questions
Differential Equations: 350 questions
And more!
Topics and Coverage
Most of our collections are not closely tied to specific
textbooks. Topics are general. For example in linear
algebra we begin with:
•Systems of Equations
•Matrix Representations of Systems of Equations
•Gaussian Elimination
•Solution Sets of Linear Systems
•Linear Combinations
•Linear Independence
•Rank of a Matrix
•Matrix Operations
•Matrix Inverses
•Determinants
•Eigenvalues and Eigenvectors
The Student Edition
• Freely available for download by anyone
• Questions numbered sequentially:
The Teacher’s Edition
• Freely available with an e-mail request
kcline@carroll.edu
• Question numbers match the student edition
• The correct answer or answers are indicated
• A solution/discussion is usually presented
• Authorship is identified (please share your
questions and allow us to include them!)
• Past voting statistics are presented
• Often pre-vote times are recorded as well (min:sec)
Selecting the Right
Questions is Essential!
Voting requires a substantial investment of class time,
often about 3 to 5 minutes per question, and we vote
several times in each class period.
1. Each question must be directly relevant and teach an
important idea, thus replacing lecture-time.
2. Students must see how the voting questions help them
work through the assignments that follow.
3. Research indicates that the most effective questions
provoke small-group discussions about important
issues.
That’s a tall order!
How do you find the right questions?
• Clickers by themselves are not magic: Using clickers in
your course can dramatically change things for the
better, or be a colossal waste of time!
• It’s all about the questions that you use.
• Creating really good multiple-choice questions is hard!
• It’s rarely obvious whether or not a question is worth
investing the time to make a vote.
• What types of questions will be likely to produce good
discussions, both small-group discussions before each
vote and class-wide discussions after each vote?
• How can we identify the best questions?
Our Innovation:
Record Voting Results
• We began informally recording the percentage of
the class voting for each option back in the fall of
2004, just so we could compare how two parallel
sections voted on the same questions.
• We realized that this could be useful and interesting
information, and it’s not difficult to record.
• We began writing these down on our teacher’s
edition every time a question was voted, then
adding these to the LaTeX files of our teacher’s
edition.
• Friends at other institutions began contributing…
Past voting results are useful
for lesson planning
Our collection has 10 or 20 questions available for many
topics, and I have time to vote perhaps 3 to 8 questions in
a 50 minute class.
Using records of past voting results, I can quickly identify
the questions that are more likely to challenge the
students and provoke serious discussions.
Every class is different, and there are no guarantees, but
past results are a very useful guide!
A Research Project:
Study Past Voting Results
• There are many questions for which we now have
records of five, ten or more votes.
• Together, these votes can give us insights into
student thinking, and common challenges.
• That’s interesting whether you use clickers or not!
• Many questions produce very different responses
from the different classes taught by different
instructors at different institutions. But some
questions produce remarkably similar votes.
Identifying Types of Questions
Quick Check: A strong majority regularly votes
correctly.
Discussion Questions: Votes are widely distributed
over different answers. We look for the smallest
average “winner.”
Misconception Magnets: A strong majority regularly
votes for a particular incorrect answer.
Checkpoint Questions: The percent voting correctly
varies widely between classes. Look for the largest
STD among the percent voting correctly.
Voting Patterns
• We have 192 questions on differential calculus.
• Each point represents one of the 101 questions for
which we have at least 5 votes.
Quick Check Questions
• In the lower right, we see lots of questions where most
students vote correctly most of the time.
• Straight-forward practice with a new ideas
A Quick-Check on the
Product Rule
• It’s better for students to do the examples themselves,
rather than watching us do them on the board.
• But Quick-Check questions rarely produce good
discussions…
Identifying Discussion Questions
• Good discussions produce great learning benefits.
• If most of the students agree with each other (rightly
or wrongly), small groups will have less to talk about.
• If substantial numbers of students vote for several
different options, small groups are more likely to have
contrasting opinions, and thus something to discuss!
• We look for questions with the smallest average
winner, the option (a,b,c,d) getting the most votes.
Good Discussion Questions are
Not Usually Obvious!
• Often I have written a question, expecting it to provoke
a good discussion, but the issue is too obvious (or too
subtle) and falls flat.
• Other times, a question that appears straight-forward to
me brings up a wide variety of important issues,
stimulating an excellent discussion.
• Past votes are no guarantee – every class is different! –
but they can be a very useful guide.
Second Derivatives
We have spent most of the class period developing an
understanding of the meaning contained in a second
derivative function. We have answered questions
relating the second derivative to concavity, we have
matched graphs of functions with graphs of the second
derivative, and we have discussed how if the original
function gives position as a function of time, then the
second derivative gives acceleration.
A Great Question:
In Star Trek: First Contact, Worf almost gets knocked
into space by the Borg. Assume he was knocked into
space and his space suit was equipped with thrusters.
Worf fires his thrusters for 1 second, which produces a
constant acceleration in the positive direction. In the next
second he turns off his thrusters. In the third second he
fires his thruster producing a constant negative
acceleration. The acceleration as a function of time is
given in Figure 2.31. Which of the following graphs
represent his position as a function of time?
(ConcepTests for Calculus by Hughes-Hallett et al.)
The Graphs:
We have about a dozen questions on this topic, but I
would never miss asking this one: This is a gem!
Small Avg. Winner: Good Discussions
Class 1
Class 2
Class 3
Class 4
Class 5
Class 6
Avg.
(a)
45%
13%
40%
48%
35%
23%
34%
(b)
0%
0%
5%
7%
0%
0%
2%
(c)
35%
48%
40%
7%
41%
54%
38%
(d)
20%
39%
15%
37%
18%
23%
25%
Winner
45%
48%
40%
48%
41%
54%
46%
Here, we calculate that for this question, the average
winner gets only 46% of the vote: A very small number!
This method usually picks out great discussion questions.
Misconception Magnets
• These are questions where a large majority regularly
vote for a specific wrong answer. The pre-vote small
group discussions may be brief, but the post-vote
discussions can be interesting: Students expect the
majority to be correct, and are surprised and intrigued
to find out that they were wrong!
• Sometimes small group discussions can be created by
simply telling the class that the majority is wrong and
calling for another round of voting.
Misconception Magnets
• In the lower left, we see a few questions where a strong
majority of students regularly vote incorrectly.
A Misconception
Magnet
A Misconception
Magnet
Most students apply the product rule and get
which is not one of the options, so
they vote for (e). But (a), (b), and (c) are all equivalent
to this, and so (d) is the correct answer. A few students
object that their calculators give them (a) or (b)…
Results from
this question.
We identified this question by calculating the average
percentage voting for the most population incorrect option:
Here this is 57%, the largest of the 101 questions. This is
an awesome question for post-vote discussions.
A Misconception Magnet
From Early Calculus II
We have computed a few left-sums and right-sums, and
then tried averaging them, seeing a few examples where
this is more accurate.
True or False: Averaging left- and right-sums always
improves your estimate.
(GoodQuestions)
Our Statistics (T/F): 70/30, 90/10, 90/10, 100/0
The results are quite devastating! A great follow up at
this point is to have the students work with their peers to
create sketches of counterexamples.
In Differential Equations:
Continuity is a rich topic…
We know that a given differential equation is in the form
y’(t) = f (y) where f is a continuous function of y. Suppose
that f (5) = 2 and f (–1) = –6.
(a) y must have an equilibrium value between y = 5
and y = –1.
(b) y must have an equilibrium value between y = 2
and y = –6.
(c) This does not necessarily indicate that any
equilibrium value exists.
(a) Because f is continuous, this means that if f goes from
positive to negative, then there must be a zero, an
equilibrium value between them.
Worse is Better!
Results:
Class 1 33/10/57
Class 2 57/43/0
Class 3 26/15/59
Class 4 0/0/100
Class 5 50/17/33
Class 6 41/0/59
time 1:45
time 2:50
time 2:20
time 2:45
There are two levels to this question, interpreting the notation, then
applying the idea of continuity. Both (a) and (b) use the word
“must” and students quickly learn to be cautious about such words
in mathematics. Good!
Checkpoint Questions
• Clickers produce an instant snapshot as to the state of
my class, with results from every student.
• Do they understand, or do we need more discussion?
• Reality check: A few bright and vocal students can
easily persuade an instructor that a topic has been
mastered, while most of the class is still lost!
• We can identify these from past voting statistics by
looking for large standard deviations in the percent
voting correctly.
Checkpoint Questions
• At the top, we see questions where different classes
voted very differently.
From Calculus I: Optimization
This is false because a local maximum can also occur at a
point where the derivative is undefined. Students either
recall this, or they do not: a focused question, a key point!
Taking derivatives of logarithms
We have a large standard deviation in the number voting
correctly because of one section where no one voted
correctly. What happened?
Questions can be used in
different ways
In most classes, this was used as a quick-check, a simple
question to practice a newly introduced idea.
In the other class, the question was used at the very
beginning of the period, before the natural log rule had
been presented, with the expectation that students should
have read the text.
• Students referred to their text, worked together, and
settled for options (c) and (d).
• No one voted correctly, but the discussion and followup was great! They worked out most of the problem
by themselves!
During Antiderivatives
True or False: If two solutions of a differential equation
dy/ = f (x) have different values at x = 3 then they have
dx
different values at every x.
(GoodQuestions)
Our Statistics (T/F): 90/10, 10/90, 100/0, 80/20, 85/15
In most classes a large majority of students get this one
right, except one class where only 10% got it right. The
statistics immediately identify this as a big warning flag:
What’s going wrong? What are they missing? What did
we do different this time? Some crucial concept has not
been mastered! This question is worth asking!
Please join us!
• Please use our questions and record the results of your
votes, so that we can put them into the teacher’s edition.
• Past voting statistics are very useful in lesson planning.
They also provide fascinating insight into common
student errors and issues.
• We have already written several papers analyzing these
results. We plan on writing lots more. If we write a
paper including your data, we will be happy to include
you as a co-author.
Pedagogy is the key,
& clickers make it possible
The best results come from a particular teaching
technique, not from a particular technology. Clickers can
be used in ways that do not engage students in the
material and do not involve them in discussions.
The clickers are fun, and the students enjoy registering
their votes, but they are really just the “magic feather”
the “spoonful of sugar that helps the medicine go down!”
Give Clickers and Voting a Try!
• You get immediate feedback.
• Students actively engage in the classroom. It keeps
them involved, they have fun, & attendance
improves.
• Use voting to replace parts of your lecture, drawing
out ideas that you would normally explicitly present.
• Emphasize that the pre-vote discussions are the real
engine of this process. When your students are
discussing and debating mathematical ideas with
their peers, then real learning is taking place!
Our Overall Results
• The students love it. They tell us how much fun
they’re having, and complain when we have a nonvoting day.
• When asked if they had to choose between a class
with voting and one without, a strong majority
would choose the one with voting!
• Attendance is outstanding. When students are
involved in the class, they are more likely to come.
• Our anecdotal testimony: Our students are learning,
they’re having fun, and we’re attracting more math
majors and minors.
MAA Notes: Teaching
Mathematics with Classroom Voting
• Released September 2011 as an e-book
• Edited by Kelly Cline & Holly Zullo
• 24 chapters by authors at a wide range of
institutions discussing classroom voting
in:
• College Algebra
• Calculus
• Statistics
• Linear Algebra
• Differential Equations
• Abstract Algebra
• And more!
Thank you for your attention!
Any questions?
Looking for Misconception Magnets
We go through the data set, looking for questions where
on average the largest percent of students voted for the
most popular incorrect answer.
The context:
After studying double integrals, we have introduced
triple integrals and how to calculate them. Now, we want
to get students to think about what triple integrals might
mean…
A simple triple integral
The Results
On average, 57% of students vote (a), the most popular
incorrect option. As a result this question ranks highly
as a misconception magnet, and our experience confirms
that this is a great question. It regularly produces some
excellent discussions!
From Linear Algebra
We discussed systems of two equations and two
unknowns, learning to solve them algebraically,
graphically, and then by putting an augmented matrix
into reduced row echelon form.
Then rather than explicitly teaching what can happen
with larger systems, I got the students thinking about
these issues with the following question.
A Linear Algebra Question
a)
b)
c)
d)
e)
f)
A system of 3 linear equations and 3 variables could not
have exactly _____ solutions.
1
2
3
Infinite
More than one of these is impossible
All of these are possible numbers of solutions
Results indicate a good discussion question:
The Resulting Discussion
Only a third of the class initially voted for the correct
answer, (e) that this system could not have exactly two or
three solutions.
Some students drew diagrams and tried to explain things
in terms of lines and intersections in two dimensions,
which didn’t work. To help, I introduced the idea that a
linear equation with three variables can be represented as
a plane in three dimensions.
The discussion came to a resolution when a student talked
about representing this with an augmented matrix and
asked what two or three solutions would look like when
the matrix was put in reduced row echelon form.
A Follow-up Question
A system of 5 linear equations and 7 variables could
not have exactly ______ solutions.
a) 0
b) 1
c) infinite
d) More than one of these is impossible.
e) All of these are possible numbers of solutions.
Students focused more quickly on what the augmented
matrix would look like when put in r.r.e.f. Again, the
discussion was very fruitful, bringing up lots of key ideas.