GROUP ACTIVITY
30 minutes discussion time and 15 minutes recording.
Aim
In your group, focus on an element of question/ voting activity design that interests you,
and that you can share with the wider community.
Outcome
By the end of the session, we hope you will have come up with some advice/guidelines, a
model or exemplar that would be of help to others considering the same aspect of
design.
Capturing your ideas
We would like you to capture this in such a way that it can be made available online to
the community. The contributions from the various groups might start to form the basis
of an online resource.
To capture your ideas, you could:




use a flip cam to record a video of a team member explaining the best practice,
capture the outcomes on paper or type the advice guidance into a Word
document, or directly into the ESTICT site. For example, this might be in the
form of a concept map or checklist.
Create a ‘screencast’(if your group has the appropriate technology). In other
words, capture the display on the screen together with someone talking about
the ideas.
Create some example good or bad questions or activities in PowerPoint with
explanations.
You only have 15 minutes, so keep it short and clear please.
Feedback
We will randomly pick one group to present back or show their film clip after coffee. All
the videos and documents produced by the community will be shared online.
Resources

Each group has a facilitator who will ensure that the group produces something
by the end of the session.

This handout includes Bruff’s Taxonomy of Voting Questions, some example
question design mistakes and some models of question and activity design to
draw on. If you have access to the internet, please make use of that.
1
RESOURCES
Bruff’s Taxonomy Bruff, Derek. Teaching with Classroom Response Systems: Creating
Active Learning Environments. 1st ed. Jossey-Bass, 2009.
Content question Focus on course content.
They range from recall questions which test at the lower end of Bloom’s taxonomy of learning
objectives, to critical thinking questions which encourage higher-level thinking skills.
Sub-type
What do they do?
Example
Recall
Test memory only, not
understanding. Don’t generate
discussion. Good for summoning
prior knowledge / establishing
a baseline in preparation for new
learning. The ‘easy’ questions –
guessable MCQs, can be helpful for
struggling students.
Which is a metalloid: C, Al, Zn, Te or Na ?
Conceptual
Understanding
Tests recall of facts and the
concepts associated with those
facts. Not easy to guess. Can
reveal misconceptions which
when explained through can
enhance learning. Promote
“enduring understanding” /
deep learning
To hold the ball in this position independently, each muscle
must provide the same… force, torque, or moment arm?
Application
Encourage students to apply their
knowledge and understanding to
particular situations and contexts.
Encourage integrative learning.
Based on the facts of problem 7 [in the students’ textbook],
in the lawsuit by the student against Mountain Law
School, a court will likely find favour of the:
Vital for disciplines like medicine.
A. student, if the court finds that the terms of the
catalogue are complete, definite and certain.
B. Student, since catalogues are usually considered
ads, and ads are always offers.
C. Law school, since catalogues can never include all
the necessary terms to be deemed definite and
complete offers.
D. Law school, since the student could not have
expected to be taught all the terms included in the
catalogue
2
Bruff’s taxonomy: Content questions continued
Sub-type
Critical
Thinking
Free Response
What do they do?
Examples
Requires students to analyse
relationships among multiple
concepts or make evaluations
based on particular criteria. Often
questions asked have no single
correct answer (although some
answers may be better than others)
- this challenges black-and-white
view of world which some new
students have. Emphasis should be
on reasoning behind selecting a
choice of answer, not the answer
itself. To examine reasoning, such
questions are best followed up with
class-wide discussion.
1. Several questions which link one after the
other in order to test a series of cognitive
processes (remembering, understanding,
applying, analysing and evaluating)
Students create their own answer
to a question (this is difficult /
impossible to do with MCQs). Used a
great deal in numerical and
quantitative disciplines. A drawback
is that answers are difficult to
aggregate / make sense of as
compared with traditional MCQbased EVS questions.
Ask students to brainstorm before they start to solve
a complex problem by submitting their ideas via
EVS.
2. Questions where students author some/all
of the possible answers, vote and lecturer
leads discussion about the question,
surfacing reasons for answers given
3. Peer assessment contexts
3
Bruff’s Taxonomy: Process Questions
Process questions gather information from students helpful in shaping how students
interact with each other and the course material.
Sub-type
What do they do?
Example
Student
Perspective
Gauge student perspective in terms of
their demographics, opinions and
personal experience. Good ‘getting to
know you’ questions that avoid
assumptions being made and allow
teaching to be adapted according to a
given student profile. Also allow
students to get to know each other.
Encourages students to reflect on
personal beliefs which can contextualise
teaching content and enhance learning
around it. Good at generating discussion
since students beliefs not all the same.
Which of the following statements most
closely matches what you think?
Confidence level
Monitoring
A. Humans evolved from other life forms
with divine assistance
B. Humans evolved from other life forms
without divine assistance
C. Humans were created by a divine
being within the past 10,000 years
These allow you to drill down into finding When your students have voted on a given
out how well students really knew the
question, before revealing correct answer, ask
answer to a previous question, and how
them ‘how confident are you in your answer to
many were just guessing. You can then
the previous question? – very confident,
adapt your teaching according to what your somewhat confident or not confident at all?’. It
students really know. Encourages
is recommended you score your students based
students to reflect on their own
on their level of confidence and accuracy in
learning.
answering so that they are encouraged to rate
their confidence accurately.
Monitor other aspects of student
learning experience. Typically used for
course evaluations but could be used
throughout.
Ask students to indicate their progress with a
particular assignment: ‘Have you written the
first draft of your paper?’ or ‘How long did it
take you to complete your first assignment’
(students who took longer encouraged to seek
help)
Quiz students on issues of course or
programme they often say they forget: e.g.
penalty for missing deadlines, key aspects
relating to assessment criteria etc
4
Classroom
Experiments
Often used to facilitate social science
experiments in class. Avoids timeconsuming task of data entry.
Many Psychology textbooks describe
classroom activities that only require little
effort for use with clickers. For example, to
illustrate several concepts in the study of
memory, show students a list of words for a
brief time, then take away the word list and
ask them to write down as many words as they
can remember. Then take several words in
sequence and ask students to report using
their clickers whether they remembered each
of the words.
Hendrich’s ABCD Model for Constructing Effective Scenario Questions
Actor/Audience
• Who is in the situation? Who is involved?
Behavior
• What situation are they in? What are they trying or unable to do?
Condition
• Under what circumstances or context is the actor behaving or
hoping to behave?
Dilemma/Decision
• What decision must the actor make? What dilemma is s/he
facing?
•
“Dr. Chang is considering prescribing a blood pressure medication for
Jim, who has a comorbid liver disorder. What class of medications is Dr.
Chang likely to choose?”
Susan Hendrich, Writing effective scenario questions « Susan E. Hendrich. Leadership in
Training Innovation Blog. Available at:
http://susanhendrich.wordpress.com/2008/10/21/writing-effective-scenarioquestions/ [Accessed April 6, 2010].
5
Example EVS learning Sequence : Class wide discussion
Could you refine/develop a sequence of use for a particular purpose like this example
below?
1. Concept question posed.
2. Small group discussion: small groups discuss the concept question (3-5 mins).
3. [voting] Students provide individual or group responses.
4. Students receive feedback -- poll of responses presented as histogram display.
5. Class-wide discussion: students explain their answers and listen to the
explanations of others (facilitated by tutor).
6.
Lecturer summarises and explains "correct" response.
Nicol, David J., and James T. Boyle. “Peer Instruction versus Class-wide Discussion in
Large Classes: a comparison of two interaction methods in the wired classroom.” Studies
in Higher Education 28, no. 4 (2003): 457.
Content free quiz – Design issues (after Phil Race and Roger Lewis See http://philrace.co.uk/?page_id=117 )
The questions below illustrate common mistakes in question design that give away the
answer. Could your group create a resource like this that illustrates good or bad design
practice
Question 1
The usual function of grunge-prowkers is to remove
A grunges
B snarts
C trigs
D grods
Question 2
Antigrottification occurs on
A spring mornings
B summer evenings provided there is no rain after dusk
C autumn afternoons
D winter nights
Question 3
6
Lurkies suffer from trangitis because
A their prads are always underdeveloped
B all their brizes are horizontal
C their curnpieces are usually imperfect
D none of their dringoes can ever adapt
Question 4
Non-responsive frattling is usually found in an
A gringle
B janket
C kloppie
D uckerpod
7
Question 5
Which of the following are exceptions to the law of lompicality?
A the miltrip and the nattercup
B the bifid pantrip
C the common queeter
D the flanged ozzle
Question 6
Which must be present for parbling to take place?
A phlot and runge
B runge
C stuke and runge
D runge and trake
Question 7
One common disorder of an overspragged uckerpod is
A copious vezzling
B intermittent weggerment
C non-responsive frattling
D uneven yerkation
Question 8
Which one of these is the correct answer?
A
B
C
D
8
ANSWERS
Question 1
Most people pick A. This is because the option contains a word from the stem, which is a sort of
give-away. Therefore avoid using words from the stem in one particular option, especially if it is
the correct answer.
Question 2
Most people pick B. This seems to be because it is longer and therefore appears more probable.
Therefore, try to keep all options to the same length, within reason. Also, it is a qualified option
whereas the others are unqualified; a qualified option may seem more likely to be correct.
Question 3
Most people pick C. This is mainly because of the word “usually”, which seems more probable
than always, none or all. It is best not to mix definites and indefinites, although all definites or all
indefinites is fine.
Question 4
Almost everyone picks D. The “an” in the stem gives this away. Many questions like this have
found their way into print – some even into examinations!
Question 5
The most probable option is A. This is given away by the plurality of the stem, and the singularity
of B, C and D.
Question 6
Most people choose B. It’s a bit harder to explain why this is – it seems to be something along the
lines of “if B isn’t right, none of them can be right, as they all involve “runge”. Take care with
overlapping options.
Question 7
This is trickier. However, most people go for C. The main reason is that the topic has already
come up in Question 4 where the indefinite article gave away the answer. So, take care not to set
questions where earlier ones give away the answer.
Question 8
The correct answer is, of course D! Most people would have recognised the pattern 1-A, 2-B, 3-C,
4-D, 5-A and so on. So, D seems ripe for coming up on question 8
9