Engaging students outside of
class as a tool for
enhanced learning
Dan Bernstein
University of Kansas
UBC Centre for Teaching and
Academic Growth
22 November 2006
djb@ku.edu
Unhappy with answers to questions
• What is the major point of quantum mechanics
that makes prediction impossible? What level of
prediction is affected? What level is not?
• What effect do quantum mechanics have on
scientific objectivity as part of our process of
knowing?
• Describe the Heisenberg uncertainty principle
(briefly). What relevance do you think it has for
contemporary research and clinical practice in
psychology?
• Does quantum theory render Newtonian
mechanics useless to humans in their interactions
with the physical environment? Why or why not?
Simple problem led me onto a
path of continuous inquiry that
has lasted a decade
Started with Experimental Studies
• In search of the best way to generate high
level student answers to those questions
• Compared multi-media presentation with
lecture, reading, and irrelevant information
• Measured rote and complex understanding
with multiple choice, fill-in, short essay
• Advantages of random assignment and
controlled conditions
• Goal ultimately a practical one
Basic Procedures
• Commercial and home-made multi-media
• Used video and live lecture with same
content as multi-media
• Created text samples with same content
• Assessment of levels of understanding
with mixed methods of responding
• Individual instruction with unlimited time
• Participants from introductory psychology
Learning from Lecture
40
35
30
25
20
Lecture
15
10
5
0
<= 69
70-79
80-89
90-100
Learning from Lecture v.
Reading
60
50
40
Lecture
Readings
30
20
10
0
<= 69
70-79
80-89
90-100
Lecture, Reading, and
Computer
60
50
40
Lecture
Readings
Computer
30
20
10
0
<= 69
70-79
80-89
90-100
Percentage Correct by
Instructional Mode and Level of
Understanding
70
60
50
40
Rote
Application
Evaluative*
30
20
10
0
Computer
Lecture
Readings
Percentage Correct by
Instructional Mode and Question Type
80
70
60
50
Multiple-choice*
Fill-in-the-blank
Essay
40
30
20
10
0
Computer
Lecture
Readings
Mean Instructional Time by Instructional Mode
30
25
20
15
Instructional time*
10
5
0
Computer
Lecture
Readings
Upgraded the lecture component
• Live lecture to portion of introductory class
• Topic is reactive measurement and
quantum mechanics
• Matched to locally authored multi-media
and text -- also irrelevant text control
• Motivation sustained by possibility of test
• Assessment layered by type of
understanding and format of question
Distribution of total test performance
resulting from four methods of
instruction
Number of Correct Responses Across
Levels of Understanding
14
12
10
lecture
computer
reading
control
8
6
4
2
0
rote
application
evaluation
Time on task (in minutes)
70
60
50
40
30
20
10
0
Computer
Lecture
Reading
What do you make of these data?
• How would it help
me teach?
• Any ideas for next
innovation?
• Critical reactions to
the evidence?
My take home messages
• Students learning by computer or reading
showed greater and deeper understanding
• Multi-media has potential but very high
cost to get started
• Reading consistently took the least time
• Reading is a good way to learn if you have
a way to assure that it is done
Next problem arose when I
presented my data
• Challenged by Tom Hatch, from Gardner’s
Project Zero group
• Described “Deep Understanding”
– Use ideas in a context not taught
– Ideas retained without memorization
• I presumed that abstract was more general
• Agreed to test the idea in my class
Tracking understanding over time
•
•
•
•
•
•
•
•
•
Learning and motivation
Sophomore level course
20 to 45 students
Taught 9 times in 7 years
Professional grade reading
Hybrid online and F2F
Largely discussion in class
All essay examinations
Conceptual writing required
Wanted Richer Essay Exams
• Why is magazine training so critical to the process of
shaping a new response class? Sometimes the sound of
the operation of the food magazine is called a bridging
stimulus; what might this metaphor mean?
• What is the evolutionary mechanism for the cultural
survival of religious taboos against slaughtering and eating
cattle? You should identify the conditions that make this
evolution most likely.
• Explain the notion of selection by consequences as a
description of the change in distribution of behavior that
takes place during shaping. How is this process different
from selection that takes place under naturally occurring
conditions?
Reasonably Conceptual Items
• Describe the results of [Razran’s] second experiment on
semantically mediated generalization (numbers) done
with a 13-year old boy. Be sure to identify the respondent
behavior, the training stimuli, and the test stimuli. Why is
this example semantically rather than physically
mediated generalization?
• What do chicken scratches, raccoon rubbings and pig
rooting have in common? What stimuli have pre-existing
relations these bits of behavior? What does the
“misbehavior of organisms” have to do with the generality
of principles of learning?
Transform into Problems
• Old question:
– What were the reinforcing consequences in the Welsh,
Bernstein & Luthans (restaurant) study? How were the
consequences identified?
• Problem-based Assessment :
– Suppose you were asked to implement a motivational
program in a business with 25 employees engaged in
the following activities: production planning, inventory
delivery, direct production, packaging, and marketing.
Based on your understanding of the restaurant study,
how would you improve the quality of the employees
work by using access to activities as a motivator?
Describe the costs and benefits of the program and
make a recommendation about implementation.
• Old question:
– The behavioral context into which punishment is
added must be considered in evaluating the likely
effect of a given stimulus. What conditions are
essential in this analysis?
• New question:
– Suppose it is your task to decrease the frequency of
college students engaging in binge drinking or
games involving high levels of intoxication. You have
been asked to set up a punishment program to
eliminate this program before someone is hurt. What
context for binge drinking would you identify first?
What punishing consequence would you use? What
characteristics would you include in your punishing
system to maximize effectiveness? Please give an
example of an additional element you would need to
include to make the plan maximally effective.
Points for Question 1
• 4pts - reinforcement for drinking
• 5 pts - ID specific severe punishing consequence
• 4 pts - Practical plan for measurement of abusive
drinking
• 12 pts - Immediate consequence (4 pts); severity
(more than change action) (4 pts); consistent
system (4pts)
• 5 pts - reinforcement plan for alternative actions
that are preferred, if possible using original class
of reinforcers
• Old question:
• Describe the results of the experiment on semantically mediated
generalization done with a 13-year-old Soviet boy. Be sure to
identify the respondent behavior, the training stimuli, and the
test stimuli. Why is this example semantically rather than
physically mediated generalization?
• New question:
• Generate your own example of classical conditioning using the
neutral stimuli “idea” and “rock” along with the eliciting relation
between a very loud noise and increased heart rate. Your
example should include all of the following components: a
description of a conditioning procedure that would produce
differing reactions to the two stimuli, a description of the
procedure that tests for the direct effects of conditioning, a
description of a procedure that would test for physically
mediated generalization, a description of a procedure that would
test for semantically mediated generalization, and the likely
results of the three test procedures.
Points for Question 2
• 4 pts - specific conditioning -- say “idea” (2) and then
puff of air in eye (2)
• 7 pts - Direct test: Present “idea” (3) w/o air (2) and
see blink (2) follows
• 7 pts - Physically mediated generalization: Present
sound similar to “idea” (3) w/o air (2) and see weak or
little blink (2)
• 7 pts - Semantically mediated generalization: Present
sound or picture or word related to “idea” (3) w/o air
(2) and see strong blink (2) more than physically
mediated
Use existing work:
Keep an archive of evidence
Distribution on 1st Attempt
Old questions
40
35
30
25
Percent
20
Students
15
10
5
0
<= 69
70-79
80-89
Levels of Achievement
90-100
Change of Questions
Old questions
New questions
50
45
40
35
Percent 30
25
Students
20
15
10
5
0
<= 69
70-79
80-89
Levels of Achievement
90-100
What do you make of the data?
• Is this a surprise?
• What examples do you
have in your own work?
• What does it take to
“problematize” a concept
for assessment?
• Keep going in this direction
or turn back to abstraction?
I relied on my experimental results
• The key is getting students to read
• If only they were prepared, we would
all be better off
• Post hoc argument from fluency -free up mental resources for problem
solving [away from details]
Organizing units by topics
Online questions on reading
Feedback until mastery of topics
Students consider reading
Complete roster services
Click on name for full record
See items, make comments
Aggregate Performance
1st Try
New Questions
35
30
25
Percent 20
Students 15
10
5
0
<=69
70-79
80-89
Levels of Achievement
90-100
Addition of EDU Out of
Class
New Questions
Web-aided
40
35
30
25
Percent
20
Students
15
10
5
0
<=69
70-79
80-89
Levels of Achievement
90-100
What do you make of these data?
Would you keep using it? I did.
Next idea came from Grant
Wiggins
•
•
•
•
Welding teacher story
Generate internalized criteria
Like composition studies and editing
Used prior student work with
permission
• No feedback from me
Bb - consider others’ work
Students share reactions w/o me
Exam - Eliminate Problem Actions
No Web
W/ Examples
70
60
50
Percent 40
Students 30
20
10
0
(0-3)
(4-6)
(7-9)
(10-12)
Categories of Achievement
(13-15)
Exam Question on
Cultural Anthropology
No Web
W/ Examples
80
70
60
50
Percent
40
Students
30
20
10
0
(0-4)
(5-8)
(9-12) (13-16) (17-20)
Categories of Achievement
Take home complex essay
Without
With Examples
70
60
50
Percent 40
Students 30
20
10
0
(0-5)
(6-10)
(11-15)
(16-20)
Categories of Achievement
(21-25)
Add Generation to Recognition
•
•
•
•
•
Posted another variation on the question
Each student was to write an answer
Also expected to comment on one answer
Collected a sample of answers
They were focus of class discussion
– Students comment on completeness
– Offer suggestions to refine answers
Learning With Full Techno Boost
Report:
http://www.unl.edu/peerrev/examples/bernstein/index.html
Old questions
New questions
Most recent
50
45
40
35
Percent 30
25
Students
20
15
10
5
0
<= 69
70-79
80-89
Levels of Achievement
90-100
Comparison with last three
offerings (percent of students)
60
50
40
Fall 00
Fall 01
Spring 03
Spring 04
30
20
10
0
<=69
70-79
80-89
90-100
Distribution of Performance
Median 80/90
8
7
6
5
spring 03
spring 05
4
3
2
1
0
<=69
70-79
80-89
90-100
General Pattern of Improvement
• Increased frequency and care of
student reading
• Dramatically improved class climate
• Increased recognition of examples of
understanding
• Improved percentage of students
reaching high levels of understanding
Note use of equivalent problems
• Same idea with different contextual cues
• Generate up to eight problems per concept
• Use several to teach, then test in new
context
• Teach again as needed in new examples
• Retest in yet additional new context
• Final exam is yet another generalization test
Inductive and deductive teaching
•
•
•
•
•
Active discovery is important learning
Didactic instruction can be efficient
Bransford -- “time for telling”
My research => advantages for both
Lendol Calder is a nice synthesis
Comments? Examples?
Questions?