Brazil Science, Technology, Humanities, Engineering and
Mathematics (STHEM) Consortium
1ST WORKSHOP: LORENA, BRAZIL,
MAY 26‐30, 2014
Changing Teaching, Transforming Learning
Session II: Assessing Learning
Gregory Light, PhD
Overview
Session I: Design Principles & Learning Outcomes
• Global & National Challenges (Interactive Presentation)
• Teaching for Learning: a Framework (Interactive
Presentation)
• Constructing Learning Outcomes (Activity)
Session II: Assessing Learning
• Assessing Student Learning (Discussion)
• Dimensions of Assessment (Interactive
Presentation/Activity)
• Aligning Assessment (Activity)
“The quickest
way to change
student
learning is to
change
the assessment
system.”
-- Elton & Laurillard, 1979, p. 10
ASSESSMENT
Group Question:
Why do we assess
students?
At Tables
Share
Purposes of Assessment
Summative
•
•
•
•
•
•
•
•
To pass or fail a student
To grade or rank a student
To license to proceed
To select for future courses
To license to practice
To predict success in future courses
To predict success in employment
To select for future employment
Purposes of Assessment
Formative
• To provide feedback to students to improve their
learning
• To motivate students
• To diagnose a student’s strengths and weaknesses
• To help students develop their skills of selfassessment
• To provide a profile of what a student has learnt
Purposes of Assessment
Evaluative
• To provide feedback to teachers
• To improve teaching
• To evaluate a course’s strengths and
weaknesses
• To make the course appear ‘respectable’ and
credit worthy to other institutions and
employers
Dimensions of Assessment
Dimensions of Assessment
Self-Referenced
Criterion Referenced
Based on self-reflection
of learning achieved on
the course
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
Norm-Referenced
Based on
comparisons with
others in the group
The extent to which the results of the assessment
method can be trusted
Peer-Referenced
Based on peer
appraisal of learning
achieved on the course
Dimensions of Assessment
Self-Referenced
Based on self-reflection
of learning achieved on
the course
Criterion Referenced
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
Norm-Referenced
Based on
comparisons with
others in the group
The extent to which the results of the assessment
method can be trusted
Peer-Referenced
Based on peer
appraisal of learning
achieved on the course
Dimensions of Assessment
Self-Referenced
Criterion Referenced
Based on self-reflection
of learning achieved on
the course
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
The extent to which the results of the assessment
method can be trusted
Norm-Referenced
Based on
comparisons with
others in the group
Peer-Referenced
Based on peer
appraisal of learning
achieved on the course
Dimensions of Assessment
Self-Referenced
Based on self-reflection
of learning achieved on
the course
Criterion Referenced
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
The extent to which the results of the assessment
method can be trusted
Norm-Referenced
Based on
comparisons with
others in the group
Peer-Referenced
Based on peer
appraisal of learning
achieved on the course
Dimensions of Assessment
Self-Referenced
Based on self-reflection
of learning achieved on
the course
Criterion Referenced
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
The extent to which the results of the assessment
method can be trusted
Norm-Referenced
Based on
comparisons with
others in the group
Peer-Referenced
Based on peer
appraisal of learning
achieved on the course
Dimensions of Assessment
LEARNING-CENTERED Criterion Referenced
Self-Referenced
Based on self-reflection
of learning achieved on
the course
Based on knowledge
and skills learned on
course
Validity
The extent to which the assessment
methods reflect student learning and the
learning goals of the course
Summative
Essentially designed to
sum up someone’s
achievement
Practicality
Formative
Essentially designed
for use in helping the
learning process
Reliability
Norm-Referenced
Based on
comparisons with
others in the group
The extent to which the results of the assessment
method can be trusted
Peer-Referenced
TEACHING-CENTERED
Based on peer
appraisal of learning
achieved on the course
Activity: Assessment
Methods
Choose an assessment
method(s) you currently use
and “map” it on to
1) a learning outcome
2) these dimensions.
Do individually, then share at table
Then with whole group
Assessment Methods
(Some Examples)
•
•
•
•
•
•
final exam (written)
Homework
projects
reports
Presentations
observed discussion
groups
• office hours
• final exam (multiple
choice)
• quizzes
• cold calling
• career performance
• standardized tests
• personal response
systems
Higher validity – less reliable
Higher reliability – less validity
Aligning Assessment
Using Rubrics
(Making valid assessments
more reliable)
Quick
Share
Aligning Assessment with a Learning Outcome
(Biology Example)
Course Goal
Learning
Outcome
Formative
Assessment
Summative
Assessment
What will
students learn?
If they have learned What will students do
it, what will students to learn it?
know and be able
to do?
How will students
demonstrate they
know it or are able
to do it?
Students will
understand the
transfer of
information from
DNA to proteins.
Students will be able
to predict changes in
amino acid sequences
caused by mutations.
On exam students will
predict the new
amino acid sequence
that results from a
mutation in a given
gene sequence.
In groups students are
given sequence of DNA
corresponding to amino
acid sequence. Students
identify reading frame
and predict amino acid
changes due to mutations
in that sequence.
Knight, (2011)
A Typical Rubric Format
STUDENT
EVIDENCE
TEACHER
CRITERIA
(1) Weak
Little or no
evidence of
outcome
(2) Basic
Some evidence
of outcome
Criterion 1
Argument
Criterion 2
Integration
of
literature
Criterion 3
Writing
quality
Adapted from Beauchamp et al 1996
(3) Proficient
Detailed
evidence of
outcome
(4) Strong
Highly
creative;
outcome
Assessing Writing: Example Rubric
(1) Weak
Little or no
evidence of
outcome
(2) Basic
Some evidence
of outcome
(3) Proficient
Detailed
evidence of
outcome
(4) Strong
Highly
creative;
outcome
Criterion 1 Rambling; poor
Argument use of logic;
personal
opinion
Basic structure
with some use of
evidence
Strong structure &
logic, evidence
used throughout
Strong structure
& evidence;
sophisticated
discussion
Criterion 2 No or minimal
Integration use of sources
of
literature
Sources used, but Sources well
no integration of integrated into
writer’s ideas
author’s ideas
Criterion 3 Poor
Writing
organization,
quality
grammar,
syntax
Some effort to
organize ideas;
grammar/syntax
problems
Adapted from Beauchamp et al 1996
Sources well
integrated;
critique made
Good
Well-developed
organization; very flow; error-free;
few grammar/
elegant style
syntax problems
Benefits of Rubrics
 Used for grading (Summative) or feedback
(Formative)
 Clear criteria (Criteria Ref.)
 Ensure grading aligns with learning outcomes
(Validity)
 Shared with multiple teacher graders/situations
(Reliability)
 Can be used with students (Self Ref.) and peers
(Peer Ref.)
Activity
In small group, identify and design
a rubric for a specific Learning
outcome
Large Group Discussion
Aligning Assessment
Using
Multiple Choice Questions (MSQ)
(Making reliable assessments
more valid)
Quick
Share
MCQs as formative instruction
Answer and provide your rationale:
1. Which of the following has/have intrinsic pacemaker
characteristics?
a) Medulla
c) Sinoatrial node
b) Pons
d) Atrioventricular node
Use rationales (short open-ended explanation for choice) to let
students demonstrate their learning - can be graded or not..
Stanford Learning Lab: Nash & Shaeffer, 1999; Schaeffer et al., 1999
Ideal rationale:
SA node is the normal pacemaker for the entire heart. AV node also has pacemaker
potential, but is overshadowed by SA node. Medulla has pacemaker potential for
breathing rhythm as well. Pons helps refine rhythm, but does not have pacemaker
potential.
Less-than-ideal rationales:
Offering an incomplete answer: Normally the SA node is responsible for generating heart
rate, and it is able to do this because of its intrinsic rhythm. The AV node also has an
intrinsic rhythm, but it is “overshadowed” by that of the SA node.
Providing a book definition: The sinoatrial node is the pacemaker of the mammalian
heart.
Providing irrelevant information: Stretch receptors are located in the aortic arch and the
carotid sinus. They have the ability to respond to changes in pressure.
Restating the question: The SA node, AV node, and medulla all possess intrinsic
pacemaker characteristics as they all serve as intrinsic pacemakers.
Blind appeal to authority: This answer is right because Professor Heller said that it was,
and Professor Heller is cool.
Activity
In small group, share other ways in
which the validity of MCQs might
be enhanced.
Share with Large Group
Testing low-level knowledge
Purely economic loss is recoverable in a product liability action.
a) True
b) False
Purely economic recovery will be barred in which of the
following causes of action?
a) Negligence
b) Fraud
“What’s the rule?”
c) Defamation
No context, not allowing for
d) Product liability
interpretation/analysis
from Case & Donohue, 2008
Revision: Higher-level understanding
A restaurant hired an exterminator to eliminate cockroaches from the
basement under the restaurant. Around midnight, the exterminator
applied to the basement floor and walls an effective pesticide that he had
purchased from the manufacturer. A toxic gas released by the pesticide
penetrated into the restaurant kitchen and did not disperse by the next
day. As a result, the restaurant was required to close that day.
The restaurant brought a tort action based on product liability against the
pesticide manufacturer for lost profits.
Will the restaurant prevail?
a) No, because in this action purely economic loss in not recoverable. *
b) No, because the exterminator was the proximate cause of the restaurant's
damages.
c) Yes, because the manufacture of pesticides is an abnormally dangerous
activity.
d) Yes, because the pesticide was being used as intended.
from Case & Donohue, 2008
3 ideas to take with you
Assessment is teaching
Students need to be involved
Assessment gives you no more
than what you ask for
Final
QUESTIONS?
References
• Entwistle, N. & Tait, H. (1990) ‘Approaches to learning, evaluations of teaching
and preferences for contrasting academic environments’, Higher Education, 19
(2): 169–94.
• Knight, J. (2011) University of Colorado. NAS/HHMI Summer Institute on
Undergraduate Biology Education, Madison, WI.
• Light, G, & Micari, M. (In press) Making Scientists: Six Principles for Effective
College Teaching, Harvard University Press.
• Beauchamp, McConaghy, Parsons & Sanford. (1996) Teaching From the Outside
In. Duval: 1996, 37.