Assessing and Promoting High-Impact
Practices at Course and Program Levels
Doing Assessment as if Deep Learning Matters Most
Tom Angelo
A Keynote Session at the
Mihaylo College of Business & Economics
XVIII Assessment Conference
14 March 2014
A WARNING TO PARTICIPANTS!
The following workshop is rated
M – for Mature Audiences only.
The workshop contains:
• Nudity
• Adult themes
(e.g., “HOT HIPs”)
• Strong language
(e.g., The “F” word)
2
If you work in assessment,
you know the “F” word
Ineffable
3
Raise your hand,
please, if you are
familiar with . . .
The ancient Greek
Myth of Sisyphus
4
Do you ever feel like Sisyphus?
5
That was the nude bit.
No, there’s no more. Sorry!
6
Or maybe more like . . .
Assess-yphus?
7
About one hour after the Accrediting Team leaves campus
8
Why Do We Need Assessment?
We’re highly educated, highly
ethical, dedicated professionals.
Why can’t they just trust us?
5/31/2016
9
Why Do We Assess Learning? - 1
Summative purposes
•
•
•
•
•
•
To compare learners against each other
To compare learning against criteria
To certify competency
To award qualifications
To ration resources
To provide accountability
5/31/2016
10
Why Do We Assess Learning? - 2
Formative purposes
•
•
•
•
•
•
•
To focus learners’ attention
To illuminate and undermine misconceptions
To increase motivation to learn
To provide learners with feedback
To improve performance
To promote self-assessment & monitoring
To develop independent, lifelong learning
5/31/2016
11
To do assessment as if
deep learning matters most,
we must privilege the
formative purposes.
Page 2 –
Background Knowledge Probe
Please answer each question regarding
the USA, Canada and Mexico.
Guessing is encouraged!
5/31/2016
13
A “Balcony” Question
If you participated actively:
Are you more interested in
finding out the answers to
these questions than you
were a few minutes ago?
5/31/2016
14
Students’ prior knowledge
and beliefs are among
the most powerful
influences on their learning
Consequently, assessing
that prior knowledge can
provide powerful leverage
15
Bloom’s Revised Taxonomy
6. Create
5. Evaluate
4. Analyze
3. Apply
2. Understand
1. Remember
5/31/2016
16
Page 3
Collaborative Learning Technique
Think-Pair-Share
This is a “Low-Threshold Application”
•Low complexity – easy to use
•Low cost – in time and effort
•Low risk – to teachers or learners
•Relatively high ROI (Return on Investment)
•Potentially worth adapting?
5/31/2016
17
Applications Card – p. 13
Interesting
IDEAS/TECHNIQUES
5/31/2016
Possible
APPLICATIONS
18
p. 4 - top half –
Plus-Minus-Question Mark
Some key terms and concepts that might be of use
Formative and Summative Assessment and Feedback
Surface, Strategic, and Deep Learning Approaches
Deliberate Practice
Prior Knowledge
Bus Test, Parrot Test, Parking Lot Test
Cognitive Load
Metacognition
The Dance Floor and the Balcony
Novice-Expert differences
The 80/20 Rule (aka, the Pareto Principle)
Threshold and Core Concepts
19
Page 4 – top half
Plus-Minus-Question Mark
Please mark each item on the list with
a plus sign, minus sign, or question mark
• Use the plus ( + ) if you understand it
• Use the minus ( – ) if you do not understand it
• Use the question mark (?) if you’re unsure
5/31/2016
20
A “Balcony” Question
If you followed directions:
Did you read and think
about the list on page 4
any differently than you
would have if you had
simply been asked to
“read it”?
21
Formative Assessment &
Feedback for Deep Learning –
A Gap-Analysis Approach
Find the Gaps
Mind the Gaps
Close the Gaps
22
p. 4 – bottom half –
The Teaching-Learning-Assessment Pyramid
23
“It’s not what we do,
but what students do
that’s the important
thing.”
Biggs, J. & Tang, C. (2007). Teaching for
Quality Learning at University, 3rd Edition.
Berkshire: McGraw-Hill, p. 19.
Page 5
Six Dimensions of Higher Learning
Metacognitive
Reflective
Conditional
Procedural
Conceptual
Factual
5/31/2016
25
Page 6
Seven Transformative Guidelines for
Doing Assessment as if Learning Matters Most
1.
2.
3.
4.
5.
6.
7.
5/31/2016
Build shared trust
Build shared language and concepts
Build shared goals and motivations
Design backward and work forward
Think and act systematically
Take a scholarly approach
Don’t assume, ask
26
The first, last, and most
fundamental principle
of assessment is . . .
Don’t assume, ask.
5/31/2016
27
What makes ConcepTests work?
•
•
•
•
•
•
•
Focus on core/threshold concepts
Conceptual questions
Prediction
Discussion
Commitment
Feedback
Extension/Variation
28
Applications Card – p. 13
Interesting
IDEAS/TECHNIQUES
5/31/2016
Possible
APPLICATIONS
29
HOT HIPs!
Higher Order Thinking
can be promoted effectively through
High-Impact Practices
30
What makes these HIPs so HOT?
•
•
•
•
•
•
High expectations
Focused instruction
Metacognitive awareness
Effective feedback
Deliberate practice
Effective collaboration
31
Impeding Critical Thinking
Approaches Contraindicated by Research
•
•
•
•
•
•
5/31/2016
Focus on rote learning
Information/work overload
One-shot assignments/assessments
Norm-referenced (curved) marking
Assessment fatigue
Incoherent curricula
32
Developing Critical Thinking
Factors Well-Supported by Research
• Engagement in an interdisciplinary,
integrated, coherent curriculum
• Mastery of an inquiry/research method
• Authentic problem-solving
• Structured collaborative work
• Standards-based assessment and feedback
• Positive experiences of diversity
• High levels of engagement and effort
5/31/2016
33
Three Key Building Blocks
for
Doing Assessment as if Learning Matters Most
1.
2.
3.
5/31/2016
Build shared TRUST
Build shared LANGUAGE & CONCEPTS
Build shared GOALS & MOTIVATION
34
Page 6
Seven Transformative Guidelines for
Doing Assessment as if Learning Matters Most
1.
2.
3.
Build shared trust
Build shared language and concepts
Build shared goals and motivations
4. Design backward and work forward
5.
6.
7.
5/31/2016
Think and act systematically
Take a scholarly approach
Don’t assume, ask
35
“From the student’s
point of view, the
assessment is the
curriculum.”
Paul Ramsden
Pages 7 & 8
Assessing and Grading
as if Deep Learning matters most
5/31/2016
37
Seven Transformative Guidelines for
Doing Assessment as if Learning Matters Most
1.
2.
3.
4.
5.
Build shared trust
Build shared language and concepts
Build shared goals and motivations
Design backward and work forward
Think and act systematically
6. Take a scholarly approach
7.
5/31/2016
Don’t assume, ask
38
p. 9
TO USE FEEDBACK WELL,
LEARNERS NEED M.O.M.
• MOTIVATION – COMPELLING REASONS TO USE IT
• OPPORTUNITIES – FOR SAFE, GUIDED PRACTICE
• MEANS – KNOWLEDGE & SKILLS FOR IMPROVEMENT
39
p. 9 – bottom
THE ORDER IN WHICH WE GIVE FEEDBACK
MATTERS
CONSIDER THE FOLLOWING FIVE STEPS:
1ST - GOOD NEWS: WHAT WAS DONE WELL
2ND - BAD NEWS: WHAT STILL NEEDS IMPROVEMENT
3RD - OPTIONS: WHAT CAN BE DONE TO IMPROVE IT
4TH - PLANS: WHAT THE LEARNER INTENDS TO DO
5TH - COMMITMENTS: WHAT BOTH PARTIES AGREE TO
DO, HOW, TO WHAT STANDARD, AND BY WHEN
42
Page 12
Seven Transformative Guidelines for
Doing Assessment as if Learning Matters Most
1.
2.
3.
4.
Build shared trust
Build shared language and concepts
Build shared goals and motivations
Design backward and work forward
5. Think and act systematically
6. Take a scholarly approach
7.
5/31/2016
Don’t assume, ask
43
The first, last, and most
fundamental principle
of assessment is . . .
Don’t assume, ask.
5/31/2016
44
Applications Card – p. 13
Interesting
IDEAS/TECHNIQUES
5/31/2016
Possible
APPLICATIONS
45
The Parking Lot Test
Tom Angelo -- 16 Nov. 2012 -- thomas.a.angelo@gmail.com
46
What, Why and How
Choose one of your possible applications.
Prepare to answer the three questions below
about that specific application:
• What is it?
• Why do you think it might be useful?
• How do you think you might use it?
47
Thanks for your attention and participation.
And good luck!