Assessment
as the
“Hidden Variable”
in Achievement
Michael Zeilik
Dept. of Physics & Astronomy
The University of New Mexico
(NSF DUE 99-85511)
What Works in Higher Ed?
20th Century Gain Results
Active Learning, Mastery Learning
(SD ≥ 0.5, PSI, discussion,
debates, games, role playing)
Cooperative Learning
(SD ≥ 0.5, cognitive and affective;
century of research, all disciplines)
One-on-One Tutoring
(SD = 2, with trained tutors)
What Does Not Work?
Standard Models (SD < 0.3)
Everything Else!
Lectures:Reinforce
memorization; promote illusion
of competence; educational
seduction (Dr. Fox)!
Still not better than lectures:
Computer-based, instructional
television, Web-based
Evidence-Based Reform
Active learning
+ Classroom assessment
=
Achievement:
Deep conceptual
understanding
(“sticky learning”)
What is Active
Learning?
Instructional activities
that engage students’ minds
(the “sticky” factor)
Interactive lectures, discussion,
cooperative learning teams, role
playing, peer teaching, minute papers,
concept mapping, games, cooperative
tests, controversies
Require alternative assessments
What Works in Context:
Physics & Astronomy
(PAER)
Disciplinary education research, “tools of
the trade” for experiments
Empirical robust knowledge base about
learning in physics & astronomy
Probe initial state (prior knowledge),
final state (learning outcomes)
Measure change:
Assessment!
Field-tested Learning Assessment
Guide (FLAG: www.flaguide.org)
FLAG CATs
 Attitudinal Surveys:
E. Seymour, E. Lewis
 Concept Tests:
A. Ellis
 Concept Maps:
M. Zeilik
 Conceptual
Diagnostic Tests:
M. Zeilik
 Interviews: M. Smith,
S. A. Southerland
 Performance
Assessments:
T. Slater
 Portfolios: T. Slater
 Scoring
Rubrics:
D. Ebert-May
 Student Assessment
of Learning Gains:
E. Seymour
 Weekly Reports:
E. Etkina
 Mathematical
Thinking: M. Swan,
J. Rideway
 Multiple-Choice
Tests: J. Parkes
 Minute Papers:
M. Zeilik
Conceptual Diagnostic Tests
Assess prior knowledge of students
“Misconceptions” research,
“think aloud” interviews;
“natural language”
Measure pre/post conceptual gains
(normalized gain, <g>)
Force Concept Inventory (FCI);
Astronomy Diagnostic Test:
“Astro 101”, nationally normed,
reliable & valid
Pre/post: ADT National Project/UNM
<g> = (post% - pre%)/(100 - pre)
56
60
Score (%)
50
Standard errors plotted
38
40
32.4
30
20
10
53.7
47.3
41.5
34
27
U
N
M
U
N
M
0
Pre-course (N=5346)
Overall: <g> = 0.22
Men: <g> = 0.25
Post-course (N=3842)
Women: <g> = 0.20
UNM: <g> = 0.31
From Quarks to Cosmos
(Committee on the Physics
of the Universe; NRC)
 11 interface areas (21st century!)
UNM Physics 102 (Conceptual
Physics) poll, top 5:
1. What is dark matter?
2. What is the nature of
dark energy?
3. How did the universe begin?
4. Are there additional spacetime
dimensions?
5. What are the masses of neutrinos?
Student Assessment of
Learning Gains (SALG)
Probes learning gains students
perceive for specific aspects
Avoids performance critiques (“Dr.
Fox”) of instructor (not correlated
with conceptual gain)
Easily customized (15 min);
access through FLAG
(hosted by University of
Colorado/Boulder)
UNM P102 SALG Spring 2003
(Mean = 3.38)
3.99
Videos
3.75
CT/CRS
4.15
Coop quizzes
3.71
Activities
3.64
Team work
3.52
Discussions
3.68
Instructor talk
1.00
2.00
3.00
4.00
Rating (scale 1 - 5)
5.00
P102 UNM Spring 2003
Gender Matters—Sometimes!
Female
Male
Posttest
Confidence
All
p < 0.002
Pretest
confidence
p < 0.0001
Last math
p < 0.002
Science SelfAptitude
NS!
Math SelfAptitude
NS!
1.00
2.00
3.00
4.00
Score (1 = very poor, 5 = very good)
5.00
Cooperative Quiz Gains
by Gender
P102 UNM Spring 2003 Quizzes
<g> Female
Effect size (F)
<g> Male
Effect size (M)
1.00
0.80
0.60
0.40
0.20
0.00
Practice
Quiz #1
Quiz #2
Quiz #3
Mean
Does it stick?
(Coop quizzes/Test)
120
100
Pre %
Post %
Test#1
80
60
40
20
Item
Q
37
Q
36
Q
27
Q
28
Q
29
Q
24
0
Q
112
Percent correct
A101 UNM Fall 2000
Identical Items (p = 0.96)
Grade Expectations
P102 UNM Spring 2003 Expected &
Actual Grades (by gender)
Percentage
A
B
C
D
F
90
80
70
60
50
40
30
20
10
0
Pretest
(F)
Pretest ICES (F) ICES (M) Ac tual
(M)
(F)
Ac tual
(M)
FC
I
g>
<
FC
I
g>
RC
G
G
RC
G
RC
)
RC
)
G
FC
I+
+
+
g>
I
FC
I
FC
Female
<
Po
st
(F
CI
Pr
e(
<
Po
st
%
Pr
e%
Percentage
P102 UNM Spring 2003 Pre/Post
Male
70%
60%
50%
40%
30%
20%
10%
0%
Assessment “C” Lessons
(Gender Sometimes Matters!)
Content matters!
Concepts matter!
Connections matter!
Context matters!
Confidence?
Competence?
Assessment probes all &
increases achievement by 0.5 SD!