Common Concerns
about the
Force Concept Inventory
Charles Henderson
Pat Heller
and the Physics Education Research Group
University of Minnesota
http://www.physics.umn.edu/groups/physed/
1
Introduction
Why do we care about the FCI?
• Quality Assurance
Quick and Easy to Administer
Robust
Responds to Changes in Instruction
• Widely Used
2
U of MN
Introductory Calc-Based Physics
•Traditional format: 3 hours of lecture, one 2-hour lab,
one 1-hour recitation each week
•Goal of course is to learn physic through problemsolving;
•Non-traditional pedagogy: Cooperative Group
Problem-Solving
FCI Testing
•Pre Test ⇒ first week of labs (ungraded)
•Post Test ⇒ as part of final exam (graded)
or
last week of labs (ungraded)
3
Common Concerns about the FCI
Question
Response
1. Can FCI pre test be used as a
diagnostic test?
No
2. Does the FCI post test correlate
with desired course outcomes
(problem solving grades)?
3. Do students take the FCI
seriously when it is not graded?
Somewhat
(r~0.51)
Yes
4
Common Concerns about the FCI
Question
Response
4. Does giving the FCI as a pre
test bias post test results?
No
5. Who gains?
6. Are there gender differences in
FCI scores?
All Students
Yes
5
Question #1
Question: Can FCI pre test be used as a
diagnostic test?
Method: Look at the correlation between
FCI pre test scores and desired
course outcome (grades on
written problems on the final
exam).
6
Typical Final Exam Problems
Introductory Calculus-Based Physics
1. On the way to work, you drive along a curved section of highway
which gets extremely slippery in bad winter weather. You want to
know how fast you can drive around the curve without having to rely
on the friction between your worn-out tires and the slippery road. You
estimate that the banking of that section of the road is about 5º relative
to the horizontal and the radius of curvature is 200 m. How fast can
you drive safely?
2. In the diagram shown below, block 1 of mass 1.5 kg and block 2 of mass 4
kg are connected by a light taut rope that passes over a frictionless pulley.
Block 2 is just over the edge of the ramp inclined at an angel of 30 o, and the
blocks have a coefficient of sliding friction of 0.21 with the surface. At time
t = 0, the system is given an initial speed of 11 m/s that starts block 2 down
the ramp.
vo
(a) Find the tension in the rope.
m1
(b) Find the speed of the two masses at t = 2 s. m2
30o
7
FCI Pre vs. PS Grade
F97, F98, F99 N=1645
2.0
1.5
1.0
1 Standard Deviation
0.5
PS Z-Score
26
36
0.0
-0.5
27
58
58
39
39
33
74
111
111
52
20
85
115
116
129
83
99
109 117
Best Fit: Y=.057x - .738
r = 0.34
36
-1.0
-1.5
Average FCI Pre
Score = 13.6
-2.0
0
5
10
Number of students shown below each data point
15
FCI Pre Score
20
25
30
8
Question #2
Question: Does the FCI post test correlate
with desired course outcomes
(problem solving grades)?
Method: Look at the correlation between
FCI post test scores and desired
course outcome (grades on
written problems on the final
exam).
9
FCI Post vs. PS Grade
F97, F98, F99 N=1818
2.0
1.5
1.0
41
90
80
113 99
114
102 114
0.0
93 103
100
111
-0.5
Best Fit:
Y=.087x - 1.746
r = 0.51
-1.0
1 Standard Deviation
PS Z-Score
0.5
90
92
93
78
24
44
57
65
53
27
-1.5
Average FCI Post
Score = 20.5
-2.0
0
5
10
Number of students shown below each data point
15
FCI Post Score
20
25
30
10
Question #3
Question: Do students take the FCI
seriously when it is not graded?
Method: Look for differences between
ungraded FCI tests and graded
FCI tests
• Obvious lack of seriousness
• Subtle lack of seriousness
11
Obvious Lack of Seriousness
Lack of Seriousness
% of Students for Ungraded*
•Refuse to take test
~0.5%
•Draws a picture
~0.2%
•Answers all A’s, B’s, etc.
none found
•Leaves a lot of blanks (6 or more)
~1.5%
•Other systematic patterns
~0.2%
*Based on data from N=1818 introductory Calculus-Based physics
students at the University of Minnesota from 1997-1999.
12
Subtle Lack of Seriousness
FCI Score
(N=500)
Fall 1997 (On Final Exam)
71.4%
Winter 1998 (Ungraded in Lab)
69.8%
Difference*
1.6%
(~0.5 FCI items)
*Significant difference on a matched sample t-test
13
Test-Retest Comparison
N=500, r=0.88
30
W98 Pre (Ungraded in Lab)
34% had higher score
25
an average of 2.3 pts.
20
15
18% had same score
10
48% had lower score
5
an average of -2.7 pts.
2 Standard
Deviations
0
0
5
10
15
20
25
30
F97 Post (On Final Exam)
14
Seriousness
FCI Gains
No significant differences in FCI relative gain by
the same professor between years when the post
test is graded and when the post test is ungraded.
University of Minnesota, 1996-1999
Introductory Calculus-Based Physics
0.60
Relative Gain <g>
0.50
0.40
0.30
0.20
0.10
1996
Post Test In Lab
1997
Post Test On Final
1998
Post Test On Final
1999
Post Test In Lab
0.00
M*
G
N*
G
O*
P*
O
K
D
N
P
G
O
G
N
M
O
K
Q*
Lecture Section
each letter represents a different instructor
I - Standard Error of the Mean
* - First Time With Course
15
Conclusions
Question
1. Can FCI pre test be used as a
diagnostic test?
Correlation between FCI pre test
and course outcome (problem
solving grades)
2. Does the FCI correlate with
desired course outcomes
(problem solving grades)?
Response
No
(r~0.34)
low slope
(0.057)
Somewhat
(r~0.51)
higher slope
(0.087)
16
Conclusions
Question
3. Do students take the FCI
seriously when it is not graded?
Response
Yes
• Most of the ways students don’t take the test
seriously are easy to spot (we remove these students
from our analysis):
Not taking the Test
Leaving 6 or more blanks
• There is no meaningful difference between FCI
scores on a graded test and an ungraded test.
17
Question #4
Question: Does giving the FCI as a pre test
bias post test results?
Method: Compare post test scores for
students who had pre test and
students who did not. (Lab groups
were ‘randomly’ assigned to the group
that received pre test or the group that
did not receive the pre test.)
18
Pre Test vs. No Pre Test
FCI Post
Fall 1998
FCI Pre (N=440) . . . . . . . . . . . . . . 70.5%
No FCI Pre (N=161) . . . . . . . . . . . 68.7%
No significant difference on a
pooled variance t-test (P=0.29)
Fall 1999
FCI Pre (N=355) . . . . . . . . . . . . . . . 65.2%
No FCI Pre (N=170) . . . . . . . . . . . 64.3%
No significant difference on a
pooled variance t-test (P=0.63)
19
Question #5
Question: Who gains?
Method: Look at the relationship between
relative gains on the FCI and pre
test scores.
20
Who Gains?
FCI pre vs. Relative Gain <g>
F97, F98, F99 All
N=1648
0.70
0.65
0.60
36 19
0.55
2
0.50
Average <g> = 0.43
58 39 39
4
58
112
Relative Gain <g>
0.45
20
52
109
10
117
85
0.30
115 116
33
27
74
83
112
0.40
0.35
16
130
13
99
26
36
0.25
0.20
0.15
0.10
9
Average Pre
Score = 13.6
0.05
0.00
3
-0.05
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
FCI Pre Score
I = Standard Error of Mean
21
Question #6
Question: Are there gender differences in
FCI scores?
Method: Compare the relationship
between pre test and post test
scores for males and females.
22
Gender
Differences
FCI pre vs. FCI post
F97, F98, F99 All, Binned by 2
N=392 Female, N=1233 Male
Each point represents at least 10 students
30.0
28.0
26.0
24.0
Average Post
Score = 20.6
22.0
FCI Post Score
20.0
18.0
Female
16.0
Male
14.0
12.0
10.0
8.0
Average Pre
Score = 13.6
6.0
4.0
2.0
0.0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
FCI Pre Score
I = Standard Error of Mean
23
FCI Measurement Error
Standard Error of Measurement = 2.18 items
•Based on Post Test data from Fall 1998 (similar for other
years and pre test)
N = 709
Ft = 5.45 items
" = 0.84 (Cronbach alpha)
•Standard error of measurement is related to reliability
Fe = Ft √ 1 - "
Fe → Standard error of measurement
Ft → Standard error of distribution of obtained scores
" → Reliability of test
• This relationship comes from:
1) F2total = F2true + F2error
2) Reliability = F2true / F2total = 1 - F2error / F2total
24