JExam - A Method to Measure
Outcomes Assessment
Charles H. Atwood, Kimberly D. Schurmeier,
and Carrie G. Shepler
Chemistry Department
University of Georgia
Athens, GA 30602
Presented to
Academic Affairs Faculty Symposium
Unicoi State Park, Helen GA
March 31, 2007
University of Georgia – Chemistry Department
Outline of Presentation
•
•
•
•
•
JExam – Computerized Testing Program
Item Response Theory (IRT)
Using IRT to measure testing effectiveness
Outcomes Assessment
Conclusion
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• Item Response Theory (IRT)
– More modern psychometric analysis tool than CTT
– Works well for large sample sizes (> 200)
– Iterative process which fits the students responses
to a mathematical formula
1
P( )  c  (1 - c)
1  e -a( -b)
  ability level
a  discrimina tion parameter
b  difficulty parameter
c  guessing parameter
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• We use the program Bilog-MG3 to fit our data.
• Bilog-MG3 assigns to every test item values for
a, b, and c and displays them on an item
characteristic curve
– Plot of the probability of a student with a given IRT
ability vs. probability that the student answered the
question
University of Georgia – Chemistry Department
Item Response Theory (IRT)
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• Less than 5 minutes
later an item response
list is generated.
– Our raw data
University of Georgia – Chemistry Department
•
00000000 991099919919919919919909919909919
00000001 919919199199199199199199199199199
00000002 919919199199199199199199199199199
00000003 919909199199199199199199199199199
00000004 919919199099099099199199199199099
00000005 919919199199099099199199199199199
00000006 919919199199199199199099199199199
00000007 919909199199099199199199199199099
00000008 919919199199099199199199199099199
00000009 919919199199099099199099099199199
00000010 919919199199199099199199199199199
00000011 919919199199199099199199199199099
00000012 919919199199099199199199199199199
00000013 919919199199199199199199199099199
00000014 919919199199199199199199199199199
00000015 919909199199199199199099199099099
00000016 990099909919909909919909919909919
00000017 990099919919909909919919919919919
00000018 991099919909909909909909909909909
00000019 991099919919909919909919919919909
00000020 991099919919909909919909919919919
00000021 991099919919919919919919919919919
00000022 991199909919919919919919919919919
00000023 990099909909919909909909909909909
00000024 991199919909909919919919919909919
Item Response Theory (IRT)
• We enter the raw data
into Bilog-MG 3.0
along with a few
parameters.
• Less then 5 minutes
later we have our
Item Characteristic
Curves for every
question on the test.
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• Item Information
Curve for every
test question
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• C-D discriminator
example:
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• B-C discriminator
example:
University of Georgia – Chemistry Department
Item Response Theory (IRT)
• A-B discriminator
example:
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• Can tell us whether
or not our tests are
effectively
assessing students
at every ability
level.
• Test information
curve
– Prior to IRT
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• Test information
curve
– After IRT
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• Generate a plot of student ability versus
test score for every test during the
semester.
• Plot for 1st test of the second semester of
General Chemistry.
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• When student ability vs. test scores is
averaged over the last 6 years
– Grading scale that appears to be consistent
for the entire year and independent of student
population
– Possible absolute grading scale
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• If we generate a plot of the number of
students that have given ability on a test:
– Get the regular bell-shaped curve
– Plot for Spring semester 2006, 1st test
University of Georgia – Chemistry Department
Using IRT to measure testing
effectiveness
• Compare that to this plot:
– Spring Semester 2006, 1st test
University of Georgia – Chemistry Department
Outcomes Assessment
• After Spring Semester 2006, we went and
looked at the test questions which had the
highest discrimination and ability levels.
• From this analysis we discovered that
there are several topics which cause the
students the most difficulty for the entire
year.
University of Georgia – Chemistry Department
Outcomes Assessment
1. Understanding the structure of ionic
compounds
2. Unit conversion problems
•
Particularly converting from volume to area or
height
3.
4.
5.
6.
Molecular polarity
Intermolecular forces
Understanding quantum numbers
Distinguishing the terms strong, weak,
concentrated and dilute
7. Inorganic nomenclature
University of Georgia – Chemistry Department
Outcomes Assessment
• Understanding the structure of ionic
compounds is crucial to student performance
in 1211/1212
• This concept is essentially a gatekeeper.
University of Georgia – Chemistry Department
Outcomes Assessment
Fall 2005 Test 1
Question:
“What is the correct name of this ionic
compound?” Al(NO3)3
aluminum nitrate
“How many ions are present in one
formula unit of the compound shown
above?”
4
Ability at 0.336
Discriminates between C and D students
University of Georgia – Chemistry Department
Outcomes Assessment
• Since understanding ions is so difficult and
so crucial to success, we decided to tackle
it first.
• Prior to start of Fall semester 2006
– Teaching faculty meeting
– Discussed difficult topics
– Stressed that we must emphasize ions in
lecture
University of Georgia – Chemistry Department
Outcomes Assessment
• Fall 2006 Test 1
• Question:
• “What is the correct name
of this ionic compound?”
Al(NO3)3
• aluminum nitrate
• “How many ions are
present in one formula
unit of the compound
shown above?”
• 4
University of Georgia – Chemistry Department
Outcomes Assessment
• Look at the Gaussian Fit to Ability Scores
for several years
– Plot for Fall Semester 2004, 1st test
University of Georgia – Chemistry Department
Outcomes Assessment
• Compare that to this plot:
– Fall Semester 2005, 1st test
University of Georgia – Chemistry Department
Outcomes Assessment
• Compare that to this plot:
– Fall Semester 2006, 1st test
University of Georgia – Chemistry Department
Outcomes Assessment
• Second Test Comparisons
– Fall Semester 2004, 2nd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Second Test Comparisons
– Fall Semester 2005, 2nd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Second test comparisons
– Fall Semester 2006, 2nd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Third test comparisons
– Fall Semester 2004, 3rd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Third test comparisons
– Fall Semester 2005, 3rd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Third test comparisons
– Fall Semester 2006, 3rd test
University of Georgia – Chemistry Department
Outcomes Assessment
• Final Exam comparisons
– Fall Semester 2004, Final Exam
University of Georgia – Chemistry Department
Outcomes Assessment
• Final Exam comparisons
– Fall Semester 2005, Final Exam
University of Georgia – Chemistry Department
Outcomes Assessment
• Final Exam comparisons
– Fall Semester 2006, Final Exam
University of Georgia – Chemistry Department
Conclusion
• Item Response Theory is a more effective
method of assessing student ability.
• We can design and implement test
questions that discriminate student
abilities at all levels.
• Appropriate interventions can measurably
improve overall student abilities.
University of Georgia – Chemistry Department
Acknowledgements
• Bob Scott and John Stickney
– Department Chairs
• Gary Lautenschlager
– Professor of Psychology and IRT expert
• UGA’s PRISM project.
– This material is based on work supported by
the National Science Foundation under Grant
No. EHR-0314953.
University of Georgia – Chemistry Department