Measurement Problems within
Assessment: Can Rasch Analysis help
us?
Mike Horton
Bipin Bhakta
Alan Tennant
Mental Arithmetic - Test 1





4+5
=
2x3
=
18 ÷ 2 =
Arithmetic is one of the ‘3 R’s’. True or False? =
17 x 13 =
Mental Arithmetic - Test 1 - Answers





4+5
=9
2x3
=6
18 ÷ 2 = 9
Arithmetic is one of the ‘3 R’s’. True or False? = True
17 x 13 = 221
Assumptions underpinning test score addition

All questions must be mapping onto the same underlying
construct
• Unidimensionality

All questions must be unbiased between groups
• Item Bias  Differential Item Functioning (DIF)

Raw score is a sufficient statistic
Mental Arithmetic - Test 1 – Potential Problems





4+5
=9
2x3
=6
18 ÷ 2 = 9
Arithmetic is one of the ‘3 R’s’. True or False? = True
17 x 13 = 221
Mental Arithmetic - Test 1 – Potential Problems





4+5
=9
2x3
=6
18 ÷ 2 = 9
Arithmetic is one of the ‘3 R’s’. True or False? = True
17 x 13 = 221
Plus: Item Bias
-Gender DIF has been shown to be a particular problem in
mathematics exams. (e.g. Scheuneman & Grima, 1997., Lane et
al. 1996)
Mental Arithmetic - Test 2





17 x 13 =
47 x 64 =
768 ÷ 16 =
532
=
73
=
Mental Arithmetic - Test 2 - Answers





17 x 13 = 221
47 x 64 = 3008
768 ÷ 16 = 48
532
= 2809
73
= 343
Assumptions of Test Equating (Holland & Dorans, 2006)

Tests measure the same characteristic

Tests measure at the same level of difficulty

Tests measure with the same level of accuracy
Requirements of Test Equating (Dorans & Holland, 2000)

The tests should measure the same construct

The measures from the tests should have the same reliability

The function used to equate measures from one test to another should be
inversely symmetrical

Examinees should be indifferent about which of the equated test forms
will be administered

The function for equating tests should be invariant across subpopulations
of examinees
Are these elements currently assessed?


Unidimensionality – is assumed on face validity
• Cronbach’s alpha
Exam Difficulty Equivalence
• Subjective procedures
• Classical Test Theory = sample dependent
What is Rasch Analysis?
Mesa Press,
Chicago 1980
Rasch Analysis

The Rasch model is a probabilistic unidimensional model
• the easier the question the more likely the correct response
• the more able the student, the more likely the question will be
passed compared to a less able student.

The model assumes that the probability that a student will
correctly answer a question is a logistic function of the difference
between the student's ability and the difficulty of the question
Rasch G. Probabilistic models for some intelligence and attainment
tests. Chicago: University of Chicago Press, 1980
Assumptions of the Rasch Model

Stochastic Ordering of Items

Unidimensionality

Local Independence of Items
What Would We Expect When These People
Meet These Items?
Hard
Easy
Least
Most Able
Item 1
Item 2
Item 3
Person 1
Correct
Incorrect
Incorrect
Person 2
Correct
Correct
Incorrect
Person 3
Correct
Incorrect
Correct
Person 4
Incorrect
Correct
Correct
Person 5
Correct
Correct
Correct
What Would We Expect When These People
Meet These Items?
Hard
Easy
Least
Most Able
Item 1
Item 2
Item 3
Person 1
Correct
Incorrect
Incorrect
Person 2
Correct
Correct
Incorrect
Person 3
Correct
Incorrect
Correct
Person 4
Incorrect
Correct
Correct
Person 5
Correct
Correct
Correct
What Would We Expect When These People
Meet These Items?
Hard
Easy
Least
Most Able
Item 1
Item 2
Item 3
Person 1
Correct
Incorrect
Incorrect
Person 2
Correct
Correct
Incorrect
Person 3
Correct
Incorrect
Correct
Person 4
Incorrect
Correct
Correct
Person 5
Correct
Correct
Correct
What Would We Expect When These People
Meet These Items?
Easy
Least
Most Able
Hard
Item 1
Item 2
Item 3
Person 1
Correct
Incorrect
Incorrect
Person 2
Correct
Correct
Incorrect
Person 3
Correct
Incorrect
Correct
Person 4
Incorrect
Correct
Correct
Person 5
Correct
Correct
Correct
What Would We Expect When These People
Meet These Items?
Hard
Easy
Least
Most Able
Item 1
Item 2
Item 3
Person 1
Correct
Incorrect
Incorrect
Person 2
Correct
Correct
Incorrect
Person 3
Correct
Incorrect
Correct
Person 4
Incorrect
Correct
Correct
Person 5
Correct
Correct
Correct
The Guttman Pattern
1
0
1
1
1
1
1
1
2
0
0
1
1
1
1
1
3
0
0
0
1
1
1
1
4
0
0
0
0
1
1
1
5
0
0
0
0
0
1
1
6
0
0
0
0
0
0
1
Total Score
0
1
2
3
4
5
6
The Rasch Guttman Pattern
1
0
1
1
1
1
1
1
2
0
0
1
1
1
1
1
3
0
0
0
1
0
1
1
4
0
0
1
0
1
1
1
5
0
0
0
0
0
1
1
6
0
0
0
0
0
0
1
Total Score
0
1
3
3
3
5
6
The Probabilistic Rasch Model
Probability of a student’s success on an item
student
27% 12% 5%
95% 88% 73%
-3
-2
-1
0
1
2
3
Difference (in logits) between the ability of the
student and the difficulty of the item
Rasch Analysis



When data fit the model, generalisability of Item difficulties
beyond the specific conditions under which they were observed
occurs (specific objectivity)
In other words…
Item Difficulties are not sample dependent as they are in
Classical Test Theory
What Else Does Rasch Offer us?

When data fit the Rasch Model, the assumptions of summation are met
• All questions must be mapping onto the same underlying construct
• All questions must be unbiased between groups (DIF)
• Raw score is a sufficient statistic

We can then test for other things
• Quality of Distractors

It gives us the mathematical basis to compare test scores via equating
Limitations of Rasch Analysis

The model tests the internal psychometric properties

The model assumes unidimensionality

The model cannot set standards
Summary

The Rasch model offers a unified framework under
which all of the assumptions can be tested together

It gives us a lot of information about individual items
which can be utilised to ensure that item and test
construction is of a high quality

It provides a rigorous mathematical basis for test
equating
References





Rasch G. Probabilistic models for some intelligence and attainment tests.
Chicago: University of Chicago Press, 1980
Dorans NJ & Holland PW. Population invariance and the equatability of tests:
Basic theory and the linear case. Journal of Educational Measurement, 2000:
37; 281-306
Holland PW & Dorans NJ. Linking and Equating. In RL Brennan (Ed.),
Educational Measurement (4th ed., p187-220). Westport, CT: American
Council on Education and Praeger Publishers, 2006.
Scheuneman JD & Grima A. Characteristics of quantitative word items
associated with differential item functioning for female and black examinees.
Applied Measurement in Education, 1997; 299-320.
Lane S, Wang N, Magone, M. Gender-Related Differential Item Functioning
on a Middle- School Mathematics Performance Assessment. Educational
Measurement, 1996: 15(4); 21-27