A Markov Chain Approach for Ranking Individuals From Item Responses
Matthew
S.
Johnson,
Baruch
Matthew_Johnson@baruch.cuny.edu
College,
CUNY
New
York,
USA,
The raw score has long been used to rank individuals from their item response data. When the
design is not balanced, for example, if different respondents receive different sets of items then
the use of the raw score may not be justified. This paper examines the efficacy of a ranking
system developed as an extension of one suggested by Callaghan, Porter, and Mucha (2003) for
paired comparisions data. The ranking system posits a latent voter whose job is to select the
“best” individual in the sample. The voter moves from one individual to another according to a
Markov chain. The transition probabilities of the Markov chain are formed by assuming that the
latent voter changes his opinion of who is “best” according to the following rules:
First the voter randomly selects another individual in the sample to compare with the current
“best” examinee.
If the two individuals have no items in common, then the voter chooses to move to the new
selected individual with probability one-half.
If the two individuals do have items in common, then the voter randomly selects one of the
common items to compare the two individuals.
If the two individuals receive the same score on the item then the voter randomly chooses one of
the individuals with probability one-half.
If the responses are not identical, then the latent voter moves chooses the individual with the
higher score on that item with probability 0.5+∂, where ∂, some number between 0 and 0.5, is a
sort of discrimination parameter that may vary by item.
The individuals are ranked according to the limiting distribution of the Markov chain. This
limiting distribution represents the likelihood that a given individual is the “best” in the sample.
In addition to describing the ranking system, the paper compares the method with suggested
alternatives from the literature, including raw scores, normits, and parametric methods. The
paper shows that the ranking system is equivallent to ranking by the raw scores when the design
is ballanced and discusses properties of the limiting distribution of the Markov chain. The
ranking system is demonstrated utilizing data from both attitudinal surveys and educationg
testing.
Keywords: Markov chains; nonparametric IRT; scaling; attitude measurement