Classical Test Theory: Example Questions and Outline Model Answers
Question 1
(a) What are the basic assumptions of classical test theory? (15% of marks for
question)
(b) Consider a variable x. One measure of the variable produces observed values
denoted by xobs. Write an expression for xobs in terms of the true values of x
(denoted by xtrue). Explain how your expression reflects the assumptions of the
theory. (15% of marks for question)
(c) Use the terms in your formula to define the reliability of the measure, and
demonstrate that a test-retest correlation provides an estimate of reliability
under classical test theory. (35% of marks for question)
(d) You have another measure of the underlying variable, with observed values on
this measure denoted by xobs2. Write further expressions, based on classical
test theory, to explain why the average of the two measures (i.e., 0.5*[xobs +
xobs2]) is a more reliable measure of x than either of the measures alone. (35%
of marks for question)
Question1: MODEL ANSWER
(Obviously, alternative algebraic symbols for the equations below can be used, as
long as they are properly defined; if nonobvious steps in the derivations are missed
student will lose some marks)
a. Assumptions of Classical Test Theory (CTT)
CTT proposes that the observed value is a combination of the true value plus a
random measurement error component. By stressing random error, CTT is making 3
assumptions about the error component:
(i)
The error component will have zero mean and so this means that the observed
mean will not be systematically distorted away from the true value by the error
(and this contrasts with a systematic bias effect).
(ii) The measurement errors are assumed to follow a normal distribution.
(iii) The measurement errors are uncorrelated with the true values.
b. Expression for Observed Value of Variable x
We can write the following expression for the observed value of x for a particular
subject:
xobs = G1(,true) + G2(0, err)
…(1)
where: Gn(x,y) denotes a value drawn at random from a normal distribution with
mean=x and s.d.=y; =the mean of the true values of variable x; true and err are the
standard deviations of the true values and the error terms respectively. Note that this
expression contains all the elements of the theory (true plus error; 0 mean of error;
random error from normal distribution). For the later parts of the question it is useful
to note that the expected value of the variance of xobs, according to the theory, will be
(true2 + err2).
c. Reliability of a Measure
The reliability of a measure is defined as the proportion of variance in the measure
that is due to the construct being measured (rather than measurement error). Thus,
according to CTT:
Reliability = true2 / (true2 + err2)
… (2)
We can show, using CTT, that a test-retest correlation for a particular measure, will
give us an estimate of reliability. A test-retest correlation uses a measure taken at two
separate times (denoted xobs and xobs2). We can write the following expressions:
xobs = G1(, true) + G2(0, err)
xobs2 = G1(, true) + G3(0, err)
… (3)
where we have assumed that the error variance of the measure is the same at each
timepoint (err1 = err2 = err) at each timepoint. (G1 must be common to the 2
expressions, because we assume that an individual’s true score on the variable is the
same at each time-point).
The test-retest correlation for x across times 1 and 2 can be written as r12.
r12 = Covar(xobs, xobs2) / sqrt(Var(xobs)*Var(xobs2))
… (4)
where Var(x) is the variance of value x, and Covar(x, y) is the covariance between two
values x and y. Using CTT for the observed variables described by expression (3), the
expected values of the variances in expression (4) are as follows:
Exp{Var(xobs)} = true2 + err2
Exp{Var(xobs2)} = true2 + err2
… (5)
and the covariance of xobs and xobs2 is true2.
Using these results in combination with expression (4) the expected value of the testretest correlation has the following value:
Exp{r12} = true2 / (true2 + err2)
… (6)
which is the same as the definition of reliability (2, above).
d. Summing Two Measures to Increase Reliability
We can use CTT to show that summing measures of the same construct increases
reliability. We have two independent measures xobs and xobs2 of the underlying
variable. Let’s define our combined measure, C = 0.5*(xobs + xobs2). Using CTT and
the expression (3) above we can write:
Cobs = G1(, true) + 0.5*G2(0, err) + 0.5*G3(0, err)
… (7)
and for simplicity we have again assumed that the error variance is the same in xobs
and xobs2. Also, it follows from the definition of standard deviation that n*G(m, s)=
G(m, n*s), we can therefore rewrite expression (7) thus:
Cobs = G1(, true) + G2(0, 0.5*err) + G3(0, 0.5*err)
… (8)
When a variable is the sum of two uncorrelated variables it has a s.d. which is the
square-root of the sum of the variances of the variables being summed together.
Therefore, we can combine the two uncorrelated error terms of expression (8) into a
single random error term, thus:
Cobs = G1(, true) + G4(0, sqrt(0.5)*err)
… (9)
We can see that the combined measure has a less variable error term than that of
either measure alone (see the expressions in 5 above) and therefore this measure is
more reliable. From the definition of reliability given earlier (2), we note that the
reliability of the combined measure is true2 / (true2 + 0.5*err2).