Evaluating theories efficiently: the nuts and bolts of contrast analysis
Even if an ANOVA is significant, if the independent variable has three or more levels the
main effect cannot directly evaluate a specific theory (it just says there is a main effect of IV
on DV). In addition, even if there is a significant interaction effect, the exact opposite of the
predictions made beforehand can lead to a significant interaction effect. To surpass this
problem, researchers usually perform a series of post hoc pairwise comparison to determine
which groups differ from each other significantly. Eventually, with enough pairwise
comparisons, it is possible to piece together enough information to evaluate the theoretical
expectations. However, this process is unnecessarily complex and relatively inefficient, and
it depends on hypothesis tests of relatively low power. Furthermore, the complexity
increases dramatically with every independent variable added to the study and with every
level of each independent variable. This method of analysis is an informative way to analyze
data, but it is a relatively inefficient way to evaluate theories.
The contrast approach to data analysis can they rectally examine the degree to which the
observed pattern of data matches the pattern of data implied by the theory (the expected
pattern of the data).
Two modes of data analysis
Confirmatory analysis and exploratory analysis are two fundamental modes of data analysis.
Although confirmatory analysis are used to evaluate theories, exploratory analysis can be
used to get to know once data, to learn from the data, to see where the action is in the
data. Paradoxically, many of the typical techniques that psychologists use are ideal for
neither exploratory nor confirmatory analysis. For example, ANOVA describes and provides
significant test of the amount of variation in a dependent variable that is associated with the
variation in independent variables. However, what if the action in the data does not map
neatly onto the analysis that ANOVA provides? Similarly, researchers’ hypotheses do not
always (or perhaps often) map neatly onto the analysis provided in the typical ANOVA
output. ANOVA may provide a host of null hypothesis tests, but it is quite possible that few
if any of these tests actually correspond to the key hypothesis that the researcher is
interested in evaluating. Contrast analyses can be very useful and informative complements
to these more conventional procedures.
What can contrasts tell us?
The major advantage of the contrast approach to data analysis is that it can provide
relatively clear and direct evaluations of theoretically driven predictions. To address their
questions efficiently researchers can conduct a contrast analysis that provides pieces of
information that give a clear and direct answer to their primary question of the
correspondence between obtained and expected patterns of data. When many
psychologists think about evaluating theories, or of analyzing data in general, the first piece
of information they may seek is a significant test. This can be informative, if used properly.
However, if researchers use a contrast approach to address their question, they can easily
compute a significance test and probability level associated with their particular expected
pattern of data. Effect sizes are the second kind of information obtained from contrast
analysis. Effect sizes are ordinarily more useful than the probability level, and they are
certainly more useful than a single dichotomous decision about statistical significance. Any
probability level should be interpreted in light of one or more effect sizes. From the effects
size perspective, confirmatory analysis is a question of theory evaluation instead of theory
testing. In other words, theory evaluation is a matter of degree (“How strongly is my theory
represented in the data?”) as opposed to a matter of passing or failing (“Is the finding
significant or not?”).
How good is this theory? Basic contrast procedures with between-groups data
Our goal is to provide a step by step, intuitive introduction to the logic and mechanics of
conscious analysis and to demonstrate its relationship to more traditional analyses. Basic
contrast analysis can be seen as a four step process.
The first step is to translate one theory into numbers. If we are dealing with group means,
approximately, what means should each group have? The outcomes of our contrast analysis
will tell us how closely they obtained pattern of results matches this expected pattern of
results. Aside from its role in contrast analysis, this step might also be a useful way to
encourage researchers to think clearly about the exact implications of a theory.
The second step is to translate these numbers into contrast weights. An easy way to move
from expected results in step one to contrast weights, is to compute the mean of the
expected values and subtract this from each of the expected values, which retains the
expected pattern but ensures that the weights sum to zero.
The third step is to compute a significance test and probability level that will evaluate the
statistical significance of the match between the observed pattern of data and the expected
pattern of data, now represented by the contrast weights.
Finally the 4th step is to compute an effect size, in this case r contrast.
Which theory is better? Comparing 2 theories
contrast analysis can provide a straightforward evaluation with a significance test and effect
sizes of the degree to which the data supports one theory more or less than another theory.
The procedure for comparing 2 theories can be seen as a five step process that builds on the
basic four step process just outlined.
In the first step comma the researcher translates both theories into numbers. in the second
step, the researcher translates both sets of numbers into contrast weights. in the third step,
the researcher computes a new set of contrast weights that represents the difference
between the two theories. In the fourth step, the researcher computes a t-test and
probability level that evaluates the significance of the match between the observed pattern
of data and the new difference contrast weights. Finally the 4th step is to compute an effect
size. This can be interpreted as the degree which the obtained pattern of data matches one
theory’s expected pattern of data more than it matches another theory’s expected pattern
of data.
Contrasts with repeated measures data
Contrast analysis can also be used profitably with repeated measures data. The
conventional approach to analyzing repeated measures data would be to conduct a
repeated measures ANOVA end, after finding a hoped for significant interaction, proceed
with a series of post hoc analyses. Again this is a relatively inefficient way to evaluate the
predictions.
Instead, the researcher could conduct a contrast analysis to obtain a significance test and
effect size reflecting the degree to which the treatment group showed a greater symptom
decrease then did the control group. Such analysis begins with a within-person pattern step
in which we assess the degree to which each participant manifests the expected pattern of
results. To do this, we create a set of contracts weights representing the pattern of results
that we expect to find in the group of interest (for example the treatment group) . For
example, one might expect to find high symptom scores at pretest, lowered symptoms
course at mid treatment, and even lower symptom scores at posttest. These predicted
scores are converted into contrast weights using the procedure outlined above.
For each participant, we then compute an L score reflecting the degree to which the
participant exhibits the expected pattern of results. This is done by multiplying each
participants 3 symptom scores by the expected symptom levels associated with the three
times of assessment and then summing across the three times of assessment.
In the between person evaluation step, a researcher then conducts an independent groups
T test on the mean L scores to obtain a significance test and an effect size evaluating the
degree to which the treatment group manifests the expected pattern of symptom decrease
more than does the control group.
Contrasts with frequency data
When working with frequency data, researchers often use a chi square analysis. As with
ANOVA in the analysis of means, a chi square analysis with more than 1 degree of freedom
is a diffuse or unfocused test, that typically provides no information about a specific theory.
For example, a significant chi square can tell us that there are differences among states in
their proportion of republican voters, but it cannot address a specific prediction about the
relative number of republican voters in each state. Again, this may be useful information,
but it is not a focused evaluation of a theory.
As with the basic contrast analysis illustrated with group means, contrast analysis with
frequencies proceeds through a four-step procedure. In the first step, the researcher
translates their prediction into numbers. In the second step, the researcher translates these
numbers into contrast weights. Again, the most efficient way to do this is to subtract the
mean of the numbers from each predicted number. In the third step the researcher
computes a significance test and probability level to evaluate whether the observed pattern
of frequencies (Proportions, to be exact) significantly match the expected pattern of
frequencies, as represented by the contrast weights. Because they are working with
frequency data, the researcher will use a Z test. In the fourth step, the researcher computes
an effect size R.
Conclusions
This article is intended to remind and convince researchers their contrast analysis is a
straightforward way to evaluate theories efficiently. In addition, it is intended to help
alleviate concerns about the adequacy or relevance of conscious analysis. Alongside
conventional analytic procedures and exploratory data analysis, contrast analysis can
provide simple, straightforward, and enlightening information about the theories and data
of social scientists.
Why are researchers unlikely to use contrast analysis? One reason may be that many
researchers are not familiar with the advantages of conscious approaches. Contrast analyses
are not strongly represented in the existing research literature, and there is a degree of
centrifugal force in the field. We see others using ANOVA in their results section so we tend
to do the same, which in turn means contrasts are no in our results section, and so on. So
many researchers may rarely, if ever, see contrasts in action, and thus remain unaware of
their benefits. In addition, planned contrast techniques may not be well represented in
statistical textbooks.
A specific issue that might be of interest is the choice of contrast weights. Which weights
should be used? Would the results be different if different contrast weights were used? The
choice is indeed crucial, as the utility of conscious analysis rests on the assumption that
researchers choose appropriate contrast weights. To make this process as straightforward
as possible, rough guidelines for choosing contrast weights are outlined previously, but at
least two deeper issues enter into such choices. First, choosing appropriate weights requires
careful consideration and familiarity with the theory or theories in question. Including the
exact weights in the research report allows reviewers and readers to judge for themselves
the adequacy of the choices. One could even argue that the careful attention to theory
required by contrast analysis is in fact a strength of the procedures distinguishing them from
many other analytic strategies. The second issue concerns the temptation to fish for
contrast weights that provide significant results. Aside from relying on professional ethics
relevant to all areas of research process, there is little to prevent this. Perhaps the best
policy is to remind ourselves of the consideration and conscientiousness required by the
research process in general and by contrast analysis in particular. A third reason that
researchers might not use planned contrast is that the procedures are perceived as difficult
to carry out or not worth the effort to relearn them.