Chapter 14
TYPES OF ANALYSES
Item Responses and Univariate Analysis
The first step is to go just a short way beyond the raw data on questionnaires by producing column, row, group, and
subgroup tabulations and percentages, often called a "code book." The codebook tells how people answered each item on
the questionnaire by frequencies and percentages for each possible response category. Going one step further in the data
analysis, evaluators can compute descriptive statistics and other indicators that help describe the frequency distributions.
Bivariate Analysis and Comparison of Two Groups
Comparisons between groups of respondents can be made. If evaluators want to study the relationship between two
variables, they use correlational techniques, which show that a change in one variable is associated with a change in
another. For example, we might want to determine whether the performance of the Federal Aviation Administration's flightstation service specialists decreases appreciably with age. We would plot the performance scores of specialists of various
ages and see whether performance is related to age. We might use an analytic technique such as correlational analysis,
which shows the degree to which two variables are related. Or we might compare the differences between two groups rather
than the association between variables. For example, we might compare the performance of younger specialists with that of
older specialists. Other primary analysis techniques would include cross tabulations, chi-square comparisons, "t" tests, and
analyses of variance.
Multivariate Analysis and Comparison of Multiple Groups
This level of analysis is used when what is wanted is look at the associations between more than two variables or at
differences between more than two groups. For example, we might want to study the effect of age and experience on
Federal Aviation Administration specialists' performance or the effect of age, experience, training and education, and
recency of training and education all together. Here, we could use such multivariate techniques as partial correlations,
multiple regression analysis, and factor analysis. We could also compare performance by looking at the differences between
groups that have varying levels of each trait (older and experienced, younger and experienced, older with limited
experience, younger with limited experience, and so on). We might use such techniques as multiple analysis of variances,
discriminant analysis, linear structural relations, or log linear analysis.
Choice of Analysis Methods
The choice of data analysis methods depends largely on the evaluation questions and subject matter under study and on the
type of variables and what levels of measurement they satisfy. For example, if we had a question about whether the
performance of Federal Aviation Administration specialists is different at different ages, and if we had reason to believe
that performance was related to age and little else, a simple correlational analysis would reveal the degree of the
relationships. But the matters GAO studies are usually more complicated than this, so we would expect other variables such
as experience, education, training, and recency of education and training to be related to performance. We would need then
to perform multivariate analysis in order to determine the relationships of the variables. Likewise, it might be important to
compare performance across several groups rather than to confine the analysis to simple contrasts between pairs. The more
complex analyses should usually be undertaken only after the results of simpler analysis have been examined.
Sometimes evaluators have a choice between using associations and using group differences, and sometimes they do not.
The shape of the data distribution, the measurement scales, and the plots of the functional relationship between the
variables may rule out the use of correlation techniques. For example, sometimes we have to study group differences
because the distribution of the observations is not normal; we could not then use certain correlational statistics.
Correlational techniques are also inappropriate when the variables are scaled with ordinal data and when the relationships
under study are not linear that is, the plot between the variables cannot be transformed into a straight line. It is important to
realize
that
correlational
techniques
cannot
by
themselves
be
used
to
show
causality.
Because questions about cause and effect are sometimes posed, we must note that special designs such as nonequivalent
comparison groups, regression discontinuity, and interrupted time-series are usually necessary for establishing causality.
The logic of the evaluation design, not the analytic technique, is crucial in drawing inferences about causality.