EXCEL Template for Obtaining Internally Consistent Subsets of Items
This EXCEL template is intended to help delete items from a measure to produce two or
more sets of items that "fit the data" (are internally consistent).
The citation for these instructions is Ping, R.A. (2006). "More about the template for obtaining
an internally consistent set of items." http://home.att.net/~rpingjr/weeding1.doc.
New measures will almost never "fit the data" using a single construct measurement
model without dropping items to attain model-to-data fit. In addition, most well established
measures developed before covariant structure analysis (LISREL, AMOS, etc.) became popular
also will not fit the data without item weeding.
It turns out that measures used with covariant structure analysis are limited to about six
items (see discussions in Anderson and Gerbing 1984, Gerbing and Anderson 1993, Bagozzi and
Heatherton 1994, and Ping 2008). One explanation is that correlated measurement errors,
ubiquitous in survey data but customarily not specified in covariant structure analysis, eventually
overwhelm model-to-data fit in single-construct and full measurement models as indicators are
added to the specification of a construct. And, that usually happens with about 6 items per
construct.
There are ways around item weeding, such as various item aggregation techniques (see
Bagozzi and Heatherton 1994), but many reviewers in the Social Sciences do not like these
approaches. Unfortunately, reviewers also may not like dropping items from measures because
of concerns over face- or content validity (how well the items "tap" the conceptual and
operational definitions of their target construct). One "compromise" is to show the full measure's
items in the paper, and assuming the full measure does not fit a single construct measurement
model, show one submeasure that does fit the data and is maximally "equivalent" to the full
measure in face or content validity. However, to do that, several submeasures are usually
required, and finding even one is frequently a tedious task.
This template will assist in finding at least two subsets of items from the target measure
that fit the data in a single construct measurement model of the items. The process is as follows.
First, exploratory (common) factor analyze the target measure with its items using Maximum
Likelihood estimation and varimax rotation. If the measure is multidimensional, start with the
Factor 1 items. The other factors and the full measure can be used later.
Next, estimate a single construct (confirmatory) measurement model using the Factor 1
items (if the measure is unidimensional Factor 1 is the full measure). If the first measurement
model fits the data item omission is not required. If this measurement model does not fit the data,
find the "First Order Derivatives" in the output. (I will assume LISREL 8, which requires "all"
on the OU line to produce First Order Derivatives. As far as I know, most other estimation
packages produce statistics equivalent to First Order Derivatives). Paste the lower triangle of
First Order Derivatives for "THETA-EPS" into the template making sure you retain the item
names so you can figure out which item to drop (see the example on the template). Then find the
largest value in the "Overall Sum" column--it will be the same as the "Max =" value in the lower
right corner of the matrix.
Now, reestimate the measurement model with the item having the largest "Overall Sum"
omitted (call this Reestimation 1). Record the Chi Square and RMSEA values on the spreadsheet
for reference. If they are acceptable, use the items in this measurement model as submeasure 1.
There is no agreement on acceptable single construct measurement model fit. I use either
a Chi Square that is slightly nonzero for single construct measurement models (e.g., 1E-07, not
0), or an RMSEA that is .08 or slightly below, but many authors would suggest much stronger fit
criteria for single construct measurement models.1
If the unomitted items do not fit the data, find the "First Order Derivatives" for "ThetaEps" in the Reestimation 1 output. Paste these into the second matrix in the template, record the
Chi-Square and RMSEA values, and reestimate the single construct measurement model
(Reestimation 2).
Repeating this process, eventually Chi Square will become nonzero, and after that
RMSEA will decline to 0.08 or less (the recommended minimum for fit in full measurement and
structural models--see Brown and Cudeck 1993, Jöreskog 1993). This should happen with about
7 or 8, down to about 5, remaining items. If acceptable fit does not happen by about 4 items, an
error has probably been made, usually by omitting the wrong item.
Each subset after Chi Square becomes non zero is a candidate subset for "best," but
because items are disappearing with each step, these smaller subsets are usually less face valid,
and thus the first acceptable subset is usually the preferred one.
To find another subset of items, repeat the above process using "Modification Indices"
for "Theta Epsilon." The theory behind Modification Indices is different from First Derivatives,
and a different subset usually results.
Another subset of items usually can be found using reliability. The reliability of all the
Factor 1 items is computed using SAS, SPSS, etc., the item that contributes least to reliability is
deleted, and the reliability of the remaining items is computed. This process is continued until
deleting any item reduces reliability. The remaining items usually will fit the data in a single
construct measurement model.
If the full measure was multidimensional, there may be several more subsets found by
repeating the above procedures using the full measure's items instead of the Factor 1 items, then
using the reliability procedure just mentioned. Experience suggests these subsets are smaller, but
they frequently include items from Factor 2, etc. and thus they may be more face valid. This
process can also be used on any Factor 2 items, Factor 3, etc.
There are many more subsets that can be found by omitting the next largest "Overall
Sum" item instead of the "Max =" item. Specifically, the second largest item in Reestimation 1
could be omitted in place of the largest. Then, continuing as before omitting the largest "Overall
Sum" items, The result is frequently a different subset of items that fits the data. Another subset
can usually be found using this "Second Largest" approach using modification indices instead of
first derivatives. Others can be found omitting the second largest overall sum item in
Reestimation 2, instead of Reestimation 1, etc., with or without deleting the second largest in
Reestimation 1. This "Second Largest" strategy can also be used on the full set of items.
Experience suggests that there are about N-things-taken-6-at-a-time combinations of
items with real world data that will fit the data, where N is the number of items in the full
measure (more, if 5, 4 and 3 item subsets are counted). For example, if the original measure has
8 items, with real world data there are about 8!(8-6)!/6! = 112 6-tem subsets of items that might
fit the data. While the above strategies will not find all of them, experience suggests they should
identify several two subsets that are usually attractive because they are comparatively large
(again however, usually with about 6 items) and they should appear to tap the target construct
comparatively well.
The above spreadsheet approaches may not always identify the highest reliability subsets
of items, but experience suggests the resulting subsets are usually larger and as, or more, face
valid than those produced by other approaches. However, with low reliability measures, even
though the "First Derivative" or "Modification Indices" subsets should be only a few points
lower in reliability than a subset found by, for example, dropping items that contribute lest to
reliability, the higher reliability subset may be preferred to a higher face validity subset.
It may be instructive to (re)submit all the subsets found to an item-judging panel for their
selection of the "best" subset for each construct.
Other comments: There are exceptions to several of the assertions made above, but this is
probably not the place for an exhaustive exposition on item deletion strategies. For emphasis, the
template assumes lower triangular matrices. There is an additional example in Appendix E of the
monograph, Testing Latent Variable Models..., on the web site.
REFERENCES
Anderson, James C. and David W. Gerbing (1984), "The Effect of Sampling Error on
Convergence, Improper Solutions, and Goodness of Fit Indices for Maximum Likelihood
Confirmatory Factor Analysis," Psychometrika, 49, 155-73.
Bagozzi, Richard P. and Todd F. Heatherton (1994), "A General Approach to Representing
Multifaceted Personality Constructs: Application to Self Esteem," Structural Equation
Modeling, 1 (1), 35-67.
Browne, Michael W. and Robert Cudeck (1993), "Alternative Ways of Assessing Model Fit," in
Testing Structural Equation Models, K. A. Bollen et al. eds, Newbury Park CA: SAGE
Publications.
Gerbing, David W. and James C. Anderson (1993), "Monte Carlo Evaluations of Goodness-ofFit Indices for Structural Equation Models," in Testing Structural Equation Models, K. A.
Bollen and J. S. Long, eds., Newbury Park, CA: SAGE Publications.
Jöreskog, Karl G. (1993), "Testing Structural Equation Models," in Testing Structural Equation
Models, Kenneth A. Bollen and J. Scott Long eds., Newbury Park, CA: SAGE.
_______ (2004), "On Assuring Valid Measures for Theoretical Models Using Survey Data,"
Journal of Business Research, 57 (2), 125-41.
ENDNOTES
1
In my opinion, some authors go too far in real world data with single construct measurement model fit, resulting in
unnecessarily small submeasures. There are several issues here, including model fit versus face or content validity,
and experience suggests that with real-world data, "barely fits" in single construct measurement models is almost
always sufficient to attain full measurement model fit. Thus, in real world data, subsets of items that each produce a
comparatively small but nonzero Chi Square or an RMSEA that is just below .08 are usually "consistent enough" to
later produce a full measurement model that fits the data. I prefer the RMSEA criterion because it seems to produce
fewer problems later. Again, however, many authors would not agree with this strategy. Later, if it turns out that the
full measurement model does not adequately fit the data, simply estimate the next item weeding single construct
measurement model and drop the next largest "Overall Sum" items to improve full measurement model fit.