Education 353C
Problems in Measurement: Generalizability Theory
Richard J. Shavelson & Edward H. Haertel
Tuesdays 9:00AM – 11:50AM
Cubberley 207
Generalizability Theory (G theory) provides a powerful class of psychometric models for
behavioral and social measurement data. G theory was developed beginning in the 1960s
by Lee J. Cronbach here at Stanford University with his collaborators Goldine Gleser,
Harinder Nanda, and Nageswari Rajaratnam (Cronbach, Rajaratnam, & Gleser, 1963;
Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It encompasses classical test theory
(CTT) as a special case, but goes far beyond CTT in clarifying conceptual confusions and
providing more powerful statistical tools. Most importantly, G theory systematizes
distinctions among the many reliability coefficients of CTT, and enables the calculation
of reliability coefficients and/or standard errors of measurement tailored to specific
measurement applications and/or interpretations. G theory also provides a powerful
extension of the Spearman-Brown "prophecy" formula, enabling the design of efficient
data collection plans for specific measurement applications.
In this course, we will provide a systematic introduction to the concepts and models of G
theory, as well as special-purpose software useful in carrying out the requisite
calculations. The goals of the course are to provide a working knowledge of common Gtheory models and applications, to enable students to read critically journal articles
employing G theory, and to introduce some unsolved problems and current research
topics related to G-theory.
The class will typically be organized as follows. The instructors might begin with a minilecture on the topic of the day with discussion. Then, a problem with data will be
introduced and students will work in teams to solve the problem within the G theory
framework. Toward the end of the quarter, the problems will be replaced by student-led
presentation of self-chosen paper that uses G theory.
Occasional homework will consist of students working on a brief problem with data.
There will be a final project in which either a student will apply G theory to a somewhat
novel set of data and report her/his findings to the class or will read up on a topic not
covered in class (e.g., newly developed standard errors and confidence intervals for
variance component estimates) and teach the class about the topic. There will be no final
examination.
The primary text for the course is Generalizability Theory: A Primer by Richard J.
Shavelson and Noreen M. Webb. In addition, Generalizability Theory by Robert L.
Brennan is recommended. Additional readings will be provided during the quarter. Note
that the Primer is just that—an introductory text on G theory. Brennan’s is the currently
authoritative text. So, when you complete a chapter in the Primer and you want more
statistical background, turn to Brennan. Brennan covers roughly the same materials in
his Chapters 1-4 as the Primer does—but in a somewhat different structure.
G Theory Seminar
Shavelson & Haertel
You can purchase both textbooks online so they are not in the bookstore. For
convenience, we have copied the first two chapters of the Primer and will email them to
you.
Some References
Brennan, R. L. (1997). A perspective on the history of generalizability theory. Educational Measurement:
Issues and Practice, 16(4), 14-20.
Brennan, R. L. (2000a). (Mis)conceptions about generalizability theory. Educational Measurement: Issues
and Practice, 19(1), 5-10.
Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.
Cardinet, J., Tourneur, Y., & Allal, L. (1976). The symmetry of generalizability theory: Application to
educational measurement. Journal of Educational Measurement,13, 119-135
Cardinet, J., Tourneur, Y., & Allal, L. (1981). Extension of generalizability theory and its applications in
educational measurement. Journal of Educational Measurement, 18, 183-204.
Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational
and Psychological Measurement, 64(3), 391-418
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral
measurements: Theory of generalizability for scores and profiles. New York: Wiley.
Cronbach, L. J., Linn, R. L., Brennan, R. L, & Haertel, E. H. (1997). Generalizability analysis for
performance assessments of student achievement or school effectiveness. Educational and
Psychological Measurement, 57, 373-399.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of
reliability theory. British Journal of Statistical Psychology, 16, 137-163.
Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110).
Westport, CT: American Council on Education/Praeger.
Shavelson, R. J. (2004). Editor’s preface to Lee J. Cronbach’s “My Current Thoughts on Coefficient Alpha
and Successor Procedures.” Educational and Psychological Measurement, 64(3), 389-391.
Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (1999). Note on sources of sampling variability in
science performance assessments. Journal of Educational Measurement, 36, 61-71.
Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973-1980.
Mathematical and Statistical Psychology, 34, 133-166.
British Journal of
Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.
Shavelson, R.J., Webb, N.M., & Rowley, G. (1989). Generalizability theory. American Psychologist, 44(6),
922-932.
Webb, N. M., Shavelson, R. J., & Maddahian, E. (1983). Multivariate generalizability theory. In L.J. Fyans
(Ed.), Generalizability theory: New directions for testing and measurement (pp. 67-82). San Francisco:
Jossey-Bass.
Webb, N. M., Shavelson, R. J., & Haertel, E.H. (2007). Reliability coefficients and generalizability theory.
In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics: Psychometrics (Vol. 26, pp. 81-124).
Amsterdam: Elsevier B. V.
Wiley, E. W. (2000). Bootstrap strategies for variance component estimation: Theoretical and empirical
results. Unpublished doctoral dissertation, Stanford University.
Spring 2007
2
Tentative Course Schedule1
Date
Topic
Readingsa
4-3
Introduction to G Theory & Statistical
Demo Software (GENVA & EduG)
SW Tables 1.1 and 1.3
SW Chapter 1
CGNR Preface &
Pp 1-14
4-10
AERA Week—No class meeting
4/17
Discuss Homework #1
See 4-3 Readings
SW Chapter 2
Brennan Appx. F
EduG Users Guide
Homework #1
A. Handout P x I G study
B. Run Exercise 3 in SW Ch. 3
with GENOVA & EduG
SW Chapters 2 & 3
Homework #2—Brief proposal
for final project (bring handouts
to class 4/24)
Read Cardinet’s Why EduG
Statistical Model for G Theory
G Studies with crossed facets
One-facet G Study simulation
1
Assignmentb
4/24
Discuss final project ideas
Discuss Why EduG
In Class Exercise #1
G studies with nested facets
SW Chapter 4
Homework #3—Read & prepare to
lead discussion:
Group 1: Cardinet et al 1981
Group 2: CLBH
Group 3: SRW
5-1
Group presentations of papers
Random & Fixed Facets
SW Chapter 5
Homework #4 Prepare final
project description;
hand in at class on 5-8
5-8
Decision Studies
Exercise#2_Cardinet #1
Collect project descriptions
SW Chapter 6
Homework #5 Cardinet
Exercise #2 & Cardinet
Complex Exercise
This schedule is tentative and may change depending on coverage in class or other unexpected events.
G Theory Seminar
Shavelson & Haertel
5-15
Discuss Homework #5
G & D Studies for same and
different Designs
SW Chapter 7 & 8
5/22
Multivariate G Theory
mGenova Exercise
WS
Brennan 267-285
(Optional)
5-29
Presentations of student projects
Work on projects
6-5
Presentations of student projects
___________________________________________
a
Readings to be done before class on date given
CGNR refers to Cronbach, Gleser, Nanda, & Rajaratnam (1972).
CLBH refers to Cronbach, Linn, Brennan, & Haertel (1997).
SW refers to Shavelson & Webb (1991).
SRW refers to Shavelson, Ruiz-Primo, and Wiley (1999).
WS refers to Webb & Shavelson (1983).
b
Assignments due the week following their assignment
Spring 2007
4