THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL
SCHOOL OF SOCIAL WORK
COURSE NUMBER: SOWO 914
TITLE, SEMESTER AND YEAR: Measurement in Intervention Research, Fall 2011
INSTRUCTOR:
Natasha K. Bowen, Ph.D.
School of Social Work
TTK Building, Rm. 245E
Phone: (919) 843-2434
Fax: (919) 843-8715
Email: nbowen@email.unc.edu
OFFICE HOURS:
Thursday 1:00 to 2:00 or by appointment
COURSE DESCRIPTION:
This course focuses on the development of knowledge and skill in measuring social,
psychological, environmental, and other factors related to intervention with individuals, families,
groups, and organizations.
COURSE OBJECTIVES (SUBJECT TO APPROVAL OF RELEVANT FACULTY COMMITTEES):
Upon completion of the course, students should be able to:
1. Describe the theoretical, conceptual, and methodological foundations of qualitative and
quantitative measurement;
2. Develop and test theory-based scales, starting from a theoretically and empirically
justified item pool;
3. Conduct cognitive testing of potential scale items with representatives of an appropriate
target audience and analyze the data;
4. Conduct exploratory factor analysis using one or more basic statistics programs to
identify and evaluate the factor structure of scales;
5. Conduct confirmatory factor analysis to further support the validity of scales, and
understand the implications of data characteristics on the choice of software and analysis
strategies
6. Evaluate the reliability and validity of quantitative indices and scales;
7. Apply principles of measurement to research that involves issues of difference arising
from culture, ethnicity, language, race, religion, sexual orientation, and other aspects of
human diversity.
EXPANDED DESCRIPTION:
Research is possible only to the extent that concepts can be measured. If we cannot measure
something, we cannot draw conclusions about how often it occurs, what conditions are
antecedent to it, what factors covary with it, and so on. In that regard, some concepts are easy to
measure. Age and education are often measured in years. Income is measured in dollars and
1
school performance is measured by GPA. Some concepts, however, are harder to measure.
“Practice wisdom,” for example, is widely acknowledged as a concept that discriminates more
successful social workers from less successful social workers. What is practice wisdom? How
do we conceptualize and measure it?
For this course, measurement will be defined broadly. It is the craft of systematically
describing something that exists. The course will focus on one qualitative scale development
step and then a sequence of quantitative steps can be used together to develop high quality
measures. Course activities are designed to build both knowledge and skill. Consequently,
students will be expected to engage in a variety of applied learning experiences.
The issues that we will address are at the heart of intervention research. How, for
example, do we measure child abuse, delinquency, client-worker relationships, skills, feelings,
attachment, and other important concepts in the profession? Most people would agree that these
exist, but they are constructs that are not easily operationalized.
The goal if the course is to strengthen your skills in a mixed methods process of
developing quantitative measures of concepts commonly used in social work research. The
course is designed as a seminar. Readings and discussion will focus on the theoretical and
conceptual foundations of quantitative measurement. Students will be expected to take a
leadership role in the course—bringing to class meetings knowledge from outside readings and
experiences, and asking and helping to answer questions related to course topics that further the
growth of all class participants.
REQUIRED TEXTS/READINGS:
DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Thousand
Oaks, CA: Sage Publications.
Pett, M. A., Lackey, N. R., Sullivan, J. J. (2003). Making sense of factor analysis.
Thousand Oaks, Sage Publications.
Additional assigned readings are available through the course Blackboard site or electronic
journals.
RELATED READINGS AND RESOURCES:
For readings on quantitative measurement, see Journal of Social Service Research, Psychological
Bulletin, Psychological Methods, Research on Social Work Practice, Social Work Research,
Educational and Psychological Measurement, Measures for clinical practice: A sourcebook, and
many other social science journals and publications, and Buros CD/ROM (for recent reviews of
published instruments – in Davis Library). In addition, the following Web sites might be useful:
Aggression Assessment Instruments for children and adolescents -http://vinst.umdnj.edu/VAID/browse.asp
Buros Mental Measurements -- http://www.unl.edu/buros/
ERIC Test Locator (http://ericae.net/testcol.htm)--http://ericae.net/testcol.htm#ETSTF
Educational Testing Service -- http://www.ets.org
SF-36 -- http://www.mcw.edu/midas/health/.
2
TEACHING METHODS:
Teaching methods include lecture, discussion, computer labs, and application activities. Students
are expected to contribute actively to the learning climate of the classroom. Questions on course
topics and other issues in research and professional development are welcome.
ATTENDANCE AND CLASS ASSIGNMENTS:
Attendance. Students are expected to attend all class meetings, complete assigned and
independent readings, and contribute to the development of a positive learning environment in
the seminar. Students who miss 3 class meetings (excused and/or unexcused, and including
cumulative partial missed classes) will be required to take an Incomplete in the course and to
retake either the missed sessions or the entire course the following year before receiving a grade.
Course discussions and activities will be responsive to student interests. It is hoped that students
will share relevant knowledge and experience they already have related to course topics to enrich
the learning of other seminar participants. In addition, contributions based on independent
exploration of measurement topics beyond the stated course content are welcome. Attendance
and participation will be assessed weekly. Students who miss a class are fully responsible for
obtaining all missed information and materials from their classmates.
Graded Assignments:
One paper on your cognitive testing procedures and findings (due Oct 6, 2:00 pm)
One critique of an EFA article (due Oct 13, 2:00 pm)
One homework on EFA (due Oct 27, 2:00 pm)
One homework on reliability (due Nov 10, 2:00 pm)
One critique of a CFA article (due Nov 17, 2:00 pm)
One homework on CFA (due Dec 1, 2:00 pm)
One homework on validity (due Dec 8, 2:00 pm)
GRADING SYSTEM
The course uses the standard grading cutoffs.
H = 94-100
P = 80-93
L = 70-79
F = 69 and below
The final grade will be based on evaluation of seminar participation and written assignments.
One hundred points will be distributed as follows:
25
Cognitive testing paper
40 (10 X 4) Homeworks
15
Critique of an EFA article
15
Critique of a CFA paper
5
Preparation and participation
100
3
POLICY ON INCOMPLETES AND LATE ASSIGNMENTS
An automatic 10% reduction in score will be made for late papers (papers turned in after
the date and time given as deadlines).
POLICY ON ACADEMIC DISHONESTY
Original work is absolutely expected. Submitted work must conform to the Honor Code
of the University of North Carolina. For information on the UNC-CH Honor Code, including
guidance on representing the words, thoughts, and ideas of others, see: http://honor.unc.edu/
Please note that plagiarism is defined in the Instrument of Student Judicial Governance, Section
II.B.1 as, “…the deliberate or reckless representation of another's words, thoughts, or ideas as
one's own without attribution in connection with submission of academic work, whether graded
or otherwise.” In keeping with the UNC Honor Code, if reason exists to believe that academic
dishonesty has occurred, a referral will be made to the Office of the Student Attorney General for
investigation and further action as required. See the School of Social Work Writing Style Guide
(on the School’s Website at the “for current students” link) for information about plagiarism.
Students must not discuss homework or paper assignments or topics evaluated with those
assignments once they have been distributed. Graded work is expected to reflect totally
independent work. On the other hand, students are encouraged to consult written and online
resources on assignment topics (as long as they don’t post questions related to their assignments).
Code of Honor Affirmation. All written products in the course must have a signed Honor
Code statement. Papers without this affirmation will not be accepted. The statement should read
as follows: I have neither given nor received any unauthorized assistance on this assignment. In
addition to a pledge of no plagiarism, inclusion of this statement on assignments is interpreted by
the instructor as a pledge that the student has not discussed the assignment with other students
nor worked on it with other students.
POLICY ON ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES
Students with disabilities that affect their participation in the course and who wish to
have special accommodations should contact the University’s Disabilities Services and provide
documentation of their disability. Disabilities Services will notify the instructor that the student
has a documented disability and may require accommodations. Students should discuss the
specific accommodations they require (e.g. changes in instructional format, examination format)
directly with the instructor.
POLICIES ON THE USE OF ELECTRONIC DEVICES IN THE CLASSROOM
Use of electronic devices for non-class related activities during classtime (e.g. checking
email, playing games) is not acceptable.
4
READINGS AND COURSE OUTLINE
1.
THURSDAY AUGUST 25
Introduction to Measurement
READINGS (PROVIDE REFERENCE FOR ENTIRE COURSE):
Bowen, N. K. (2008). Validation. Invited chapter for W. A. Darity, Jr. (Ed.)
International Encyclopedia of Social Sciences, Vol. 8. (2nd ed., pp. 569-572.)
Detroit: Macmillan Reference.
Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats
itself again. In R. W. Lissitz (Ed.). The concept of validity: Revisions, new
directions, and applications (pp. 19-37). Charlotte, NC: Information Age
Publishing
Introduction to the Course and Measurement
Themes for the course
Implications for measurement
Review of measurement basics
A new look at validity
Latent variables
CLASS ACTIVITY: Discussion of your constructs and qualitative work so far; Practice
creating a set of questions to measure a construct
2.
THURSDAY SEPTEMBER 1
Mixed Methods in Measurement Research
Test Content as a Source of Validity Evidence
READINGS:
Bowen, N. K., Bowen, G. L., & Woolley, M. E. (2004). Constructing and validating
assessment tools for school-based practitioners: The Elementary School Success
Profile. In A. R. Roberts & K. Yeager (Eds.), Evidence-based practice
manual: Research and outcome measures in health and human services
(pp. 509-517). New York: Oxford University Press.
DeVellis, Chapters 1 and 5
Pett, Lackey, & Sullivan, Chapter 2
Use the readings to answer the following questions:
What similarities and differences do you see in the scale development procedures
described in the three readings?
How could you use the readings to justify the use of mixed methods in your study?
ASSIGNMENT DUE TODAY: Bring a preliminary list of scale items to class.
If you haven’t done so, complete the online human subjects training and print out your
certificate.
5
CLASS ACTIVITY: Peer review of item quality using published standards. Discussion of
informed consent for your target population.
3.
THURSDAY SEPTEMBER 8
Cognitive Testing of Scale Items
Response Process as a Source of Validity Evidence
READINGS:
Your choice—one chapter in the Special Populations section of S. Presser, J. M.
Rothgeb, M. P. Couper, et al. (Eds.). Methods for testing and evaluating survey
questionnaires. Hoboken, N.J., John Wiley & Sons:
Evolution and Adaptation of Questionnaire development, evaluation, and
testing methods for establishment surveys (ch. 19),
Pretesting questionnaires for children and adolescents (ch. 20) ,
Developing and evaluating cross-national survey instruments (ch. 21), or
Survey questionnaire translation and assessment (ch. 22).
Bowen, N. K. (2008). Cognitive testing and the validity of child-report data from
the Elementary School Success Profile. Social Work Research. 32,18-28.
(focus on data collection)
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire
design. Thousand Oaks, CA: Sage Publications. Chapters 4 and 10.
Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2004). Cognitive pretesting and
the developmental validity of child self-report instruments: Theory and
applications. Research on Social Work Practice, 14, 191-200.
Use the readings to answer the following questions:
When is it appropriate to use cognitive testing? When is it unnecessary or inappropriate to
use cognitive testing?
What are the major features of cognitive testing?
How might cognitive testing with children differ from testing with adults?
What cognitive testing steps would be appropriate for your study, and how could you justify
them with the readings?
What special issues in scale development relevant to your work did your “choice” reading
identify?
ASSIGNMENT DUE TODAY: Bring your revised items (based on last week’s feedback) to
class. Recommended: start drafting the methods section of your cognitive testing paper.
CLASS ACTIVITY: (a) Develop a plan for cognitively testing your questionnaire items
with up to 5 members of the target population. The plan should include sampling,
data collection procedures, a data recording strategy, and a timeline. (b) Test out your
protocol and recording plan with a classmate.
ASSIGNMENT HANDED OUT: Cognitive Testing Paper
6
4.
THURSDAY SEPTEMBER 15
Analyzing Cognitive Testing Data
Response Process as a Source of Validity Evidence, continued
READINGS:
Bowen, N. K. (2008). Cognitive testing and the validity of child-report data from
the Elementary School Success Profile. Social Work Research. 32,18-28.
(focus on analysis)
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire
design. Thousand Oaks, CA: Sage Publications. Chapter 11.
Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2006). The development and
evaluation of procedures to assess child self-report item validity. Educational
and Psychological Measurement, 66, 687-700.
.
ASSIGNMENT DUE TODAY: Start collecting data! Recommended: work on methods section
of paper.
CLASS ACTIVITY: Share data collection experiences. Discuss possible data analysis
strategies for your cognitive data and start designing analysis tools.
5.
THURSDAY SEPTEMBER 22
Results of Cognitive Testing
Introduction to Internal Structure as a Source of Validity Evidence
READINGS:
Forsyth, B., Rothgeb, J. M., & Willis, G. B. (2004). Does pretesting make a
difference? An experimental test. In S. Presser, J. M. Rothgeb, M. P.
Couper, et al. (Eds.). Methods for testing and evaluating survey
questionnaires. Hoboken, N.J., John Wiley & Sons.
Find and read an article about a scale developed in your research area that made
use of cognitive testing.
ASSIGNMENT DUE TODAY: Finish collecting data. Recommended: work on methods section
of paper.
CLASS ACTIVITY: Discuss the article you found, especially analysis procedures. Group
practice analyzing data
6.
THURSDAY SEPTEMBER 29
Exploratory Factor Analysis and Latent Variables
Internal Structure as a Source of Validity Evidence, continued
READINGS:
Pett, Lackey, & Sullivan, Chapters 1 and 3
DeVellis, Chapter 2
7
Use the readings to answer the following questions:
What is the purpose of exploratory factor analysis? (EFA)
What are the primary methodological choices to be made when conducting an EFA?
What methods can you use to assess the factorability of your data?
What are the evaluation criteria for each of those methods?
What are the primary EFA output components?
How do you evaluate each of the primary EFA output components?
How could you use the readings to support your choice of EFA methods and the evaluation
of your EFA results?
CLASS ACTIVITY: Discussion of latent variables in your area of research, and in your
cognitively tested Scale. Introduction to EFA with SPSS
7.
THURSDAY OCTOBER 6
Exploratory Factor Analysis, continued
Internal Structure as a Source of Validity Evidence, continued
READINGS:
Bowen, N. K. (2006). Psychometric properties of the Elementary School Success
Profile for Children. Social Work Research, 30, 51-63.
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor
analysis: Four recommendations for getting the most from your analysis.
Practical Assessment, Research & Evaluation, 10, 1-9.
DeVellis, chapter 6, pp. 102-137.
Pett, Lackey, & Sullivan, Chapters 4 and 5
CLASS ACTIVITY: Present highlights of cognitive testing paper; EFA with SPSS in the lab.
ASSIGNMENT DUE: Cognitive testing paper
ASSIGNMENT HANDED OUT: EFA article critique
8.
THURSDAY OCTOBER 13
Exploratory Factor Analysis, continued
Internal Structure as a Source of Validity Evidence, continued
READING:
Find and read two EFA articles in your area of research. Use one for your critique.
Pett, Lackey, & Sullivan, Chapter 6 up to p. 174, Chapter 7
CLASS ACTIVITY: Working with problematic EFA results.
ASSIGNMENT DUE TODAY: EFA article critique
ASSIGNMENT HANDED OUT: EFA homework
OCTOBER 20
FALL BREAK NO CLASS
8
9.
THURSDAY OCTOBER 27
Item Response Theory
READINGS:
DeVellis, Chapter 7
Others TBA
CLASS ACTIVITY: GUEST LECTURE, DR. MICHAEL LAMBERT
ASSIGNMENT DUE TODAY: EFA homework
10.
THURSDAY NOVEMBER 3
Reliability and Relations to other Variables
Internal Structure as a Source of Validity Evidence, continued
Relations to other Variables as a Source of Validity Evidence
READINGS:
Devellis, Chapter 3
Baugh, F. (2003). Correcting effect sizes for score reliability. In B. Thompson
(Ed.), Score reliability: Contemporary thinking on reliability issues (pp.
31-41). Thousand Oaks: Sage Publications.
Pett, Lackey, & Sullivan, Chapter 6 from p. 174 to end
Thompson, B. (2003). Understanding reliability and coefficient alpha, really. In B.
Thompson (Ed.), Score reliability: Contemporary thinking on reliability
issues (pp. 3-23). Thousand Oaks, CA: Sage Publications.
Use the readings to answer the following questions:
What is measurement reliability? How does it differ from measurement validity?
What different kinds of reliability exist?
What type of reliability is most commonly reported? Why?
What internal consistency reliability cutoff would you use and how would you justify it?
CLASS ACTIVITY: Demonstration of reliability with SPSS. Working with problematic
reliability results.
ASSIGNMENT HANDED OUT: Reliability homework
11.
THURSDAY NOVEMBER 10
Confirmatory Factor Analysis
Internal Structure as a Source of Validity Evidence, continued
READINGS:
Bowen & Guo (2011). Structural Equation Modeling. Oxford University Press,
Chapters 1 and 2.
9
Hoyle, R. H. (1995). The structural equation modeling approach: Basic concepts
and fundamental issues. In R. H. Hoyle (Ed.), Structural equation
modeling: Concepts, issues, and applications (pp. 1-15). Thousand Oaks,
CA: Sage Publications.
Use the readings to answer the following questions:
How does confirmatory factor analysis (CFA) differ from EFA in terms of purpose, methods,
and evaluation criteria? Which evaluation criteria apply to both methods?
What is the purpose of CFA?
What is model identification (just conceptually)?
What are the primary methodological choices to be made when conducting a CFA?
What are the primary CFA output components?
How do you evaluate each of the primary CFA output components?
How could you use the readings to support your choice of CFA methods and the evaluation
of your CFA results?
CLASS ACTIVITY: Specification of CFA models with graphics. Demonstration of CFA with
AMOS
ASSIGNMENT DUE: Reliability homework
ASSIGNMENT HANDED OUT: CFA article critique
11.
THURSDAY NOVEMBER 17
Confirmatory Factor Analysis, data preparation and best practices
READINGS:
Find and read two CFA articles in your area of interest. Choose one for your
critique.
Other reading to be announced
CLASS ACTIVITY: CFA with Amos in the lab; Running and interpreting models
ASSIGNMENT DUE: CFA article critique
ASSIGNMENT HANDED OUT: CFA homework
NOVEMBER 24 THANKSGIVING NO CLASS
13.
THURSDAY DECEMBER 1
Confirmatory Factor Analysis, multiple group analysis
Statistical Evidence of Validity
Consequences of Testing as a Source of Validity Evidence
10
READINGS:
Bowen, N. K., Bowen, G. L., & Ware, W. B. (2002). Neighborhood
disorganization, families, and the educational behavior of adolescents.
Journal of Adolescent Research, 17, 468-490.
Bowen, N. K., & Powers, J. D. (2005). Knowledge gaps among school staff and the role
of high quality ecological assessments in schools. Research on Social Work
Practice, 15, 491-500.
Kane, M. (2009). Validating the interpretations and uses of test scores. In R. W.
Lissitz (Ed.). The concept of validity: Revisions, new directions, and
applications (pp. 19-37). Charlotte, NC: Information Age Publishing.
CLASS ACTIVITY: Demonstration of multiple group CFA
ASSIGNMENT DUE TODAY: CFA homework
ASSIGNMENT HANDED OUT: Validity homework
THURSDAY DECEMBER 8: NO CLASS
ASSIGNMENT DUE TODAY: Validity homework
11