THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL
SCHOOL OF SOCIAL WORK
COURSE NUMBER: SOWO 910
COURSE TITLE:
Research Methods for Social Intervention
SEMESTER & YEAR: Fall, 2010
INSTRUCTOR:
Mark Testa, PhD
Spears-Turner Distinguished Professor
School of Social Work
University of North Carolina at Chapel Hill
Tate-Turner-Kuralt Building
325 Pittsboro St., Campus Box 3550
Chapel Hill, NC 25599-3550
Tel: (919) 962-6496 Fax: (919) 962-1486
mtesta@unc.edu
OFFICE HOURS:
Fridays, 12:00 – 1:30 PM or by appointment
COURSE DESCRIPTION:
This course provides an introduction to basic research processes and methods for use in
planning, implementing, evaluating, and improving social interventions. Topics include
outcomes monitoring, problem formulation, assessment, measurement, research review, human
subjects’ protection, evaluation design, data analysis, and the application of findings to practice
improvement and theory refinement.
COURSE OBJECTIVES:
This course affords students an opportunity to gain knowledge about the following issues in
social intervention research:
•
•
•
•
•
The need for broadly inclusive processes to plan, implement, and evaluate social
interventions, and how researchers' approaches to these processes can facilitate or impede
research
The quantitative-comparative experimental (counterfactual) paradigm that currently
prevails in social intervention research;
How various policy and implementation constraints often necessitate the use of designs
other than fully randomized experiments;
Special legal and ethical issues pertaining to the protection of human subjects; and
The need for culturally aware social intervention research that is responsive to the
diversity of community values and preferences.
1
Students taking the course will be able to:
•
•
•
•
•
•
develop “well-built” research questions for estimating the causal impact of social
interventions on desired outcomes for target populations;
develop conceptual models to support proposed social interventions and explicate
underlying theories of change;
assess the validity and reliability of alternative qualitative and quantitative measures of
constructs in conceptual models that guide social intervention research;
understand basic aspects of data analysis, sample design and statistical power analysis;
critically evaluate various experimental, quasi-experimental, and non-experimental
research designs by identifying various threats to the validity of each design;
and prepare an application for IRB approval of human subjects research.
Recommended Prerequisites
SoWo 102 or equivalent
SoWo 292 or equivalent
Relationship to Other Courses in the Curriculum
This course introduces basic concepts and skills that are reinforced and further developed
in Development of Social Intervention Models (SOWO 940) and Advanced Research Methods in
Social Intervention (SOWO 913) during the 2nd and 3rd years of the program. It extends students’
knowledge about basic research processes and orients them to the variety of professional issues,
theoretical perspectives, statistical methods, and advanced methods of intervention research that
they will examine in related doctoral-level courses (SOWO 900, 911, & 914) and can pursue in
courses outside of the School.
REQUIRED TEXTS:
Testa, M. F. & Poertner, J. (2010). Fostering accountability: Using evidence to guide and
improve child welfare policy. Oxford: Oxford University Press.
Shadish, W.R, Cook, T.D., & Campbell, D.T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin
Company.
REQUIRED AND SUPPLEMENTARY READINGS:
Required readings outside of the textbook are listed in the course outline. Supplementary
readings are assigned at the instructor’s discretion.
2
TEACHING METHODS
The development of a supportive learning environment, reflecting the values of the social
work profession, is essential for the success of this class. A supportive learning environment is
fostered by listening to the ideas and views of others, being able to understand and appreciate a
point of view which is different from your own, articulating clearly your point of view, and
linking experience to readings and assignments. Everyone will appreciate your contributions to
making this a safe and respectful class for learning and growth.
CLASS ASSIGNMENTS
SEMINAR PARTICIPATION
1. Register with JSSWR: Sign up as a subscriber to the Journal of the Society for Social
Work and Research (edited by our own, Mark Fraser) by registering online. September
3
2. Obtain Certification in Protection of Human Participants: Students must complete
the on-line course developed by the Collaborative IRB Training Initiative (CITI) to
certify that they are familiar with the ethical principles and guidelines governing research
involving human participants; completing the course requires several hours; each student
is required to bring a completion certificate to class on September 10 (this assignment is
required, but is not graded).
3. Identify potential resources and collaborators. Become familiar with potential sources
of information about obtaining financial support for your research. Also you may wish to
make yourself known to potential research collaborators by registering as a member of
the Community of Science. September 17.
4. Participate in seminar: Students are expected to read assignments and come to class
prepared to ask questions and to respond to questions about the material. Each student
should prepare a 1-page self-evaluation of his or her efforts to make this a productive
learning experience. Award yourself 1-5 points for those efforts (see description of
grading). This is due the last day of class (December 3) and should be submitted to the
instructor by email.
WRITING ASSIGNMENTS
During the semester, you will write five papers (varying in length from 1 to 7 pages) on
different components of the research process. The final assignment will require revision and
expansion of each of these components into a full IRB application. The written assignments are
as follows:
1. Research Question: After meeting with the instructor to discuss preliminary ideas, each
student will formulate a “well-built” PICO question that consists of the following
components: 1) the target population about which you wish to draw inferences; 2) the
outcome you intend to achieve or problem you hope to address; 3) the intervention you
3
2.
3.
4.
5.
6.
are interested in evaluating; 4) and the alternative course of action with which you will
draw a comparison (e.g., no intervention, regular services, or different interventions).
This one (1) - pager, including a brief statement about the significance of the problem or
outcome you are studying, will be due September 17.
Research Review: Conduct a computerized search of electronic databases using
keywords from your PICO question. Select the strongest four to six (4-6) studies that bear
on your topic. Write up a research review using narrative descriptions that also assess the
strength of the evidence for supporting the use of your intervention and identify the
limitations of these studies and their applicability to your population. This paper (5-7
pages) will be due October 1.
Logic Model: Expand your PICO question into a logic model that lays out the expected
causal mechanisms and mediating pathways from the intervention to the desired outcome.
Your description of the target population should also list any population conditions that
you believe may moderate the intervention’s impact on the outcome. The model should
also enumerate any background factors and external conditions that contribute to the
significance of problem you are addressing. It should list the key assumptions of the
theory of action you are positing will effectuate the desired change. Finally the model
should identify general end-values for reconciling diverse outcomes and evaluating the
ultimate worth of the resulting change. This one (1) page figure will be due October 15.
Measurement Review: Identify and assess the relative strengths and weaknesses
of alternative measures or approaches to measuring a construct (population, intervention,
or outcome) referenced in your research question. The purpose of the exercise is to select
the measure (or set of measures) that will yield the most valid and reliable data
concerning this construct. This paper (5-7 pages) will be due October 29.
Evaluation Design: Based on your research question, you will outline the basic features
of an experimental or quasi-experimental design for evaluating the impact of the
identified social intervention on your outcome construct. The description should identify
the unit(s) of analysis, comparison group(s), and how threats to the validity of your
research will be addressed. The discussion should provide a rationale for the method you
are proposing and how concessions to design constraints may make the research
vulnerable to criticism. This 5-7 page paper will be due November 12.
IRB Application: This final assignment will follow the instructions issued by the UNC
Office of Human Research Ethics for application for IRB approval. We will be reviewing
various resources throughout the course that will be helpful in preparing it. It will
incorporate material from each of the other written assignments as well as a description
of risks to human subjects and measures to minimize those risks and a discussion of the
benefits to subjects and/or society. The application form can be downloaded from
http://research.unc.edu/offices/human-research-ethics/researchers/forms/index.htm. The
Word document runs 18 pages and with the narrative can expand to 25-30 pages. This
application will be due on December 3.
4
WRITING ASSIGNMENT GUIDELINES:
All written assignments must be typed and follow APA format. Several writing resources
are posted on the website. Students should also refer to the following:
• American Psychological Association. (2009). Publication manual of the American
Psychological Association (6th ed.). Washington, DC.
• Note: You can find a self-paced tutorial for APA style at
http://www.lib.unc.edu/instruct/citations/apa/index.html
ONLINE RESOURCES:
Course materials for SOWO 910 will be accessible to you on http://blackboard.unc.edu/.
In addition, Angela Bardeen, Behavioral and Social Sciences Librarian at UNC, has created a
SOWO 910 course library website for finding articles, evidence-based practice resources,
measurement tools, and human subjects protection. It can be accessed at
http://ris.lib.unc.edu/course-guide/124.
COURSE PERFORMANCE ASSESSMENT:
Final grades will be determined on the basis of points earned on each assignment and on
participation in seminars. Letter grades will correspond to the following point totals:
94 - 100:
80 - 93:
70 - 79:
< 70:
High Pass
Pass
Low Pass
Fail
Points are apportioned to individual assignments as follows:
1.
2.
3.
4.
5.
6.
7.
Seminar participation: 10 points (5 points by instructor and 5 points by self-evaluation)
Research Question: 10 points
Research Review: 15 points
Logic Model: 10 points
Outcome Measurements: 15 points
Evaluation Design: 15 points
IRB Application: 25 points
POLICY ON INCOMPLETES AND LATE ASSIGNMENTS
The instructor will only entertain requests to hand in assignments late in circumstances of
special hardships or emergencies. The potential for handing in papers late needs to be discussed
with the instructor in person at least three days before the assignment is due. If not approved
beforehand, late assignments will be graded five points off for each day the assignment is late,
including weekends. A grade of Incomplete is given on rare occasions when there is sufficient
reason to warrant it. It is the student’s responsibility to initiate a conversation with the instructor
to request an Incomplete—instructors have no responsibility to give an Incomplete without such
a request.
5
POLICY ON ACADEMIC DISHONESTY
Please refer to SSW Writing Resources and References website for information on
attribution of quotes, plagiarism and appropriate use of assistance in preparing assignments. All
written assignments should contain a signed pledge from you stating that, "I have not given or
received unauthorized aid in preparing this written work".
In keeping with the UNC Honor Code, if reason exists to believe that academic
dishonesty has occurred, a referral will be made to the Office of the Student Attorney General for
investigation and further action as required.
POLICY ON ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES
Students with disabilities that affect their participation in the course and who wish to
have special accommodations should contact the University’s Disabilities Services and provide
documentation of their disability. Disabilities Services will notify the instructor that the student
has a documented disability and may require accommodations. Students should discuss the
specific accommodations they require (e.g. changes in instructional format, examination format)
directly with the instructor.
OUTLINE OF CLASS TOPICS
Date
Topics for Discussion
Reading Assignments Completed
before Class
UNIT I: ISSUES IN SOCIAL INTERVENTION RESEARCH
Week 1
Introduction and Overview of Social
(August 27)
Intervention Research
Week 2
PICO & Logic Models
Testa & Poertner, pp. 75-100; Kenny et
(September 3)
al., pp. 294-324.
Week 3
Scientific Validity & Agency
Testa & Poertner, pp. 3-13; Shadish,
(September 10) Accountability
Cook & Campbell, pp. 34-42; Merton,
pp. 267–276.
UNIT II: STAGES OF INTERVENTION RESEARCH
Week 4
Research Reviews
Testa & Poertner, pp. 166-194; Shadish,
(September 17)
Cook & Campbell, pp. 417-455; Littell,
pp. 66-93.
Week 5
Outcomes Monitoring
Testa & Poertner, pp. 114-135;
(September 24)
Goodwin, pp.100-106.
Week 6
Sampling Designs & Power Analysis Testa & Poertner, pp. 147-152; Orme et
(October 1)
al., pp. 3-10.
Week 7
Data Analysis
Testa & Poertner, pp. 136-147, 153-165;
(October 8)
Yule, pp. 249-295.
6
Week 8
(October 15)
Week 9
(October 29)
Week 10
(November 5)
Evaluation Designs
Testa & Poertner, pp. 269-290
Freedman, et al., pp. 3-28.
UNIT III: INTERVENTION RESEARCH DESIGNS
Experimental Designs
Testa & Poertner, pp. 195-205;
Boruch, et al. pp. 330-353.
Quasi-experimental Designs
Testa & Poertner, pp. 205-230;
Angrist, pp. 23-44.
UNIT IV: RESULTS-ORIENTED ACCOUNTABILITY
Human Subjects Protections
The Belmont Report
Week 11
(November 12)
Week 12
Quality Improvement
(November 19)
Week 13
Qualitative Research and Reflexive
(December 3)
Practice
Testa & Poertner, pp. 231-268, 291-327;
Moynihan, pp. 203—216.
Testa & Poertner, pp. 357-379; D’Cruz
et al., pp.73–90.
UNIT I. ISSUES IN SOCIAL INTERVENTION RESEARCH
Week 1 (August 27)— INTRODUCTION AND OVERVIEW
Social intervention research serves three distinct but interrelated purposes: 1) developing or
shaping a social intervention to improve practice and policy; 2) rendering a summary judgment
of the efficacy and effectiveness of a social intervention; and 3) describing and explaining the
effects of a social intervention as a contribution to scientific knowledge and theory. Although the
research process varies somewhat depending on the specific purpose, in general, it builds on a
common foundation that conceives of social intervention research as cycling through the
following five successive stages: 1) outcomes monitoring, 2) data analysis, 3) research review, 4)
program evaluation, and 5) quality improvement.
The course is organized around these five stages of social intervention research, which align with
five principles of agency accountability and four types of scientific validity as follows:
7
5
1
Quality
Improvement
Outcomes
Monitoring
(Reflexivity/
Construct validity)
(Scope of interest/
Construct validity)
4
2
Evaluation
Data Analysis
(Causality/
Internal validity)
(Transparency/
Statistical validity)
3
Research
Review
(Evidence-supported/
External validity)
During the semester, we will discuss the four types of scientific validity and reflect upon the
implications of the five principles of accountability for conducting socially responsible and
culturally sensitive social intervention research.
Your engagement with these issues begins by identifying yourself to potential collaborators by
registering as a member of the Community of Science.
Week 2 (September 3)— CONCEPTUAL FRAMEWORKS: PICO & LOGIC MODELS
(Guest Instructor, Angela Bardeen, Behavioral and Social Sciences Librarian, UNC)
The research process begins with the formulation of a well-built research question that can be
parsed into the four components of population, intervention, comparison, and outcome. Grouped
together under the acronym PICO, the formulation of a well-built question guides the process of
computerized research reviews and the construction of logic models. There are a variety of
designs for constructing logic models. The design used in this course elaborates on the PICO
framework and visually depicts the mediating activities that link interventions and population
conditions to the short-term outputs produced by the activities and the longer-term outcomes
these outputs are expected to yield. A full logic model also identifies the external conditions that
prompted concern over the problem, the underlying theory of change, end-values for evaluating
the ultimate worth of the resulting change, and any moderating conditions that may affect the
intervention’s impact.
8
You should come to class with a PICO formulation of the research question addressed by Kenny
et al. (2004) and a logic model of the intervention they studied using the template described in
Testa and Poertner (2010, p. 86). We will be joined by Angela Bardeen, Behavioral and Social
Sciences Librarian, who will provide an overview of library resources for conducting research
reviews.
Required Readings:
Testa & Poertner, pp. 75-100.
Kenny, D.A. et al. (2004). Evaluation of treatment programs for persons with severe mental
illness: Moderator and mediator effects. Evaluation Review, 28, 294-324.
Supplementary Readings:
The Evaluation Exchange, 11(2): pp. 2-15.
Week 3 (September 10)—SCIENTIFIC VALIDITY AND AGENCY INTEGRITY
Social intervention research entails working with people--those who design and implement
interventions (agents) and the groups that provide funding for testing them and the
participants who are affected by the interventions (principals). Best practice involves holding
agents answerable for the validity and integrity of the actions they take on behalf of these
individual and corporate principals. As such, social intervention research is an “agency
relationship” in which agents are responsible for abiding by a set of scientific standards and
ethical principles in fulfilling the interests of their principals. In this respect, research
accountability goes beyond accumulating valid evidence of the efficacy and effectiveness of
child welfare interventions to ensuring that the research process reliably and responsibly serves
the results valued by clients, research sponsors, and the public at large. To be accountable, social
intervention researchers must appreciate the context in which they work and develop the
interpersonal skills as well as the technical knowledge and skills that can make them both
effective and responsible in their roles.
Scientific validity refers to the best available approximation to the truth or falsity of a given
hypothesis, inference, or conclusion and provides a set of standards by which the quality of the
research can be judged. There are four generally recognized types of scientific validity: 1)
statistical conclusion validity that is concerned with whether there is a statistically significant
association between an intervention and the desired outcome; 2) internal validity that focuses on
whether the statistical association results from a causal relationship between the intervention and
the outcome or is a spurious association; 3) construct validity that assesses the degree of
correspondence between the observational particulars of population, intervention, comparison,
and outcome (PICO) and their higher-order constructs; and 4) external validity that addresses
9
how generalizable the particular causal relationships are over variations in PICO. The
demonstration of scientific validity usually proceeds cumulatively and the order of
demonstration varies with the purpose or purposes served by the social intervention research. For
example, for purposes of rendering a summary judgment of the efficacy and effectiveness of a
social intervention, the cumulative order typically proceeds from statistical conclusion validity to
internal validity, construct validity and finally to external validity.
Agency integrity refers to how reliability and responsibly the research enterprise fulfills the
interests of its principals. In this respect, integrity is greater when human beings are treated as
“participants” in research rather than as “objects” of research. Your engagement with these
issues begins with completing the on-line course related to the protection of human participants
in research. Each student will submit a CITI certificate indicating their completion of the course.
This certification and the automatic registration with UNC's Office of Human Research
Ethics satisfy the requirements of many of the research assistantships held by students in the
doctoral program.
Required Readings:
Testa & Poertner, pp. 3-13.
Shadish, Cook & Campbell, pp. 34-42.
Merton, R. K. (1996). The ethos of science. In P. Sztompka (Ed.), Robert K. Merton: On social
structure and science (pp. 267–276). Chicago: The University of Chicago Press.
Supplementary Readings:
NASW (1997). Code of Ethics, section 5.02, Evaluation and research (available at
http://www.socialworkers.org/pubs/code/code.asp).
UNIT II: STAGES OF INTERVENTION RESEARCH
Week 4 (September 17)—RESEARCH REVIEWS
Research reviews involve the explicit search and selection of relevant studies, the assessment of
their scientific validity, and a critical synthesis of their findings to reach a tentative conclusion
about the efficacy and effectiveness of social interventions. This stage of research-oriented
accountability is what the field most commonly understands as evidence-based practice (EBP). A
commonly cited definition by Sackett et al. (1997) is that EBP is “the conscientious, explicit and
10
judicious use of current best evidence in making decisions about the care of individual [clients].”
EBP is emerging also as the guiding paradigm within which social intervention research is
pursued by university-based researchers and applied by various actors in the areas of medicine,
psychology, public health, criminology, social work and public policy. While paradigms provide
a framework within which knowledge can be developed, they can also intellectually straitjacket
their adherents. Given this potential agency risk, there is a need to subject EBP to the same
methods of critical review and reflective assessment that its adherents espouse for validating
clinical practices and public policies.
Required Readings:
Testa & Poertner, pp. 166-194.
Shadish, Cook & Campbell, pp. 417-455.
Littell, J. (2008). Ch. 4: How do we know what works? The quality of published reviews of
evidence-based practices. In Lindsey, D. & Shlonsky, A. (Eds.) Child welfare research:
Advances for practice and policy (pp. 66-93). Oxford: Oxford University Press.
Supplementary Readings:
Chatterji, M. (2007). Grades of evidence: Variability in quality of findings in effectiveness
studies of complex field interventions. American Journal of Evaluation, 28, 239-255.
Oosterman, M., Schuengel, C., Slot, N. W., Bullens, R.A.R, &, Doreleijers, T.A.H. (2007).
Disruptions in foster care: A review and meta-analysis. Children and Youth Services
Review, 29, 53–76.
Winokur, M., Holtan, A., & Valentine, D. (2009) Kinship care for the safety, permanency, and
well-being of children removed from the home for maltreatment. Cochrane Database of
Systematic Reviews, 1, Art.No.: CD006546. DOI:10.1002/14651858.CD006546.pub2.
Webb, S.A. (2001). Some considerations on the validity of evidence-based practice in social
work. British Journal of Social Work, 31: 57-79.
Gibbs, L. & Gambrill, E. (2002). Evidence-based practice: Counterarguments to objections.
Research on Social Work Practice, 12: 452-476.
11
Week 5 (September 24)—OUTCOMES MONITORING
The construction of a well-built research question begins with the identification and
measurement of an outcome or set of outcomes that principals and their agents want to monitor
and, if desired, change. Outcome is synonymous with the effect, result or dependent variable of a
cause, intervention or independent variable. In mathematical terms, it is y or a function of x: y =
f(x). In this course we will also substitute O for y in keeping with the PICO framework. In order
to estimate the effect of an intervention on an outcome, it is necessary to translate the higherorder, theoretical construct of the outcome into a lower-order, operational measurement of the
variable. Construct validity refers then to the degree of correspondence between the higher order,
theoretical constructs and the lower-order, observational particulars. To demonstrate construct
validity, you need to show evidence that the observational particulars (data) support the
theoretical structure of the construct. How researchers conceptualize this task is evolving, and we
will review some of the latest thinking on these endeavors.
Some researchers oppose the routine use of outcome indicators to improve public management.
They argue that monitoring and evaluating agency units, e.g., schools, departments, and courts,
invariably lead to the corruption of the indicators used to monitor results and to the degradation
of the agency relationships that program evaluation is supposed to improve. It is important to
acknowledge these agency risks and take cognizance of such threats to agency integrity. We will
consider the precautions that can be taken which can help decrease these agency risks and
increase the opportunities for responsible public policy and management.
Required Readings:
Testa & Poertner, pp. 114-135.
Goodwin, L. (2002). Changing conceptions of measurement validity: An update on the new
Standards. Journal of Nursing Education, 41, 100-106.
Supplementary Readings:
Manly, J.T. (2005). Advances in research definitions of child maltreatment, Child Abuse &
Neglect 29, 425–439.
Pelton, L. (2008). A note contesting Mark Testa's version of national foster care population
trends, Children and Youth Services Review, doi:10.1016/ j.childyouth.2008.09.005.
Testa, M. (2008). How the bear evolved into a whale: A rejoinder to Leroy Pelton's note
contesting Mark Testa's version of national foster care population trends, Children and
Youth Services Review, doi:10.1016/ j.childyouth.2008.10.009.
12
Week 6 (October 1)—SAMPLING DESIGNS AND POWER ANALYSIS
(Guest Instructor: Rod Rose, Research Associate and Project Director, SSW-UNC)
When steady progress is being made toward remedying a problem or attaining a desired
outcome, the monitoring process continues another round of assessment and review. Otherwise,
the detection of a worrisome gap between a valued result and an observed outcome may signal
the need for some corrective action or at least a reasonable accounting for the shortfall. When the
gap is of both practical importance and statistical significance, data analysis can be initiated to
identify the areas for improvement and the risk factors that may be contributing to the result. The
first step in assessing whether a gap is worrisome enough is to decide on the smallest difference
or distance that is deemed important enough to matter. This is best done in conjunction with
practitioners and administrators who are able to draw comparisons from experience, history, an
established standard, or some other reference point. With the availability of large administrative
databases, an assessment of practical importance can proceed without much concern for the
statistical significance of the conclusion. However when information is based on a sample from
a larger population, there is always a chance that the conclusion will be wrong due to the lack of
complete information about the population. Sampling theory provides a basis for evaluating the
chances of error and for taking those risks into account when designing a study.
Statistical power is a concept that has been commanding greater attention in social intervention
research. It refers to the probability that a real difference, effect size, or pattern of association of
a certain magnitude in a population will be detectable in a particular study. Power analysis
proceeds from assumptions about available sample size and the amount of error one is willing to
tolerate. It can also be used to determine sample size. To do this, we set an effect size that we
want to be able to detect and select a high probability (typically, 80%) that a statistical test will
reject the null hypothesis when the hypothesis of no effect is false. In this section, we will look at
some statistical power software that is typically used to determine the sample size necessary to
demonstrate an intervention’s effect. We will be joined by Rod Rose, Research Associate and
Project Director at the Jordan Institute for Families, who will provide an overview of sampling
design and power analysis for social interventions research.
Required Reading:
Testa & Poertner, pp. 147-152.
Orme, J. G. & Combs-Orme, T. D. (1986). Statistical power and type II errors in social work
research. Social Work Research & Abstracts, 22, 3-10.
Supplementary Readings:
Boruch, R. (2007). The null hypothesis is not called that for nothing: Statistical tests in
randomized trials. Journal of Experimental Criminology, 3, 1–20.
13
Week 7 (October 8)—DATA ANALYSIS
The conclusion that an observed difference or distance is of both practical importance and
statistical significance does not necessarily indicate that agency performance is deficient or in
need of correction. There could be some other antecedent conditions beyond the practitioners’
and agents’ immediate control that could be causing the difference. For example, a practically
important and statistically significant difference in post-operative mortality rates between two
hospitals may not necessarily mean that the hospital with the lower rate is the superior
performing hospital. It may instead mean that the hospital with the higher rate admits a sicker
group of patients, on average, than the other. Therefore before a summary judgment can be
rendered about agency performance or the type of improvement needed, it is first important to
identify those factors that may be clouding the comparison with data analysis methods for
purging or adjusting for external confounding influences.
Required Readings:
Testa & Poertner, pp. 136-147, 153-165.
Yule, G. U. (1899). An investigation into the causes of changes in pauperism in England, chiefly
during the last two intercensal decades (part I.). Journal of the Royal Statistical Society,
62, (2), 249-295.
Week 8 (October 15)—EVALUATION DESIGNS
The question that clients, practitioners and policymakers really want to answer from social
intervention research is what would happen if people who are given a certain treatment or
intervention option were instead denied this possibility. Of course such comparisons can never
be made at the individual level because it is impossible simultaneously to deny and to offer a
treatment option to each individual or group. Instead researchers have to fall back on high quality
approximations to this impossible “what if” scenario (what statisticians call the “counterfactual”)
by conducting rigorous studies that allow them to draw causal inferences at the group level.
A compelling case can be made for more routine use of randomized controlled experiments in
social work than is currently the practice. But there are situations in which controlled
experimentation is inadvisable, unethical, or just plain impossible. Over the past several
decades, researchers have made tremendous strides in conceptualizing the assumptions that need
to be satisfied in order to draw valid causal inferences from non-experimental research and how
best to approximate the necessary conditions using statistical methods. The material in this
section will help students acquire technical knowledge related to the development and
14
application of different research designs—experimental, quasi-experimental, and nonexperimental.
Required Readings:
Testa & Poertner, pp. 269-290
Freedman, D., Pisani, R. & Purves, R. (2007). Controlled experiments & Observational Studies.
Statistics- 4th edition, (pp. 3-28). New York: W.W. Norton & Company.
Supplementary Readings:
Rubin, D.B. (2004). Teaching statistical inference for causal effects in experiments and
observational studies. Journal of Educational and Behavioral Statistics, 29 (3), 343-367.
October 22: Fall Break – No Class
UNIT III.
INTERVENTION RESEARCH DESIGNS
Week 9 (October 29)—EXPERIMENTAL DESIGNS
The most rigorous method for drawing causal inferences about the potential effects of a
treatment is the randomized controlled experiment. By employing a process such as a lottery,
coin flips or table of random numbers to treat a chance selection of persons or group, we can
feel confident that the characteristics of the intervention group (who are offered the option) and
the comparison group (who are denied the option) are statistically equivalent on average within
the bounds of statistical error. The thinking is that if the two groups start out looking statistically
similar at the initiation of the experiment and then if any significant differences later emerge
after implementation, it is reasonable to conclude that the cause of the differences is the
intervention itself rather than any pre-existing group dissimilarities or concurrent policy changes
(which affect both groups).
Despite these advantages, some social work practitioners find randomized controlled
experiments to be ethically suspect and of limited use because of the lengthy observational
period before a summative judgment can be confidently rendered. This section will consider the
appropriateness of constraining practitioner discretion by experimental protocols and the
conditions under which randomization constitutes a justifiable interference with agent discretion
15
when the empirical evidence for the efficacy or effectiveness of a promising intervention is
weak.
Required Readings:
Testa & Poertner, pp. 195-205.
Shadish, Cook & Campbell, pp. 279-291.
Boruch, R. F., Victor, T. & Cecil, J.S. (2000). Resolving Ethical and Legal Problems in
Randomized Experiments. Crime Delinquency, 46, 330-353.
Supplementary Readings:
Testa, M. (2002). Subsidized guardianship: Testing an idea whose time has finally come. Social
Work Research, 26 (3), 145-158.
Brotman, L. M., O’Neal, C. R., Huang, K., Gouley, K. K., Rosenfelt, A., & Shrout,P. E. (2009).
An experimental test of parenting practices as a mediator of early childhood physical
aggression. The Journal of Child Psychology and Psychiatry, 50 (3), 235—245.
Week 10 (November 5)—QUASI-EXPERIMENTAL DESIGNS
Although experiential research is the “gold standard” in social work and many related fields such
as medicine, education, mental health, and criminology, randomization in and of itself provides
no guarantee of construct or statistical validity. Sample attrition, failure to receive the intended
intervention, and crossovers from the comparison to the intervention group can result in
misestimates of the actual treatment effect. Many randomized controlled experiments in social
work are actually better understood as “randomized encouragement designs” that involve the
randomization of subjects to an encouragement condition, which is intended to induce
compliance with an intended plan of treatment. As a consequence, studies that start out as
experimental can up ‘‘quasi-experimental’’ at the end because of differential selection into
alternative compliance states (e.g. compliers, defiers, always treated, never treated).
Evaluation methods that approximate the statistical equivalence that is best obtained through
random assignment are called quasi-experiments. Econometricians and statisticians have
developed a variety of statistical methods for handling selection biases in order to uncover the
genuine causal effect of an intervention on an outcome. The most commonly used approaches for
adjusting for selection biases in the absence of randomization include linear regression analysis,
16
propensity score and other matching methods, and instrumental variable analysis. This section
will also look at recent developments in the estimation of intent-to-treat (ITT) and treatment-ontreated (TOT) effects in randomized encouragement designs.
Required Readings:
Testa & Poertner, pp. 205-230.
Angrist, J.D. (2006). Instrumental variables methods in experimental criminological research:
What, why and how. Journal of Experimental Criminology, 2, 23-44.
Supplementary Readings:
Doyle, J. J. (2007). Can’t buy me love? Subsidizing the care of related children. Journal of
Public Economics, 91, 281—304.
Grossman, J. & Tierney, J.P. (1993). The fallibility of comparison groups. Evaluation Review,
17: 556 - 571.
Lehman, C. M., Liang, S. & O'Dell, K. (2005). Impact of flexible funds on placement and
permanency outcomes for children in child welfare. Research on Social Work Practice,
15, 381-388.
UNIT IV: AGENCY ACCOUNTABILITY
Week 11 (November 12)—HUMAN SUBJECTS PROTECTION
(Guest Instructor, Mary Anne Salmon, Clinical Associate Professor, UNC)
This section will provide the “nuts-and-bolts” on how to prepare a UNC application for IRB
approval of research proposals. We will be joined by Mary Anne Salmon who will provide an
overview of the IRB application process at the School. After this, you should have the tools and
knowledge needed to complete the final assignment.
17
Required Readings:
The National Commission for the Protection of Human Subjects of Biomedical and Behavioral
Research (1979). The Belmont Report: Ethical Principles and Guidelines for the
Protection of Human Subjects of Research.
Supplementary Readings:
White, V. M., Hill, D.J. & Effendi, Y. (2004). How does active parental consent influence the
findings of drug-use surveys in schools? Evaluation Review, 28, 246-260.
Week 12 (November 19)— QUALITY IMPROVEMENT
Quality improvement requires that researchers move beyond causal description and acquire a
behavioral and deeper interpretative understanding of the intervening processes by which
program resources are disbursed within a social system and transformed into program outputs
and client outcomes. Initially quality improvement takes the form of repeating a ”single-loop”
learning cycle several times as needed to ensure that the intervention is supplied in sufficient
dosage and with adequate fidelity to the original program design. If the results of the formative
evaluations fall short of the desired targets, show no change, or else run contrary to other endvalues, quality improvement takes the form of a ”double-loop” leaning cycle in which the
existing theory of action and its assumptions and values are questioned and reflexively changed.
Required Readings:
Testa & Poertner, pp. 231-268, 291-327.
Moynihan, D. P. (2005). Goal-based learning and the future of performance management. Public
Administration Review, 65(2), 203—216.
Supplementary Readings:
McBeath, B. & Meezan, W. (2009) Governance in motion: Service provision and child welfare
outcomes in a performance-based, managed care contracting environment. Journal of
Public Administration Research and Theory. The State of Agents: Special Issue, 20, i101i123.
November 26 University Holiday – No Class
18
Week 13 (December 3)— QUALITATIVE RESEARCH AND REFLEXIVE PRACTICE
Some of the misunderstanding and distrust that occurs during the conduct of research arises from
the failures of researchers to acquire an interpretative understanding of client values and
perspectives and to take account of those preferences. To establish the mutual trust upon which a
research enterprise must rest, it is important to create recurring opportunities for the impulses,
desires, and values of the intended beneficiaries to enter into the design of a social intervention
and continuous revision of existing routines in light of new knowledge about the impact of those
practices. The coordination of the quantitative results with the qualitative feedback from clients,
practitioners, and researchers through peer review and “learning forums” constitutes an essential
ingredient in the integration of ethical, evidentiary, and practical concerns that is necessary for
the validity and integrity of social intervention research.
Required Readings:
Testa & Poertner, pp. 357-379.
D’Cruz, H., Gillingham, P. & Melendez, S. (2007). Reflexivity, its meanings and relevance for
social work: A critical review of the literature. British Journal of Social Work 37, 73–90.
Supplementary Readings:
Shadish, W.R. (1995). Philosophy of science and the quantitative-qualitative debates: Thirteen
common errors. Evaluation and Program Planning, 18 (1), 63-75.
19