THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL
SCHOOL OF SOCIAL WORK
COURSE NUMBER:
COURSE TITLE:
SEMESTER & YEAR:
Room:
Time:
SOWO 910
Research Methods for Social Intervention
Fall, 2013
TTK 226
9:00AM – 11:50 AM
INSTRUCTOR:
Mark Testa, PhD
Spears-Turner Distinguished Professor
School of Social Work
University of North Carolina at Chapel Hill
Tate-Turner-Kuralt Building
325 Pittsboro St., Campus Box 3550
Chapel Hill, NC 25599-3550
Tel: (919) 962-6496 Fax: (919) 962-1486
mtesta@unc.edu
OFFICE HOURS:
Room 548C
Fridays, 12:00 – 1:30 PM or by appointment
COURSE DESCRIPTION:
This course provides an introduction to basic research processes and methods for use in
planning, implementing, evaluating, and improving social interventions at the explorative,
formative, summative, translative, and confirmative stages of program implementation and
evaluation. Topics include outcomes monitoring, problem formulation, needs assessment,
construct measurement, theory of change, research review, human subjects’ protection,
evaluation design, implementation science, data analysis, and the application of findings to
practice improvement and theory refinement.
COURSE OBJECTIVES:
This course affords students an opportunity to gain knowledge about the following issues in
social intervention research:


The need for broadly inclusive processes to plan, implement, and evaluate social
interventions at the explorative, formative, summative, translative, and confirmative
stages of program implementation and evaluation, and how researchers' approaches to
these processes can facilitate or impede research;
The quantitative-comparative experimental (potential outcomes) paradigm that currently
prevails in social intervention research;
1



How various policy and implementation constraints sometimes necessitate the use of
designs other than fully randomized experiments;
Special legal and ethical issues pertaining to the protection of human subjects; and
The need for culturally aware social intervention research that is responsive to the
diversity of community values and preferences.
Students taking the course will be able to:






develop “well-built” research questions for estimating the causal impact of social
interventions on desired outcomes for target populations;
develop logic and other conceptual models to support proposed social interventions and
explicate underlying theories of change;
assess the validity and reliability of alternative qualitative and quantitative measures of
constructs in conceptual models that guide social intervention research;
understand basic aspects of data analysis, sample design and statistical power analysis;
critically evaluate various experimental, quasi-experimental, and non-experimental
research designs by identifying various threats to implementation integrity and
intervention validity; and
prepare an application for IRB approval of human subjects research.
Recommended Prerequisites
SoWo 102 or equivalent
SoWo 292 or equivalent
Relationship to Other Courses in the Curriculum
This course introduces basic concepts and skills that are reinforced and further developed
in Development of Social Intervention Models (SOWO 940) and Advanced Research Methods in
Social Intervention (SOWO 913) during the 2nd and 3rd years of the program. It extends students’
knowledge about basic research processes and orients them to the variety of professional issues,
theoretical perspectives, statistical methods, and advanced methods of intervention research that
they will examine in related doctoral-level courses (SOWO 900, 911, & 914) and can pursue in
courses outside of the School.
REQUIRED TEXTS:
Ayres, I. (2007). Super crunchers: Why thinking-by-numbers is the new way to be smart.
New York: Bantam Books
Shadish, W.R, Cook, T.D., & Campbell, D.T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin
Company.
Testa, M. F. & Poertner, J. (2010). Fostering accountability: Using evidence to guide and
improve child welfare policy. Oxford: Oxford University Press.
2
REQUIRED AND SUPPLEMENTARY READINGS:
Required readings outside of the textbook are listed in the course outline. Supplementary
readings are assigned at the instructor’s discretion.
TEACHING METHODS
The development of a supportive learning environment, reflecting the values of the social
work profession, is essential for the success of this class. A supportive learning environment is
fostered by listening to the ideas and views of others, being able to understand and appreciate a
point of view which is different from your own, articulating clearly your point of view, and
linking experience to readings and assignments. Everyone will appreciate your contributions to
making this a safe and respectful class for learning and growth.
CLASS ASSIGNMENTS
SEMINAR PARTICIPATION
1. Register with JSSR: Sign up as a subscriber to the Journal of the Society for Social
Work and Research (edited by Mark Fraser) by registering online. August 30.
2. Obtain Certification in Protection of Human Participants: Students must complete
the on-line course developed by the Collaborative IRB Training Initiative (CITI) to
certify that they are familiar with the ethical principles and guidelines governing research
involving human participants; completing the course requires several hours; each student
is required to submit electronically on Sakai or email a completion certificate before
September 6 (this assignment is required, but is not graded).
3. Identify potential resources and collaborators. Become familiar with potential sources
of information about obtaining financial support for your research. Also you may wish to
make yourself known to potential research collaborators by registering as a member of
the Community of Science. September 13.
4. Participate in seminar: Students are expected to read assignments and come to class
prepared to share concepts from the readings, ask questions, and respond to questions
about the material. Each student should prepare a 1-page self-evaluation of his or her
efforts to make this a productive learning experience. Award yourself 1-5 points for those
efforts (see description of grading). This is due the last day of class (November 22) and
should be submitted to the instructor electronically on Sakai or by email.
WRITING ASSIGNMENTS AND IN-CLASS EXAM
During the semester, you will write five papers (varying in length from 1 to 7 pages) on
different components of the research process. The final assignment will require revision and
expansion of each of these components into a full IRB application. There will also be a one-hour,
in-class exam. The assignments are as follows:
3
1. Research Question: After meeting with the instructor to discuss preliminary ideas, each
student will formulate a “well-built” PICO question that consists of the following
components: 1) the target population about which you wish to draw inferences; 2) the
outcome you intend to achieve or problem you hope to address; 3) the intervention you
are interested in evaluating; 4) and the alternative course of action with which you will
draw a comparison (e.g., no intervention, regular services, or different interventions).
This one (1) - pager, including a brief statement about the significance of the problem or
outcome you are studying, will be due September 13.
2. Research Review: Conduct a computerized search of electronic databases using
keywords from your PICO question. Select the strongest four to six (4-6) studies that bear
on your topic. Write up a research review using narrative descriptions that also assess the
strength of the evidence for supporting the use of your intervention and identify the
limitations of these studies and their applicability to your population. This paper (5-7
pages) will be due October 4.
3. Logic Model: Expand your PICO question into a logic model that lays out the expected
causal mechanisms and mediating pathways from the intervention to the desired outcome.
Your description of the target population should also list any population conditions that
you believe may moderate the intervention’s impact on the outcome. The model should
also enumerate any background factors and external conditions that contribute to the
significance of problem you are addressing. It should list the key assumptions of the
theory of action you are positing will effectuate the desired change. Finally the model
should identify general end-values for reconciling diverse outcomes and evaluating the
ultimate worth of the resulting change. A modified Logic Model Template for filling in
this one (1) page figure is available under Resources on Sakai. It will be due October 11.
4. Measurement Review: Identify and assess the relative strengths and weaknesses
of alternative measures or approaches to measuring a construct (population, intervention,
or outcome) referenced in your research question. The purpose of the exercise is to select
the measure (or set of measures) that will yield the most valid and reliable data
concerning this construct. This paper (5-7 pages) will be due October 25.
5. Evaluation Design: Based on your research question, you will outline the basic features
of an experimental or quasi-experimental design for evaluating the impact of the
identified social intervention on your outcome construct. The description should identify
the unit(s) of analysis, comparison group(s), and how threats to the validity of your
research will be addressed. The discussion should provide a rationale for the method you
are proposing and how concessions to design constraints may make the research
vulnerable to criticism. This 5-7 page paper will be due November 8.
6. In-Class Exam: There will be a one-hour, in-class, multiple-choice exam based on
questions that you and your fellow students bring to class each week about Shorthand
Abstractions (SHAs) that you found improved or challenged your thinking and which you
think could also improve the cognitive toolkit of your fellow classmates and instructor.
Professor James Flynn uses the term to refer to concepts drawn from science which make
people smarter by providing widely applicable templates. Concepts such as control
group, random sample, regression coefficient, the 2SD rule, placebo, and falsifiability are
examples of SHAs that make available complex ideas in unified cognitive chunks that
4
can be used as elements in critical thinking and debate. As you read the assigned readings
for each class, take note of a concept or two that you think could qualify as a SHA. The
concept can be understood in a broad sense as a valid and reliable way of gaining
knowledge as long as it is a rigorous conceptual tool that may be summed up succinctly
in a word or a phrase and has broad application to interpreting the world. Come prepared
to each class with a question each week about a SHA. These questions will be sampled
for the in-class exam that will be administered on November 15.
7. IRB Application: This final assignment will follow the instructions issued by the UNC
Office of Human Research Ethics for application for IRB approval. We will be reviewing
various resources throughout the course that will be helpful in preparing it. It will
incorporate material from each of the other written assignments as well as a description
of risks to human subjects and measures to minimize those risks and a discussion of the
benefits to subjects and/or society. The online application form can be accessed from
http://irbis.unc.edu after you enter your Onyen and password. Instructions and training
videos can be accessed at http://research.unc.edu/offices/human-researchethics/index.htm by clicking on online submission on the left. This application will be
due on November 25.
WRITING ASSIGNMENT GUIDELINES:
All written assignments must be typed and follow APA format. Several writing resources
are posted on the website. Students should also refer to the following:
 American Psychological Association. (2009). Publication manual of the American
Psychological Association (6th ed.). Washington, DC.
 Note: You can find a self-paced tutorial for APA style at
http://www.lib.unc.edu/instruct/citations/index.html?page=apa_sample
ONLINE RESOURCES:
Course materials for SOWO 910 will be accessible to you on https://www.unc.edu/sakai/.
In addition, Angela Bardeen, Behavioral and Social Sciences Librarian at UNC, has created a
SOWO 910 course library website for finding articles, evidence-based practice resources,
measurement tools, and human subjects protection. It can be accessed at
http://guides.lib.unc.edu/F12sowo910.
COURSE PERFORMANCE ASSESSMENT:
Final grades will be determined on the basis of points earned on each assignment and on
participation in seminars. Letter grades will correspond to the following point totals:
94 - 100:
80 - 93:
70 - 79:
< 70:
High Pass
Pass
Low Pass
Fail
5
Letter grades in this course generally follow a distribution with a mean of 87 and a standard
deviation of 6 points.
Points are apportioned to individual assignments as follows:
1.
2.
3.
4.
5.
6.
7.
8.
Seminar participation: 10 points (5 points by instructor and 5 points by self-evaluation)
Research Question: 10 points
Research Review: 10 points
Logic Model: 10 points
Outcome Measurements: 10 points
Evaluation Design: 15 points
In-Class Exam: 10 points
IRB Application: 25 points
POLICY ON INCOMPLETES AND LATE ASSIGNMENTS
The instructor will only entertain requests to hand in assignments late in circumstances of
special hardships or emergencies. The potential for handing in papers late needs to be discussed
with the instructor in person at least three days before the assignment is due. If not approved
beforehand, late assignments will be graded five points off for each day the assignment is late,
including weekends. A grade of Incomplete is given on rare occasions when there is sufficient
reason to warrant it. It is the student’s responsibility to initiate a conversation with the instructor
to request an Incomplete—instructors have no responsibility to give an Incomplete without such
a request.
POLICY ON ACADEMIC DISHONESTY
Please refer to SSW Writing Resources and References website for information on
attribution of quotes, plagiarism and appropriate use of assistance in preparing assignments. All
written assignments should contain a signed pledge from you stating that, "I have not given or
received unauthorized aid in preparing this written work".
In keeping with the UNC Honor Code, if reason exists to believe that academic
dishonesty has occurred, a referral will be made to the Office of the Student Attorney General for
investigation and further action as required.
POLICY ON ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES
Students with disabilities that affect their participation in the course and who wish to
have special accommodations should contact the University’s Disabilities Services and provide
documentation of their disability. Disabilities Services will notify the instructor that the student
has a documented disability and may require accommodations. Students should discuss the
specific accommodations they require (e.g. changes in instructional format, examination format)
directly with the instructor.
6
OUTLINE OF CLASS TOPICS
Date
Topics for Discussion
Reading Assignments To Be
Completed before Class
UNIT I: INTRODUCTION TO SOCIAL INTERVENTION RESEARCH
Week 1
PICO Questions & Logic Models
Start readings and complete PICO &
(August 23)
Logic Model exercise for next week.
Week 2
Overview of Social Intervention
Testa & Poertner, pp. 75-100; Kenny et
(August 30)
Research
al., pp. 294-324; Testa & White, pp.135.
Week 3
Scientific Validity & Agency
Testa & Poertner, pp. 3-13; Shadish,
(September 6) Integrity
Cook & Campbell, pp. 34-42; Dane &
Schneider, pp. 23-45.
UNIT II: EVALUATIVE, FORMATIVE, SUMMATIVE, TRANSLATIVE &
CONFIRMATIVE RESEARCH
Week 4
Outcomes Monitoring
Testa & Poertner, pp. 114-135;
(September 13)
Goodwin, pp.100-106.
Week 5
Data Analysis
Ayres, pp. 1-45; Testa & Poertner, pp.
(September 20)
136-147, 153-165; Freedman, et al., pp.
202-217.
Week 6
Research Reviews
Ayres, pp. 81-102; Testa & Poertner, pp.
(September 27)
166-194; Shadish, Cook & Campbell,
pp. 417-455.
Week 7
(October 4)
Week 8
(October 11)
Week 9
(October 18)
Week 10
(October 25)
Week 11
(November 1)
Week 12
(November 8)
UNIT III: INTERVENTION EVALUATION DESIGNS
Evaluation Designs
Ayres, pp. 46-80; Freedman, et al., pp.
3-28; Akin et al., pp. 19-30.
Experimental Designs
Testa & Poertner, pp. 195-205;
Boruch, et al. pp. 330-353.
Fall Break
A Brief Tutorial on DAGs
Testa & Poertner, pp. 205-230; West et
al., pp. 1359-1366; Fleischer & Diez
Roux, pp. 842-846.
UNIT IV: RESULTS-ORIENTED ACCOUNTABILITY
Statistical Precision & Power
Ayres, pp. 192-220; Testa & Poertner,
Analysis
pp. 147-152; Orme et al., pp. 3-10.
Human Subjects Protections
The Belmont Report; Homer, 200-207.
7
Week 13
Confirmative Implementation &
(November 15) Quality Improvement &
In-Class Exam
Week 14
Confirmative Research and Reflexive
(November 22) Practice
Ayres, pp.103-128; Testa & Poertner,
pp. 231-268, 291-327; Moseley &
Solomon, pp. 12-17.
Ayres, pp. 156-191; Testa & Poertner,
pp. 357-379; Nijnatten, pp. 131-136.
UNIT I. INTRODUCTION TO SOCIAL INTERVENTION RESEARCH
Week 1 (August 23)— PICO QUESTIONS AND LOGIC MODELS
The research process begins with the formulation of a well-built research question that can be
parsed into the four components of population, intervention, comparison, and outcome. Grouped
together under the acronym PICO, the formulation of a well-built question guides the process of
data analysis, computerized research reviews, logic model construction and evaluation design.
There are a variety of designs for constructing logic models. The design used in this course
elaborates on the PICO framework and visually depicts the mediating activities that link
interventions and population conditions to the short-term outputs and proximal outcomes
produced by the activities and the longer-term distal outcomes these intermediate actions are
expected to produce. A full logic model also identifies the external conditions that prompted
concern over the problem, the underlying theory of change, end-values for evaluating the
ultimate worth of the resulting change, and any moderating conditions that may affect the
intervention’s impact.
You will spend this first week working on the PICO question (Testa and Poertner, 2010: 81-82)
that you think Kenny et al. (2004) addresses in next week’s reading assignment. Also construct a
logic model of the intervention they studied using the modified template posted on Sakai under
Resources and described in Testa and Poertner (2010: 85-98). Submit both the PICO question
and logic model electronically on Sakai before 12:00 pm on Thursday, August 29 and bring hard
copies to class the next day.
Week 2 (August 30)— OVERVIEW OF SOCIAL INTERVENTION RESEARCH
Social intervention research serves five distinct but interrelated purposes: 1) developing and
testing a social intervention to improve practice and policy (formative research); 2) comparing a
promising intervention to its counterfactual and learning about its average causal effect
(summative research); 3) replicating and adapting evidence-supported interventions to different
local contexts and sub-populations (translative research); 4) applying the intervention broadly
and improving its integrity and validity to decide whether it should be maintained as is, changed
in some way, or discarded completely, with or without replacement (confirmative research) and
5) describing and explaining the effects of a social intervention as a contribution to scientific
knowledge and theory (explorative research).
8
Figure 1 Cycle of Social Intervention Research
Figure 1 depicts these five interrelated purposes as a series of multi-level transactions that move
interventions from the micro level through formative implementation and evaluation of practiceinformed innovations to the macro level through summative implementation and evaluation of
their average causal effects (ACEs). Further, if the success of the interventions is empirically
supported, the process continues and makes linkages back down to the micro level through
translative implementation and evaluation that adapt evidence-supported interventions to
different populations and local contexts. After an evidence-based practice has been implemented
for a period of time, its validity is periodically reexamined through confirmative implementation
and evaluation that helps to decide whether the practice or program should be maintained as is,
changed in some way, or discarded completely, with or without replacement. The decision to
change or replace initiates another cycle of formative implementation and evaluation.
9
Surrounding the diagram is a set of explorative research, problem definition, scientific
knowledge and theory-building, and research review activities that precede intervention
development and implementation and are informed by the results of the process. These activities
typically proceeds through a series of steps and can include 1) outcomes monitoring-- defining
the desired outcomes; 2) data analysis – identifying the potentiating and compensatory factors
for attaining the outcomes and determining the extent of need and potential target populations for
intervention; 3) research review – converting the information into an answerable research
question, summarizing the best available evidence for answering the question, and selecting an
intervention for implementation; and 4) evaluation design– developing a plan for insuring the
integrity of implementation and evaluating the efficacy and effectiveness of the intervention.
Your engagement with the process of social intervention research begins by identifying yourself
to potential collaborators by registering as a member of the Community of Science.
Required Readings:
Testa & Poertner, pp. 75-100.
Kenny, D.A. et al. (2004). Evaluation of treatment programs for persons with severe mental
illness: Moderator and mediator effects. Evaluation Review, 28, 294-324.
Testa, M. F. & White, K. R. (In press). Insuring the integrity and validity of social work
interventions: The case of the subsidized guardianship waiver experiments. Journal of
Evidence-Based Social Work, pp. 1-35.
Supplementary Reading:
The Evaluation Exchange, 11(2): pp. 2-15.
Week 3 (September 6)— SCIENTIFIC VALIDITY AND AGENCY INTEGRITY
Accountability for social welfare outcomes involves holding practitioners and other change
agents answerable both for the integrity of the actions taken on behalf of their clients and other
principals and for the validity of those actions in achieving the outcomes valued by their
principals and the public (Testa & Poertner, 2010). The success of a principal-agent (clientpractitioner) relationship is a product of the two. Failure of an agency relationship to achieve its
intended purposes thus may reflect either a problem with the integrity of the implementation or a
problem with the validity of the intervention (Klein & Sorra, 1996).
Social intervention research is an agency relationship that involves both people who design and
implement interventions (agents) and the people that provide funding for testing them and the
participants who are affected by the interventions (principals). As such, agents bear
responsibility for abiding by a set of scientific standards and ethical principles in fulfilling the
10
interests of their principals. In this respect, research accountability goes beyond accumulating
valid evidence of the efficacy and effectiveness of child welfare interventions to ensuring that the
implementation process reliably and responsibly serves the purposes valued by clients, research
sponsors, and the public at large. To be accountable, social intervention researchers must
appreciate the context in which they work and develop the interpersonal skills as well as the
technical knowledge and skills that can make them both effective and responsible in their roles.
Your socialization into the role of a responsible agent begins with completing the on-line course
related to the protection of human participants in research. Each student will submit a CITI
certificate indicating their completion of the course. This certification and the
automatic registration with UNC's Office of Human Research Ethics satisfy the requirements of
many of the research assistantships held by students in the doctoral program.
Scientific validity refers to the best available approximation to the truth or falsity of a given
hypothesis, inference, or conclusion and provides a set of standards by which the quality of the
research can be judged. There are four generally recognized types of scientific validity: 1)
statistical conclusion validity that is concerned with whether there is a statistically significant
association between an intervention and the desired outcome; 2) internal validity that focuses on
whether the statistical association results from a causal relationship between the intervention and
the outcome or is a spurious association; 3) external validity that addresses how generalizable the
particular causal relationships are over variations in PICO; and 4) construct validity that assesses
the degree of correspondence between the observational particulars of a population, intervention,
comparison, and outcome (PICO) and their higher-order constructs. The demonstration of
scientific validity usually proceeds cumulatively and the order of demonstration varies with the
purpose or purposes served by the social intervention research.
Required Readings:
Testa & Poertner, pp. 3-13.
Shadish, Cook & Campbell, pp. 34-42.
Dane, A.V. & Schneider, B. H. (1998). Program integrity in primary and early secondary
prevention: Are implementation effects out of control? Clinical Psychology Review,
18(1), 23-45,
Supplementary Readings:
Webb, S.A. (2001). Some considerations on the validity of evidence-based practice in social
work. British Journal of Social Work, 31: 57-79.
Gibbs, L. & Gambrill, E. (2002). Evidence-based practice: Counterarguments to
objections. Research on Social Work Practice, 12: 452-476.
11
UNIT II: EXPLORATIVE, FORMATIVE, SUMMATIVE, TRANSLATIVE &
CONFIRMATIVE RESEARCH
Week 4 (September 13)—OUTCOMES MONITORING
The construction of a well-built research question begins with the identification and
measurement of an outcome or set of outcomes that principals and their agents want to monitor
and, if desired, change. Outcome is synonymous with the effect, result or dependent variable of a
cause, intervention or independent variable. In mathematical terms, it is y or a function of x: y =
f(x). In this course we will also substitute O for y in keeping with the PICO framework. In order
to estimate the effect of an intervention on an outcome, it is necessary to translate the higherorder, theoretical construct of the outcome into a lower-order, operational measurement of the
variable. Construct validity refers then to the degree of correspondence between the higher order,
theoretical constructs and the lower-order, observational particulars. To demonstrate construct
validity, you need to show evidence that the observational particulars (data) support the
theoretical structure of the construct. How researchers conceptualize this task is evolving, and we
will review some of the latest thinking on these endeavors.
Some researchers oppose the routine use of outcome indicators to improve public management.
They argue that monitoring and evaluating agency units, e.g., schools, departments, and courts,
invariably lead to the corruption of the indicators used to monitor results and to the degradation
of the agency relationships that program evaluation is supposed to improve. It is important to
acknowledge these agency risks and take cognizance of such threats to agency integrity. We will
consider the precautions that can be taken which can help decrease these agency risks and
increase the opportunities for responsible public policy and management.
Required Readings:
Testa & Poertner, pp. 114-135.
Goodwin, L. (2002). Changing conceptions of measurement validity: An update on the new
Standards. Journal of Nursing Education, 41, 100-106.
Supplementary Readings:
Pelton, L. (2008). A note contesting Mark Testa's version of national foster care population
trends, Children and Youth Services Review, doi:10.1016/ j.childyouth.2008.09.005.
Testa, M. (2008). How the bear evolved into a whale: A rejoinder to Leroy Pelton's note
contesting Mark Testa's version of national foster care population trends, Children and
Youth Services Review, doi:10.1016/ j.childyouth.2008.10.009.
12
Week 5 (September 20)—DATA ANALYSIS
Observation of a deterioration in outcomes, which is of both practical importance and statistical
significance, does not necessarily indicate that agency performance is deficient or in need of
correction. There could be some other antecedent conditions beyond the practitioners’ and
agents’ immediate control that could be causing the difference. For example, a practically
important and statistically significant difference in post-operative mortality rates between two
hospitals may not necessarily mean that the hospital with the lower rate is the superior
performing hospital. It may instead mean that the hospital with the higher rate admits a sicker
group of patients, on average, than the other. Therefore before a summary judgment can be
rendered about agency performance or the type of improvement needed, it is first important to
identify those factors that may be clouding the comparison with data analysis methods for
purging or adjusting for external confounding influences.
In this week, we will introduce the regression method of adjusting for confounding variables that
can exaggerate or obscure the true causal effect of an intervention on an outcome. Freedman et
al. offer a simple description of the correlation coefficient and how it is easily transformed into a
regression coefficient. We will then replicate the results obtained by one of the first applications
of the regression method to a social policy question: G. Udny Yule’s investigation into the
causes of changes in pauperism in England published in 1899.
Required Readings:
Ayres, pp. 1-45.
Testa & Poertner, pp. 136-147, 153-165.
Freedman, D., Pisani, R. & Purves, R. (2007). The Regression Line. Statistics- 4th edition, (pp.
202-217). New York: W.W. Norton & Company.
Supplementary Reading:
Yule, G. U. (1899). An investigation into the causes of changes in pauperism in England, chiefly
during the last two intercensal decades (part I.). Journal of the Royal Statistical Society,
62, (2), 249-295.
Hayes, A. F. (2013). Simple linear regression. Introduction to mediation, moderation, and
conditional process analysis: A regression-based approach (pp. 23-43).New York: The
Guildord Press.
13
Week 6 (September 27)—RESEARCH REVIEWS
(Guest Instructor, Qi Wu, Doctoral Student, UNC)
Research reviews involve the explicit search and selection of relevant studies, the assessment of
their scientific validity, and a critical synthesis of their findings to reach a tentative conclusion
about the efficacy and effectiveness of social interventions. This stage of research-oriented
accountability is what the field most commonly understands as evidence-based practice (EBP). A
commonly cited definition by Sackett et al. (1997) is that EBP is “the conscientious, explicit and
judicious use of current best evidence in making decisions about the care of individual [clients].”
EBP is emerging also as the guiding paradigm within which social intervention research is
pursued by university-based researchers and applied by various actors in the areas of medicine,
psychology, public health, criminology, social work and public policy. While paradigms provide
a framework within which knowledge can be developed, they can also intellectually straitjacket
their adherents. Given this potential agency risk, there is a need to subject EBP to the same
methods of critical review and reflective assessment that its adherents espouse for validating
clinical practices and public policies.
Required Readings:
Ayres, pp. 81-102.
Testa & Poertner, pp. 166-194.
Shadish, Cook & Campbell, pp. 417-455.
Supplementary Readings:
Chalmers, I. (2003). Trying to do more good than harm in policy and practice: The role of
rigorous, transparent, up-to-date evaluations. Annals of the American Academy of
Political and Social Science, 589, 22-40.
Littell, J. (2008). Ch. 4: How do we know what works? The quality of published reviews of
evidence-based practices. In Lindsey, D. & Shlonsky, A. (Eds.) Child welfare research:
Advances for practice and policy (pp. 66-93). Oxford: Oxford University Press.
Winokur, M., Holtan, A., & Valentine, D. (2009) Kinship care for the safety, permanency, and
well-being of children removed from the home for maltreatment. Cochrane Database of
Systematic Reviews, 1, Art.No.: CD006546. DOI:10.1002/14651858.CD006546.pub2.
14
UNIT III.
INTERVENTION EVALUATION DESIGNS
Week 7 (October 4)—EVALUATION DESIGNS
The question that clients, practitioners and policymakers really want to answer from social
intervention research is what would happen if people who are given a certain treatment or
intervention option were instead denied this possibility. Of course such “potential outcomes” can
never be compared at the individual level because it is impossible simultaneously to deny and to
offer a treatment option to an individual person. Instead researchers have to fall back on high
quality approximations to this impossible “what if” scenario (what statisticians call the
“counterfactual”) by conducting rigorous studies that allow them to draw causal inferences at the
macro level.
A compelling case can be made for more routine use of randomized controlled experiments in
social work than is currently the practice. But there are situations in which controlled
experimentation is inadvisable, unethical, or just plain impossible. Over the past several
decades, researchers have made tremendous strides in conceptualizing the assumptions that need
to be satisfied in order to draw valid causal inferences from non-experimental research and how
best to approximate the necessary conditions using statistical methods. The material in this
section will help students acquire technical knowledge related to the development and
application of different research designs—experimental, quasi-experimental, and nonexperimental.
Required Readings:
Ayres, pp. 46-80.
Freedman, D., Pisani, R. & Purves, R. (2007). Controlled experiments & Observational Studies.
Statistics- 4th edition, (pp. 3-28). New York: W.W. Norton & Company.
Akin, B., Bryson, S., Testa, M., Blase, K., McDonald, T. & Melz, H. (2013). Usability testing in
the initial implementation and formative evaluation of an evidence based intervention to
reduce long-term foster care. Evaluation and Program Planning, 41,(19-30).
Supplementary Reading:
Testa & Poertner, pp. 269-290
Rubin, D.B. (2004). Teaching statistical inference for causal effects in experiments and
observational studies. Journal of Educational and Behavioral Statistics, 29 (3), 343-367.
15
Week 8 (October 11)—EXPERIMENTAL DESIGNS
The most rigorous method for drawing causal inferences about the potential effects of a
treatment is the randomized controlled experiment. By employing a process such as a lottery,
coin flips or a table of random numbers to treat a chance selection of persons or group, we can
feel confident that the characteristics of the intervention group (who are offered the option) and
the comparison group (who are denied the option) are statistically equivalent on average within
the bounds of statistical error. The thinking is that if the two groups start out looking statistically
similar at the initiation of the experiment and then if any significant differences later emerge
after implementation, it is reasonable to conclude that the cause of the differences is the
intervention itself rather than any pre-existing group dissimilarities or concurrent policy changes
(which affect both groups).
Despite these advantages, some social work practitioners find randomized controlled
experiments to be ethically suspect because of its denial of services to the comparison group and
to be of limited use because of the lengthy observational period before a summative judgment
can be confidently rendered. This section will consider the appropriateness of constraining
practitioner discretion by experimental protocols and the conditions under which randomization
constitutes a justifiable interference with agent discretion when the empirical evidence for the
efficacy or effectiveness of a promising intervention is weak.
Required Readings:
Testa & Poertner, pp. 195-205.
Shadish, Cook & Campbell, pp. 279-291.
Boruch, R. F., Victor, T. & Cecil, J.S. (2000). Resolving Ethical and Legal Problems in
Randomized Experiments. Crime Delinquency, 46, 330-353.
Supplementary Readings:
Testa, M. (2002). Subsidized guardianship: Testing an idea whose time has finally come. Social
Work Research, 26 (3), 145-158.
Ludwig, J., Liebman, J.B., Kling, J.R., Duncan, G.J., Katz, L.F., Kessler, R.C. & Sanbonmatsu,
L. (2008). What can we learn about neighborhood effects from the moving to opportunity
experiment? American Journal of Sociology, 114(1), 144-188.
Prinz, R.J., Sanders, M.R., Shapiro, C.J., Whitaker, D.J. & Lutzker, J.R. (2009). Populationbased prevention of child maltreatment: The U.S. Triple P system population trial.
Prevention Science, DOI 10.1007/s11121-009-0123-3
16
October 18: Fall Break – No Class
Week 10 (October 25)—A BRIEF TUTORIAL ON DIRECTED ACRYLIC GRAPHS
(DAGs)
(Guest Instructor, Kevin White, Doctoral Student, UNC
Although experiential research is the “gold standard” in social work and many related fields such
as medicine, education, mental health, and criminology, oftentimes it is difficult or impossible to
randomize clients or other units to intervention and comparison groups. Evaluation methods that
approximate the statistical equivalence that is best obtained through random assignment are
called quasi-experiments or more broadly observational studies.. Econometricians and
statisticians have developed a variety of statistical methods for handling selection biases in order
to uncover the genuine causal effect of an intervention on an outcome. The most commonly used
approaches for adjusting for selection biases in the absence of randomization include, propensity
score matching, instrumental variable analysis, and regression discontinuity designs.
Directed Acyclic Graphs (DAGs) are a rigorous, largely graphic approach to causal inference
from quasi-experimental and observational data. This approach helps to separate causation from
association by identifying what control variables must be included and which variables mustn’t
be included in order to reduce the observed (conditional) association between intervention and
outcome to its purely causal (i.e., non-spurious) component.
Required Readings:
Testa & Poertner, pp. 205-230.
West, S. G., Duan, N., Pequegnat, W., Gaist, P.,Des Jarlais, D.C., Holtgrave, D., Szapocznik, J.,
Fishbein, M., Rapkin, B., Clatts, M., & Mullen, P.D. (2008). Alternatives to the
randomized controlled trial. American Journal of Public Health, 98(8), 1359-1366.
Fleischer, N.L. & Diez Roux, A.V. (2008). Using directed acyclic graphs to guide analysis of
neighborhood health effects: An introduction. Journal of Epidemiology and Community
Health, 62(9), 842-846.
Supplementary Reading:
Elwert, F. (2013). Graphical causal models. In S. Morgan (Ed.) Handbook of causal analysis for
social research (pp. 245-273). Dodrecht: Springer.
17
UNIT IV: RESULTS-ORIENTED ACCOUNTABILITY
Week 11 (November 1)— STATISTICAL PRECISION AND POWER ANALYSIS
When steady progress is being made toward remedying a problem or attaining a desired
outcome, the monitoring process continues another round of assessment and review. Otherwise,
the detection of a worrisome gap between a valued result and an observed outcome may signal
the need for some corrective action or at least a reasonable accounting for the shortfall. When the
gap is of both practical importance and statistical significance, data analysis can be initiated to
identify the areas for improvement and the risk factors that may be contributing to the result. The
first step in assessing whether a gap is worrisome enough is to decide on the smallest difference
or distance that is deemed important enough to matter. This is best done in conjunction with
practitioners and administrators who are able to draw comparisons from experience, history, an
established standard, or some other reference point. With the availability of large administrative
databases, an assessment of practical importance can proceed without much concern for the
statistical significance of the conclusion. However when information is based on a sample from
a larger population, there is always a chance that the conclusion will be wrong due to the lack of
complete information about the population. Sampling theory provides a basis for evaluating the
chances of error and for taking those risks into account when designing a study.
Statistical power is a concept that has been commanding greater attention in social intervention
research. It refers to the probability that a real difference, effect size, or pattern of association of
a certain magnitude in a population will be detectable in a particular study. Power analysis
proceeds from assumptions about available sample size and the amount of error one is willing to
tolerate. It can also be used to determine sample size. To do this, we set an effect size that we
want to be able to detect and select a high probability (typically, 80%) that a statistical test will
reject the null hypothesis when the hypothesis of no effect is false. In this section, we will look at
some statistical power software that is typically used to determine the sample size necessary to
demonstrate an intervention’s effect.
Required Reading:
Ayres, pp. 192-220.
Testa & Poertner, pp. 147-152.
Orme, J. G. & Combs-Orme, T. D. (1986). Statistical power and type II errors in social work
research. Social Work Research & Abstracts, 22, 3-10.
Supplementary Readings:
Boruch, R. (2007). The null hypothesis is not called that for nothing: Statistical tests in
randomized trials. Journal of Experimental Criminology, 3, 1–20.
18
Week 12 (November 8)— HUMAN SUBJECTS PROTECTION
This section will provide the “nuts-and-bolts” on how to prepare a UNC application for IRB
approval of research proposals. You will be given an overview of the IRB application process at
the School. After this, you should have the tools and knowledge needed to complete the final
assignment.
Required Readings:
The National Commission for the Protection of Human Subjects of Biomedical and Behavioral
Research (1979). The Belmont Report: Ethical Principles and Guidelines for the
Protection of Human Subjects of Research.
Homer, C.S.E. (2002). Using the Zelen design in randomized controlled trials: debates and
controversies. Journal of Advanced Nursing, 38(2), 200–207
Week 13 (November 15)— CONFIRMATIVE IMPLEMENTATION AND QUALITY
IMPROVEMENT
Confirmative implementation and quality improvement requires that researchers move beyond
causal description and acquire a behavioral and deeper interpretative understanding of the
intervening processes by which program resources are disbursed within a social system and
transformed into program outputs and client outcomes. Initially quality improvement takes the
form of repeating a ”single-loop” learning cycle several times as needed to ensure that the
intervention is supplied in sufficient dosage and with adequate fidelity to the original program
design. If the results of the formative or translative evaluations fall short of the desired targets,
show no change, or else run contrary to other end-values, quality improvement takes the form of
a ”double-loop” leaning cycle in which the existing theory of action and its assumptions and
values are questioned and reflexively changed.
Required Readings:
Ayres, pp.103-128.
Testa & Poertner, pp. 231-268, 291-327.
Moseley, J.L. & Solomon, D.L. (1997). Confirmative evaluation: A new paradigm for
continuous improvement. Performance Improvement, 35(5), 12-16.
19
Supplementary Readings:
Moynihan, D. P. (2005). Goal-based learning and the future of performance management. Public
Administration Review, 65(2), 203—216.
Week 14 (November 22)— CONFIRMATIVE RESEARCH AND REFLEXIVE
PRACTICE
Some of the misunderstanding and distrust that occurs during the conduct of research arises from
the failures of researchers to acquire an interpretative understanding of client values and
perspectives and to take account of those preferences. To establish the mutual trust upon which a
research enterprise must rest, it is important to create recurring opportunities for the impulses,
desires, and values of the intended beneficiaries to enter into the design of a social intervention
and continuous revision of existing routines in light of new knowledge about the impact of those
practices. The coordination of the quantitative results with the qualitative feedback from clients,
practitioners, and researchers through peer review and “learning forums” constitutes an essential
ingredient in the integration of ethical, evidentiary, and practical concerns that is necessary for
the validity and integrity of social intervention research.
Required Readings:
Ayres, pp. 156-191.
Testa & Poertner, pp. 357-379.
van Nijnatten, C. (2010). Children’s agency, children’s welfare: A dialogical approachto child
development, policy and practice. Bristol: The Policy Press, pp. 131- 136.
Supplementary Readings:
Shadish, W.R. (1995). Philosophy of science and the quantitative-qualitative debates: Thirteen
common errors. Evaluation and Program Planning, 18 (1), 63-75.
D’Cruz, H., Gillingham, P. & Melendez, S. (2007). Reflexivity, its meanings and relevance for
social work: A critical review of the literature. British Journal of Social Work 37, 73–90.
November 29 University Holiday – No Class
20