Making Sense of
the Social World
th
4 Edition
Chapter 11, Evaluation Research
Evaluation Research
Evaluation Research is conducted for a distinctive
purpose: to investigate social programs.
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Evaluation Terms






Inputs: resources, raw materials, clients, and staff that
go into a program
Program process: the complete treatment or service
delivered by the program
Outputs: the services delivered or new products
produced by the program process
Outcomes: the impact of the program process on the
cases processed
Feedback: information about service delivery system
outputs, outcomes, or operations that is available to
any program inputs
Stakeholders: individuals and groups who have some
basis of concern with the program, often setting the
research agenda and controlling research findings
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
History of Evaluation Research





Began after expansion of federal government during
the Great Depression and WWII
Became more important with Great Society programs
of 1960s, because program evaluation became a
requirement
The New Jersey Income Maintenance Experiment was
the first large scale experiment to test social policy in
action
Decline of evaluation research firms in early 1980s as
Great Society programs also declines
Government Performance and Results Act of 1993
required some type of evaluation for all government
programs
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Design Alternatives
Black Box or Program Theory:


Is it important how the program gets results?


Black box: if program results are of primary importance,
how it works may be of secondary importance.
Program theory: a descriptive or prescriptive model of how
a program operates and produces effects.


descriptive program theory specifies impacts that are generated
and how this occurs (suggesting a causal mechanism, intervening
factors, and context), generally empirically based
prescriptive program theory specifies what ought to be done by
the program, but has not yet been empirically tested or observed

Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Design Alternatives, cont’d.
Researcher or Stakeholder Orientation: Is the primary
evaluator of research a set of social scientific peers or
a funding agency?




Stakeholder approaches encourage researchers to be
responsive to stakeholders (aka responsive evaluation)
Social science approaches: emphasize the importance of
researcher expertise and maintenance of some autonomy in
order to develop the most trustworthy, unbiased program
evaluation
Integrative approaches: attempt to cover issues of concern to
both stakeholders (including participants) and evaluators,
balancing stakeholder concern with scientific credibility
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Design Alternatives, cont’d.
Quantitative or Qualitative Methods



Evaluation research that attempts to identify the effects of a
social program typically use quantitative methods
Qualitative methods useful for investigating program
process, learning how individuals react to treatment,
understanding actual operation of programs, understanding
more complex social programs
Simple or Complex Outcomes




Most evaluation research attempts to measure multiple
outcomes
Unanticipated outcomes can be missed
Single outcomes may miss the process of how a program
works
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Foci of Evaluation Research
1.
2.
3.
Needs assessment: an attempt to
determine if a new program is needed or an
old one is still required
Evaluability assessment: a determination if
a program may be evaluated within
available time and resources
Process Evaluation (implementation
assessment): evaluation research that
investigates the process of service delivery
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Foci of Evaluation Research, cont’d.
4.
5.
Impact analysis: evaluation research that
compares what happened after a program
was implemented with what would have
happened had there been no program at all
Efficiency analysis: cost-benefit and costeffectiveness evaluation
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Ethical Issues

The direct impact on participants and their
families through social programs heightens the
attention to human subjects concerns



Needs assessments, evaluability assessments,
process analysis, and cost-benefit analysis have
few special ethical considerations
When program impact is focus, human subjects
problems multiply
Federally mandated IRBs must assess all
research for adherence to ethical practice
guidelines
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Solving Ethical Issues

To lessen any detrimental program impact:





Minimize number in control group
Use minimum sample size
Test only new parts of the program, not the
entire program
Compare treatments that vary in intensity rather
than presence and absence
Vary treatments between settings, rather than
among individuals in a single setting
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications
Obstacles to Evaluation Research





Evaluation research can miss important
outcomes or aspects of the program process
Researchers can be subjected to crosspressures by stakeholders
Answering to stakeholders can compromise
scientific design standards
Researchers may be pressured to avoid null
findings or find their research findings ignored
Evaluation reports might need to be overly
simplified for a lay audience, and thus subject to
some distortion
Chambliss/Schutt, Making Sense of the Social World 4th edition
© 2012 SAGE Publications