Evaluation for the People
Research Methods for Community
Change
Reassessing the Pyramidal
Structure of Evaluation Research


The typical structure of evaluation is quite pyramidal, with
interlocking directorates of funding organizations, government
institutions, and academic evaluators imposed on lower levels
of service agencies, bureaucrats, and community members
The top imposes programs on the bottom, which implements
projects to fulfill the program mandates
Funders and
Interlocking Elites
Programs
Projects
Service
Agencies
Bureaucrats
Community Members
Adapted from Stoecker p. 65
Programs and Projects

Programs are comprehensive social change
systems that are generally broad in scope
with long term goals


LBJ’s Great Society Program
Projects are delimited implementation of
specified program goals usually bounded in
time and space by statute

The Texas Workforce Commission or more
specifically, the Upper Rio Grande Workforce
Development Board
Evaluation on the “Pyramid
Scheme”

Evaluation is rarely done of entire
programs, it is commonly found a the
level of projects


Elites don’t mind evaluating others… so
long as they are not evaluated in turn
As a practical matter, even socially aware
researchers must comply with this unstated
basic structure of evaluation research
Navigating the Political Landscape –
Anticipate or Be Tackled

Keep the following basic questions about political opportunity
structures in mind when proposing to engage in policy evaluation and
policy formation as an SMO member

How is formal power distributed?



How is informal power distributed?



Power brokers
Linkage figures
What are the relations between formal and informal power structures?



Power brokers
Linkage figures
Which is more powerful?
Which is your SMO better connected to?
What are the controversial “sore” issues?

Degree of polarization



Elites
Public
Where does the project fit into the community’s political cleavage structure?

Will it build bridges or dig ditches?
Taking Research From the
Activist’s Perspective


To many activists, the groundwork that
they do to prepare for a successful
action campaign is not “academic”
research
This view is slightly erroneous
Old Research Paradigm

Classical academic research is to be
value neutral


The researcher is to have no “horse that
he/she is backing”
This is to avoid slanting questions, design,
and methods to get “desired answers”
Post Modern/Realistic
Research Paradigm

No one is value free. The best we can
hope for an open accounting of the
researcher’s biases


This means that there is no reason that a
frank and open activist cannot do quality
research
There is really no such thing as “in too
deep,” unless the researcher decides to not
reveal how deep they are in
Intensive and Extensive
Approaches


Intensive – case study/qualitative
approaches concentrate in detail on one
or a few examples of a phenomena
Extensive – survey/quantitative
approaches try to discern general
patterns of behaviors or events
Intensive

The goal is really to get deep enough into the
situation to derive causal predictions



By getting into intense scrutiny, we try to
eliminate alternate explanations for a an event
With enough scrutiny, we also can become quite
familiar with possible history and maturation
processes that extensive analysis tends to gloss
over
The problem with this approach is that findings
tend to be idiosyncratic (unique to the case)
Extensive

The goal here is generalizability based on
causal hypotheses gained from initial
intensive studies


By dealing with repeated observations over space
and time, we can make a case for the observed
connection being a causal relationship or law of
human behavior
The problem with such studies is that findings
rarely investigate extensively multiple lines of
causality

One line is preferred and any others are by default either
used as controls or are not even investigated
Project-Based Research Cycle

The project-based
research cycle that
Stoecker (2003)
uses is akin to the
logic described in
Levin’s research
cycle (see CRIJ 3300
powerpoints)
Evaluation
Diagnosis
Implementation
Prescription
The Goals are Deceptively
Simple





Engage community members in evaluation
Diagnose community needs and strengths
Define potential solutions that a community
would find acceptable and consonant with
community norms
Use the community evaluators to help gain
acceptance of the implementation of the
selected solution(s)
Evaluate achievement of objectives according
to both sponsors and community clients
Finding One’s Place in the
Cycle

Diagnosis





Prescription




New services are in demand
New problem exists, cause unknown
General need to be “in touch” with community clients
Strategic planning
Finding best practices (common solutions) for our problem
Seeking to decide if best practice(s) will apply to our community
Seeking to efficiently mobilize SMO and service provider extant resources
Implementation


Moving aspect of SMO to the fore in aiding community
Using POS (political opportunity structure, see Tarrow 1994) to get policy enacted



Public
Private (corporate)
Evaluation
Seeking to find if our project had an impact

Seeking to determine if a change in strategy or tactics is needed to meet SMO goals
and community needs
Adapted from Stoecker 2005, p.76.

Who are These Community
Persons?


Staff and volunteers of local social movement organizations and
movements
The academic institutions and government institutions that
house evaluation units





Interested college students that either research the community out
of a personal connection or are committed to evaluation projects as
part of their degree
Academic researchers who are engaged in community service or
contract-research with sponsors of community affecting projects
Funding organizations
Service providers that do not house evaluation units
General community members who come in for a wide variety of
personal reasons
Who are These Community
Persons? (cont’d)
Government or
Academic
Institution
Service Provider
SMO
General Community
that Has Both
Assets and Needs
SMO
Service Provider
Government or
Academic Institution
Linkage
Adapted from p. 46 of Stoecker (2005)
Staff and Volunteers of SMOs1

These folks are the foot soldiers of
project based evaluation – the civil
society version of street level
bureaucrats

They have the history and maturation
knowledge essential for good research
design and implementation
SMO – Social Movement Organization, the general class of
organizations that community organizations belong to.
1
College Students and Faculty

These are the “specialists” or “guns for hire”


They have the methodological expertise to merge
with the case expertise of SMO members to
complete the designs and implementation plans
Keep in mind, the hero of “Have Gun Will Travel”
was not a callous fellow. He was actually an
idealist. Just because you are a methodological
expert does not mean that you are aloof
General Community Persons

These folks add perspective



SMO members tend to be a bit
overcommitted at times
Academics may not know enough facts on
the ground
Thus incorporating additional views can be
a real bonus, if done with a view to
improving the study design
Steps for Doing Project-Based
Research





Choosing questions
Designing the methods
Data collection
Data analysis
Reporting findings
Choosing Questions


The selection of what to ask is always a
negotiated process between stakeholders
The difference with project-based research is
the relative emphasis one puts on the
stakeholders

Here community stakeholders take moral (if not
actual) priority over financial and institutional
stakeholders

Theoretically the people matter most, only after their
needs are addressed do the considerations of finances
and legal mandates come into play
Designing the Methods



Design can be top-down or bottom-up
Top-down – prefabricated designs are
placed in many contexts to insure
comparability and reliability
Bottom-up – using community input
contextually relevant designs are
employed to insure validity and
consensual implementation of studies
Data Collection

A personal favorite…

I like to use students to do phone surveys and
intercept surveys. The students generally belong
to the El Paso “community”


This increases their ability to converse in a linguistically
appropriate manner with respondents
I would be abysmal at doing such interviews myself and
hiring folks from even a prestigious firm like Princeton
Research Associates would not make situation much
better
Data Analysis

Community contact brings nuance to the analysis


Non-SW US surveyors rarely understand the ire they raise
when not considering Latino/Hispanic a race
History and maturation processes for time series
analysis are fundamental

The totally bungled testimony by ASARCO’s expert witness
on the health impacts of reopening the plant foundered on
the out of towner’s ignoring the impacts that the plant would
have on Cd. Juárez and that Juárenses would make it an
issue at an El Paso hearing
Reporting Findings

While the word of an impartial judge is the epitome
of legal jurisprudence, it does tend to lack for the
persuasive impact one might desire


Thus the classical academic discourse of findings lacks as
Israelis say, “Zazooah” or “Ooomph”
Having a trusted community member express
findings that are counter to their known positions on
an issue is both highly credible and attention-getting

The challenge is getting a reputable community member to
express such counter-ideologicals

The best way to do this is to have credible community
members from all sides involved in the entire evaluation
process so that they buy into the end products
So How Do We Bring the
Community in?



Try to be useful
Use a multimethod evaluation style
Listen as well as talk, learn as well as
teach
Useful


Not every evaluation need be stripped down
to only the quintessential nuts and bolts of
measurement and data analysis. In fact, to
do so is generally counterproductive
Like in all genuine forms of politics, logrolling
is an essential practice

Trade of services – as part of the evaluation,
provide some community needed functions


Help organize community archives
Help generate donor lists
Multimethod Evaluation



Despite my desires to the contrary, not
all of us are statisticians at heart…
nertz!
Thankfully, nor are all of us folklore
specialists like my sister-in-law
We need both extensive/quantitative
and intensive/qualitative methods
Multimethod Evaluation (cont’d)

As my favorite Aesop’s fable goes, the
ability of three blind men to describe a
elephant/pachyderm depended on two
things



Touching multiple sites of the “big tusks”
(elephant)
Coordinating findings about the “wrinkly
skinned” (pachy-derm) critter
So do our evaluations
Don’t Be Your Professor or a
College Freshman



Dr. Levin is a notorious talker and a
lousy listener
College freshmen are notorious for not
actively engaging in classroom
discussions
Genuine collaboration needs a
maximization of all parties to both listen
and talk
What to Think About When
Collaborating with Community SMOs



SMO skills
SMO financial and personnel capacity
Compatibility of evaluation with SMO
goals
SMO Skills

By default, SMOs ought to be able to
contribute



Community member contacting expertise
Street credibility
In addition, SMOs may be able to
contribute


Archives
Funder/donor databases
SMO Financial and Personnel
Capacity





Clerical staff
Surveyors
Data entry specialists
Historians
Morale boosters – never underestimate
the impact of a warm relationship on
research productivity
Compatibility of Evaluation
With SMO Goals

Try to limit the input on evaluation
goals to consensus goals that SMOs
from several sides can agree on

This requires you to be slightly vague


This does not mean you lie!
This means keeping the goals to what is to be
evaluated. Refrain from goals about what is to
be done with the findings
Flipping the Coin to the Other
Side – SMO Questions




Can the academic/institutional evaluator
listen to the SMO when needed?
Can the academic/institutional evaluator
keep deadlines?
Can the academic/institutional evaluator
speak to the SMO in its own language?
What credentials does the
academic/institutional evaluator have?
Can the Evaluator Listen?

Despite Dr. Levin’s notoriety, the
Planned Parenthood Center of El Paso
decided to take the plunge.


Since that decision, Dr. Levin has learned a
new vocabulary of three letter words
If the formal evaluator can’t listen, the
SMO should drop the relationship

Otherwise they risk having “words put in their
mouth”
Can the Evaluator Keep
Deadlines?

In the period between retiring from the federal service and prior
to starting at IPED, I have had nothing but deep schooling in
the inability of academics to keep schedules

It is a norm for academics to





Solution 1 – work only with academic research units


Come late
Turn in reports late
Turn in budgets late
Never write contracts out fully
They are built to work with federal and state budgeting and grant
application schedules
Solution 2 – work with private research units that share
interests with your SMO

They too generally work on business schedules
What Credentials Does the
Evaluator have?


Credentials are a useful way to separate
out the wheat from the chaff, but have
one serious deficiency – young
talents/institutions have limited
credentials
Get as much input from the evaluator’s
colleagues on the evaluator’s abilities as
possible
Tackling the Four Aspects of
Project-Based Research

Diagnosis, Prescription,
Implementation, and Evaluation are all
worthy of discussion on their own.
Thus the remainder of this lecture, like
the Stoecker text, will address them
separately
Diagnosis




Forming a core
group
Needs assessment
Asset mapping
Melding needs and
assets
Evaluation
Diagnosis
Implementation
Prescription
Forming a Core Group

All social policy projects, like most complex
games, require teams. Project success
hinges on building a group that can



Survive mild degrees of internal conflict
Has enough breadth of external contacts to build
allies and exploit POS
Either intentionally links in with the existing issue
arena institutional structures or intentionally
decides to “work around” the existing issue arena
institutions
Needs Assessment

As discussed last week, needs
assessment is a complex form of
evaluation

In this respect, I differ with Stoecker, who
views only outcome evaluation/impact
assessments and process evalutations as
evaluation
Extensive Needs Assessment

Extensive needs assessment

This is essentially what we reviewed last
week



Surveys
Census data
SWOT analysis
SWOT Analysis

Survey of Strengths, Weaknesses,
Opportunities and Threats

This is a focus group approach




Bring in core stakeholders for a few 1-2 hour discussion
sessions to
Discussion 1 – Generate a list of SMO or community
successes and failures within a relevant time frame
Discussion 2 – Generate a list of SMO or community
strengths and weaknesses within a relevant time frame
Discussion 3 – Maximize strengths and avoid
weaknesses by utilizing POS of issue arena
Intensive Needs Assessment



Beyond the quantitative assessment of
community needs from surveys or census
data exist the views of informed stakeholders
The basic SWOT method does not address
the perceived depth of needs, polarization on
goals, etc
Intensive assessments will add greater
emphasis in SWOT to building lists of ranked
priorities for needs

Weaknesses and threats will be prioritized on
which need to be addressed most
Asset Mapping




Some of us find a glass of water half empty (your
Prof.), others find the same glass to be half full (his
pal Marika). It takes both perspectives to realize
what real potential the glass holds
Asset mapping is all about seeking to know what
resources the general community, relevant SMOs,
and service providers have as opposed to what they
need
The goal behind asset mapping is to sensitize
communities to their own potential to solve
community issues without government largesse
Appropriate tools include surveys and census data
Melding Needs and Assets




In truth, the best approach to a diagnosis of a body
politic is the same as of a body – you need to know
both what is healthy and what hurts
In the course of plotting out what hurts, you know
what might be healthy
In the course of plotting out what is healthy, you
know what you can rely on to deal with what hurts
It is much more sensible to approach a community
diagnosis with a mind to seek to find the gap
between internal needs and assets. This will avoid
the overstatement of community and SMO
helplessness that needs assessment often encouages
Prescription




Service versus policy
Finding alternatives
Evaluating
alternatives
Choosing an
alternative – C-B/CE Analysis
Evaluation
Diagnosis
Implementation
Prescription
Service Versus Policy
Service
 Inward-focused on
community
 Concrete plans
 Narrow application to a
specific goal and
context
 Can have policy
changes in it
 Think of this as an
experiment in Kuhn’s
(1970) sense
Policy
 Outward-focused on
setting agenda for best
practices solution to
problem
 Abstract rules that
service prescriptions will
fill in
 Wide application
 Think of this as a
paradigm in Kuhn’s
(1970) sense
Finding Alternatives

To come up with a vision that is broad, long-range, and has
substantive meaning

Dig in deep in the existing literature


Work with trade publications


In many service professions the federal government sponsors trade
publications. So do unions. Find the union websites and you will find
the publications
Ask friends in the field for their experiences


Use academic books, online academic journals, and research websites
We all know someone who “knows someone.” Screw up your courage
and ask around
Brainstorm with stakeholders


This is another focus group approach
Beware of groupthink!
Evaluating Alternatives

Seek the field recognized best practices


Pull evaluation criteria from prior formal evaluation studies






While specific criteria are not always available for a new issue, the basic
methods (C-B analysis) and measures ($, QLI, etc) are very adaptable
If you have access, convene a panel of experts
Add specific standards from the theoretical literature
Bring in the community context


Often the trade publications are better sources for overall assessments of
what works and what are duds
Different communities have different baseline conditions, what most folks
define as poor in El Paso is really “poorer” than what most folks define as
“poor” in Chicago
This will also help to establish the practical limits of the expected impacts
Bring in the project’s stated goals
Seek the stakeholder’s input for measurement criteria


After all, it is their lives that are impacted the most
Expect this input to be political in nature

This can help to integrate dissenting groups early in the process though
Choosing an Alternative

Rank criteria



Calculate benefits and costs


We will go into this in detail with Levin and McEwan (2000)
later this semester
Do forecasts


When resources are scarce, the easiest method is bottomline priority
Other methods are paired comparisons and Q-sorts
We will look at forecast modeling next week as part of the
REMI and REDYN models
Do decision tree analysis

We will cover this in Ch. 12 of Stokey and Zeckhauser (1978)
during November
Implementation



Research as action
Community research
Target research
Evaluation
Diagnosis
Implementation
Prescription
Research as Action

Organizing the community on its own behalf

The goal here is to use the project to bring the
community to support SMOs and service providers
– “Help us to help you”


Getting Hudspeth residents to support their public
libraries
Knowledge for its own sake

To get a community more aware of its own
problems so as to address them more effectively

Educate an AIDS susceptible community about safe sex –
the PPCEP-IPED project for the CDC
Community Research

History


Living documentaries


The community’s knowledge embodied via plays, “home movies,”
exhibitions, local cable TV shows
Community media


Most communities have persons and local archives – the Rogers
Park Historical Society has a great photo of some 1900s farm
horses fertilizing what would become David’s childhood backyard
Local newspapers, the internet, and local cable
GIS, ARCView, and “Mapping the Neighborhood”

The National Institute of Justice pushes crime mapping heavily on
us poor criminologists

NIJ loves to fund GIS based research in to the epidemiology of crime
Target Research


This is where you research an
organization or policy in order to target
it as needing to be changed by public
policies
This is an essentially confrontational
method
Target Research (cont’d)

Types of Targets

Government (agencies)


Corporations – includes financial corporations


Wells Fargo lending practices, ASARCO
Nonprofits


Texas Women, Infants, and Children Program help service
website speeds
Boys and Girls Club in NE El Paso embezzlement dispute
Foundations

Kaiser Foundation’s pattern of choices in spending money on
health care issues in the US tends to avoid questions about
reassessing the valuation of medicines, but tends to push
questions about supplementing purchaser ability to pay – in the
scale of things, this is not a great sin, but it is a needless
limitation on research
Government



Governments are the typical targets of
research since they are the primary source of
service provision and evaluation funds
These evaluations can range from purely proforma validation to highly investigative
A key tool for such research is the federal
Freedom of Information Act (FOIA) and its
state and local equivalents

You can open most government records upon
request, if the Homeland Security bills have not
closed them off
Corporations


Corporations are another typical target
Their actions are often traceable
through stockholder reports, Equal
Employment Opportunity Commission
complaints, National Labor Relations
Board complaints, local Better Business
Bureau complaints, and financial area
specific databases
Evaluation



External versus
participatory
Outcome versus
process
The participatory
approach
Evaluation
Diagnosis
Implementation
Prescription
Evaluation


As we discussed last week, evaluation can
come in many forms, from early stage needs
assessments and formative evaluations to the
more end stage process evaluations and
impact assessments
Stoecker shows a preference here for
outcome evaluations/impact assessments, but
I think this has more to do with the
organization of the text than that he rejects
needs assessment as being a forma of
evaluation
External Versus Participatory
External

Controlled by someone not
directly involved in the project
to avoid the evaluator “going
native”

Use




When project stakeholders are
riven by factions
When sponsors or agencies
demand it
When time is an issue
When highly specialized skills
are required


Participatory
Partly controlled by persons
directly involved in the project


Full control would really have a
bias problem – most of us really
cannot self critique
Use



When there is a manageable
consensus on goals and
methods
When SMOs and service
providers truly want to get
critiqued
When SMOs and service
providers have time and skills
to commit to project evaluation
Outcome Versus Process
Outcome

Focus is on the net impact of a
project on the welfare of a target
community

Tends to be holistic in evaluation
measures

Many measures will be combined to
present a “complete picture” at a
single point in the project – the
“end”


No social project has a real “end”
unless all subjects exposed and
those they contact are dead – e.g. a
whole community is exterminated
Will use standard array of social,
economic, and biomedical indicators
of well being – income, education,
leisure time, toxic content of air or
soil, incidence of cases of disease
etc
Process

Focus is on the steps towards
reaching an outcome

Can be incremental in evaluation
methods


Often multiple methods and points
in the process will be examined and
adjustments in both process and
evaluation will be done “on the fly”
Will use array of efficiency
indicators – service time, number of
cases processed, complaints
registered, etc
The Participatory Approach

SMO members, service providers, and
general community members can be
fruitfully involved in





Determining the question
Choosing the methods
Gathering the data
Analyzing the data
Presenting the data
Determining the Question


While a specialist has a prefabricated set of
questions to ask about a project, those most
engaged in the context will be able to add
specific questions or eliminate ones that are
unlikely to show sufficient variance due to
context
The naysayers among the stakeholders may
be co-opted at this stage by adding a
question or two
Choosing the Methods


Again, the specialist will have a prefab
set of common indicators of project
success
Stakeholders will know which are
practical to obtain, which are expensive
but manageable, and which are simply
boondoggles
Gathering the Data

Like Dorothy once said “There’s no data like
free data”… or was that “There’s no data like
home cooked2 data”? Anyway, the best data
is that which is



Stakeholders who have an interest in getting
useful evaluations will be the persons best in
position to


2This
High fidelity
Low cost
Access original source data
Be/recruit volunteer labor
is not an instruction to cook (fake your data). If you do this, I and every other
evaluator worthy of the title will hound you to your grave for dishonoring our
profession.
Analyzing the Data

Herein lies the evaluator’s best chance to be an educator



As a teacher of evaluation methodology for the MPA program and the Criminal
Justice program, I can attest to you that the skills needed for being a skilled
evaluator are in short supply in El Paso. You have a valuable commodity, but
like most forms of education, it gains value by teaching your consumers how to
use it and do rudimentary stages of it on their own
Have the SMO members, service providers, general community members, and
other interested stakeholders come in for





While participation in other stages may involve education, it tends to be more about
alerting people to skills they already possess, not giving them new ones
Training on SPSS, SAS, STATA, etc
A working knowledge of what the term “statistically significant” means
Basic cost-benefit/effectiveness analysis
Decision analysis
Remember, the idea is to give them enough to


Make them aware of the potential benefits to them from using evaluations
Make them aware of the need to bring you or a colleague in for specialized analysis
services

Yes, I guess this is saying that we want you to get people addicted to evaluation
Presenting the Data


Picture this: a city budget analyst stands in front of a city
council meeting and is roundly booed by all and sundry for
giving them news they don’t want to hear about programs that
must be cut to meet the council’s demand for a balanced budget
without raising taxes. Sound familiar?
Avoid! How? Simple! Bring the SMO members, service
providers, and other stakeholders into the presentation process



That way they are unlikely to boo, since they would be booing
themselves
They would be more understanding of the actual implications of the
findings
It allows findings to leak early… yes, leaks can be good sometimes

Policymaking subsists on strategic leaks to co-opt and demobilize
opposition
Bringing it All Around Again

Having touched on the high points of the project-based research cycle,
let’s be aware of some of its limitations

It tends to be too darn inward focused if the specialist is edged out to the
margins of the process



It cannot and should not entirely insulate the evaluators or evaluation
targets from criticism


Not all projects are created equal. Let’s face it, some projects stink and should
be terminated. Someone has to take the heat for the flawed design,
implementation, or less than kindly breaking of the bad news
It tends to be antagonistic towards traditional service providers (e.g.,
government agencies)


It is the evaluation specialist’s job to look beyond the project to the program and
to social welfare as a whole
Participatory process tend to marginalize experts in the name of an ersatz
“average Jane/Joe”
Sorry “Neos” of the world, but the government is rarely a “Matrix” enmeshed in
extracting a parasitic existence from you
Nevertheless, the project-based research cycle’s participatory emphasis
is well-worth the risks and limitations enumerated above