EVIDENCE BASED
PROGRAMS
Dr. Carol Albrecht
Assessment Specialist
Utah State Extension
carol.albrecht@usu.edu
Evidence Based Programs




Relatively new approach
Program decisions are based primarily on scientific
method and scientific evidence
Use best evidence available to make decisions
about program
Recognizes that knowledge is provisional and
subject to refutation
Evidence Based Programs combines (1) Program Manager’s Expertise,
(2) Client Attributes, and (3) Research Findings
Research
Findings
Program
Manager’s
Expertise
EBP
Client
Attributes
Best Research Evidence Available Can
Be Found in Current Literature
Step 1 – Review The Literature
“Track down” the evidence- ongoing process
 Note Inconsistencies
 Note research suggesting some interventions are
ineffective

Step 2 – Critique the Literature You Find

Examine methods used*
Client Attributes



Take values, concerns, expectations of client into
consideration
Involve clients in decision making process
Determine if this program will “fits” your client.
Not “one size fits all” situation
 Consider race, gender, idiosyncratic characteristics

Program Manager Expertise


Use Common Sense
Avoid “fallacies” which include
 Overgeneralization
- Stereotyping
 Selective Perception/Observation
 Ex-post Facto Hypothesizing
 Provide
alternative explanation after the “study”
 Accepting
“new” as better
 Accepting “traditional” as better
How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies


Step 4 – Apply the Literature to Your Question and/or
Program
Step 5 – Conduct Your own Research
Step 1 – Formulate a Question

What objectives do you want to accomplish?
 What
intervention is necessary to reach objectives?
 Does
Intervention A work?
 Is Intervention A cost effective?
 Does Intervention A work better than Intervention B?
 Does Intervention A work for this type of client?
 Does intervention A work in this kind of situation?
Useful Acronym for Formulating
Question

Use acronym CIAO
 Client
characteristics
 Intervention being considered
 Alternative intervention being considered (if any)
 Outcome of intervention (does it work?)
Example

Client Characteristics
If children involved in 4-H

Intervention
…..personally interact with a mentor for several
years…..

Alternative Intervention
…..or communicate with a mentor over the internet…..

Outcome
…..are they more likely to seek post-high school
education?
Step 2 – Search for the Evidence

Helpful “hints” to finding articles

Use Google or Google Scholar and type in correct search
terms
Combine general terms (e.g., volunteers, evaluation etc.)
 With more specific terms (i.e., grades in high school, crop yield
etc.)


Top Down and Bottom Up Searches (Mullen 2006)
Top Down – Find Articles/Books that have Compiled the Literature
(i.e., meta-analysis on your topic)
 Bottom Up – Find a very recently published article and look at
references.

Examples of Resources for “Top Down” Searches









*Campbell collaboration: http://www.campbellcollaboration.org/index.html
**Cochrane Collaboration: http://www.cochrane.org
Substance Abuse:
http://nrepp.samhsa.gov/htpp:/www.modelprograms.samhsa.gov/index.
htm
Crisis Intervention and Domestic Violence National Resource:
http://www.crisisinterventionnetwork.com
Expert Consensus Guidelines series: http://www.psychguides.com
National Guidelines Clearinghouse http://www.guidelines.gov
National Institute on Drug Abuse: http://www.nida.nih.gov/
Substance Abuse and Mental Health Services Administration:
http://www.samhsa.govindex.aspx
Focuses on Social Welfare, Education and Criminal Justice
**Focuses on Health Care Interventions
*
Step 3 – Critically Appraise Relevant
Articles

Emphasis Should be on Methods Used to Collect
Empirical Information
How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies


Step 4 – Apply the Literature to Your Question and/or
Program
Step 5 – Conduct Your own Research
Step 4 – Apply the Literature to Your Research
Question



Which intervention currently has the best empirical
support?
Is the situation same/similar to yours?
Are the participants in the study the same or similar
to the ones in your program/study?
Step 5 – Conduct Your Own Research





Use Literature Review to help you develop valid and reliable
indicators.
Use Literature Review to help you determine your sampling
technique and sample size.
Use the Literature Review to help you determine the most
appropriate type of data collection.
Use the Literature Review to help you determine ways you can
further contribute to the research (i.e., what is missing?)
Evaluate impact of program on your participants
Concerns About Evidence
Based Programs

Merely Cost Cutting


Have to use the methods “in stock”
Evidence Based May not be Applicable
Leads to rigid implementation
 Your participants may be different



Evidence is in Short Supply – not enough research
Real World Obstacles Hinder Implementation of
Programs



Time and money are scarce
Devalues Qualitative Research
Argument that “Empirical Evidence” is not obtainable
Suggestions for Appraising
Journal Articles in the Literature

Step 1 – Identify Purpose
 Stated
 This

purpose of article
should appear in first paragraph(s).
Step 2 – Literature Review
 Is
it adequate?
 They
should have at least 15 references.
Appraising Literature

Step 3 - Identify objectives/outcomes
 Do
they identify and define their outcomes/objecties
 It
must be clear and succinct.
 They must be clearly related to the implementation of
program.
 How
do they measure their concepts/outcomes?
 Are
measurements valid? (face validity)
 Are measurement reliable (consistent)

Could you repeat study?
Appraising Literature

Step 4 – Appraising Sampling Technique
 It
must be clearly specified?
 What
is their unit of analysis?
 Do you know exactly how these units will be selected?
 It
must be appropriate?
 Does
the sampling technique “fit” with the data collection
technique?
 Do
they draw valid conclusions based on sampling
technique?
 Note
– if they generalize they must have random sample
Remember the Sampling Techniques

Random Sample – can generalize
 Simple
or Systematic Random
 Multi-stage Cluster
 Stratified Random

NonRandom Sample – cannot generalize
 Theoretical
 Snowball
 Convenient
or Purposive
Appraising Literature

Step 5 – Research Design/Data Collection
Technique
 It
must be clearly specified?
 Do
 It
you know exactly how they will collect their data?
must be appropriate
 Keep
in mind the strengths and weaknesses of the different
research designs/data collection techniques
Remember Research Designs or
Data Collection Techniques

Data Collection/Research Design
 Survey
or Questionnaire
 Secondary
 Experimental
 Lab
Experiments
 Quasi – Field Experiment/Trial
 Field
Research (Qualitative)
 Participant
Observation
 Content Analysis
 Intensive Interviews
 Case Studies
Appraising Literature

Step 6 - Critique Tables/Graphs/Charts

Their analysis must be correct?
Did they include descriptive statistics of the sample?
 Did they include inferential statistics and discuss level of
significance?
 Are their tables constructed correctly?


They must be interpreted accurately.

Does the text accurately describe the data?
They should look professional
 Do they provide sufficient information?


Do they tell you how they measured the indicators used in the
tables?
Appraising Literature

Step 7 – Summary and Conclusions
 Do
they accurately summarize their research?
 Do they identify strengths and weaknesses?
 Do they acknowledge limitations?
 Do they make suggestions for future research?
Easy Steps







1.
2.
3.
4.
5.
6.
7.
Identify Purpose
Critique Literature Review
Identify Success
Sample Technique
Data Collection Technique/Research Design
Tables and Charts
Summary and Conclusions
Contact Information





Dr. Carol Albrecht
Assessment Specialist
Utah State University
979-777-2421
Carol.albrecht@usu.edu