From Concept to
Implementation: The
Challenges Facing EvidenceBased Social Work
Faculty Research & Insights: A Series Featuring
CUSSW Faculty Research
Edward Mullen & Aron Shlonsky
September 22, 2004
Evidence-Based Social Work Is A
Concept Awaiting Implementation
• Much discussed
• Required in EPAS
• Not clearly Defined
• Not yet implemented
in pure form
“Placing
“Evidence-based
the client’smedicine
benefits has
first,
“--the
conscientious,
evidence-based
been defined
practitioners
as “--- the adopt
a process
conscientious,
of lifelong
explicit
learning
and
that
explicit
and
judicious
use
judicious
involves continually
use of current
posing
best
ofevidence
current
best evidence
specific
in
questions
making
decisions
of direct
practical
about the
importance
care of individual
to clients,
in
making
decisions
patients”
searching
(Sackett,
objectively
et al., 1996,
and p.
71);
efficiently
and thefor
"integration
the current
ofbest
best
regarding
the
welfare
of
evidence
research
relative
evidence
to each
with question,
clinical
service-users
and values"
carers”
and
expertise
takingand
appropriate
patient
action
guided
(Sackett,
by evidence”
Straus, Richardson,
(Gibbs et al.,
(Sheldon,
2003,
p. 1).
Rosenberg,2003,
& Haynes,
p. 6). 2000,
p. 1).
Origin of Evidence-Based Social
Work
– Adaptation of evidence-based approaches found in
medicine and healthcare
– Evolution from earlier attempts to join policy-practiceresearch
–
–
–
–
–
–
Effectiveness research
Eclecticism
Empiricism & the scientist-practitioner model
Social intervention research
Outcomes measurement
Systematic review methodologies – meta-analysis
– Eileen Gambrill early translator (1999)
– Leonard Gibb’s first workbook-text (2003)
– Aaron Rosen & Enola Proctor on guidelines (2003)
The EBP Model
Practitioner’s
Individual
Expertise
Best
Evidence
EBP
Client Values and
Expectations
Sackett et al., 1997
Newest EBP Model
Clinical state
and
circumstances
Clinical
Expertise
Client
Preferences
and actions
Research
Evidence
Haynes, Devereaux, and Guyatt, 2002
Steps of EBP
 Become Motivated to Apply EBP
 Step 1—Convert information need (prevention,
assessment, treatment, risk) into an answerable
question.
 Step 2—Track down current best evidence.
 Step 3—Critically appraise the evidence.
 Step 4—Integrate critical appraisal with practice
experience, client’s strengths, values, and
circumstances.
 Step 5—Evaluate effectiveness and efficiency in
exercising steps 1-4 and seek ways to improve them
next time.
 Step 6—Teach others to follow the same process
Gibbs, 2003
The Cycle of EBP
Appropriate for
this client?
Contextual
assessment
Clinical state
and
circumstances
Actuarial
risk
assessment
Clinical Expertise
Client’s
preferred
course or
at least
willing to
try?
Client
Preferences and
actions
Current
Best
Evidence
Other valid
assessment
measures?
Effective
services
Barriers (e.g.,
cultural conflict
Adapated from Haynes, Devereaux, and
Guyatt, 2002; as presented in Shlonsky and
Wagner, in press.
Hallmarks of EBP
• Individualized assessment (strengths, needs,
values, preferences of client)
• Well-formulated question (database query)
• Well-executed search of research literature
• Deciding its applicability to clients
• Considering the evidence together with the
values and preferences of the client.
Sackett et al., 1997; Gibbs, 2003
Origins of Evidence-Based
Healthcare & Public Policy
• Used to inform health care policy at hospital, regional, &
national levels (Davies & Boruch, 2001; Romanow, 2002;
Gray, 2001; Davies, Nutley, & Smith, 2000)
• Became component of New Labour’s modernizing
government strategy - extended to other sectors
including social care (Davies, 2004; Walker, 2001)
• Facilitated by:
–
–
–
–
–
Cochrane & Campbell Collaborations
UK ESRC’s evidence-based practice & policy centres
UK Social Care Institute for Excellence
Sweden Institute for Evidence-Based Social Work
Publications – e.g., Evidence-based Policy: A Journal of
Research, Debate and Practice
Policy Applications
• An approach to policy that:
– helps people make well informed decisions
about policies, programs & projects
• EBPolicy contrasts with opinion-based policy
– which relies heavily on either the selective
use of evidence (single studies irrespective
of quality)
– untested views of individuals or groups
Philip Davies (2004), Director of Policy Evaluation in the
Cabinet Office of the UK Prime Minister’s Strategy Unit
What is evidence-based
healthcare?
•
•
•
Discipline centered on evidence-based
decision-making about:
-Groups of patients
-Populations
Evidence from wide range of disciplines
Use of scientific research findings and
logic
– Healthcare problems
– Health improvement
– Decision-making in healthcare
Gray 2001
Evidence-Based Policy Definition
Decision-making process in
which policy-makers,
purchasers, managers, or
practitioners together with
key stakeholders make
decisions about problem or
need assessment, goals &
objectives, interventions &
outcomes
EBPolicy Process (Gray, 2001)
Getting Evidence
into Practice
Finding &
Appraising Evidence
Capacity
Building
Sources of Evidence in EBP
Impact
Evaluation
Attitudinal
Studies
Implementation
Evaluation
Descriptive
Studies
Ethical
Analysis
Economic
Analyses
Statistical
Modeling
EBPolicy Steps
• Finding and appraising evidence pertaining to the
decision at hand:
– Assessment or intervention options
– Intended outcomes
– Research method
• Developing organizational capacity
• Getting the evidence into practice
– Preparing a policy
– Making cultural changes
– Designing systems for implementation
– Implementation
– Monitoring
– Audit
Examples of Evidence-based
Policy Initiatives
• EVIDENCE AND POLICY: A JOURNAL OF
RESEARCH, DEBATE AND PRACTICE
• ESRC Evidence Network - UK Centre for
Evidence Based Policy & Practice
• U.K. Cabinet Office
• University of St. Andrews
• Romanow Commission – Canada
• U.S. Implementing Evidence-based Practices
Project
Arguments For Evidence-Based
Social Work (EBSW)
•
•
•
•
•
•
•
•
Enhances decision quality
Fosters learning of assessment skills
Incorporates client values and expectations
Fosters evidence search and appraisal skills
Makes best use of best evidence
Framework for self-directed, life-long learning
Identifies gaps in knowledge
Common interdisciplinary language
Sackett et al, 2000
Arguments For Evidence-Based
Social Work (EBSW)
• Eschews authoritarian practices and policies
• Promotes SW ethics through:
– Informed consent
– Using effective services
– Wisely allocating scarce resources
– Involving clients in practice process
– Individualizing & personalizing decisions
Gambrill, 2003
Arguments For Evidence-Based
Social Work (EBSW)
– Helping clients develop critical appraisal skills
– Involving clients in design and critique of
practice and policy related research
– Involving clients as informed participants who
share in decision making
– Recognizing client’s unique knowledge in terms
of application concerns
• Promoting transparency and honesty
• Encouraging a systemic approach for integrating
practical, ethical and evidentiary issues
• Maximizing the flow of knowledge and information
about knowledge gaps
Gambrill, 2003
Challenges Facing EBSW
•
•
•
•
•
•
•
•
•
•
Evidence-of-effectiveness
Authority
Conflicting hierarchy
Definition of evidence
Shortage of evidence
– Variation by field of practice
– Inflation of evidence
– Small & transient effects
– Assessment validity
Nomothetic versus ideographic knowledge
Efficacy versus effectiveness
Diffusion, adoption and implementation
Policy & system level issues
Training & sustaining
Evidence-of-effectiveness
Challenge
• A key assumption of EBP is that it results in
better outcomes than other forms of practice
– Yet, this assumption is not empirically testable:
• “This proof is no more achievable for the new paradigm
(EBM) than it is for the old, for no long-term randomized
trials of traditional and evidence-based medicine are
likely to be carried out.” (Evidence-Based Medicine
Working Group, 1992, 2424)
Authority Challenge
• As originally conceived the founders of
evidence-based medicine envisioned that
medical practitioners would seek out
evidence & appraise the quality of that
evidence, alone or in teams of practitioners
• “The new paradigm puts a much lower value
on authority. The underlying belief is that
physicians can gain the skills to make
independent assessment of evidence and
thus evaluate the credibility of opinions
being offered by experts” (Evidence-Based
Medicine Working Group, 1992)
Authority Challenge
• Lack of evidence that EBM can be
transmitted into practice
– Medical students have not shown an interest
in becoming “evidence-based practitioners” –
only “evidence users”
• As evidence users rather than
independent appraisers, practitioners are
simply replacing one authority with another
– researchers are the new authorities!
Authority Challenge
• To date it has not been shown that average
practitioners can gain the skills needed to
conduct independent assessments of the
evidence
• Practitioners can become “evidence-users”
not “evidence-based practitioners” say
Upshur & Tracy
• “Evidence-users” accept evidence that has
been vetted by others – thus replacing one
authority for another! But what is the basis
for the authority of the evidence appraisers?
Conflicting Hierarchy Challenge
• Hierarchy of Evidence Ranked by Quality
– N of 1 randomized trials
– Systematic reviews/Meta-analysis of
randomized trials
– Randomized Controlled Trials
– Systematic review of observational studies
addressing patient-important outcomes
– Physiological/Laboratory experiments
– Unsystematic clinical observation
Conflicting Hierarchy Challenge
• Practice recommendations (e.g., practice
guidelines) are to be based on the
soundness of the evidence
• Many evidence hierarchies have been
published & these hierarchies are not
commensurable (e.g., randomized, controlled
trials and meta-analysis versus well
designed epidemiological studies)
• Fundamentally these hierarchies do not rest
on evidence but rather on the consensus of
experts – the beliefs of the few!
Definition of Evidence Challenge
Because contemporary health care and
human services are multidisciplinary
there are many differing perspectives
that require consideration
Accordingly, the concept of a univocal
understanding of evidence is not
sustainable
A restrictive view of evidence is not
sustainable
Shortage of Evidence Challenge
• The question that faces proponents of EBP is whether
there are enough high-quality studies so that evidencebased decisions can be made.
• Surprisingly for a field that places a high premium on
research, few studies have examined this.
• The conclusion at this point, based on just a few studies,
is that there are still many decisions that are made that
are not based on good evidence, but the picture is not
nearly as bleak as opponents to EBP would have us
believe.
• Professionals must remember, though, that when they
make decisions for which little or no evidence exists, that
they should exercise caution and perhaps be even more
vigilant in monitoring outcomes.
Shortage of Evidence
• Varies by field of practice
• Confounded by inflation of evidence by
proponents and users
• Clouded by dominance of small and
transient effects
• Applies to assessment evidence as well as
intervention evidence
Shortage of Evidence Varies by
Field of Practice
• Adequacy varies considerably by field of
practice
• Most robust is mental health
• In other fields of practice much more of a
patchwork
• Unevenness of the research base across
fields of practice looms as a major
challenge
Inflation of Evidence Challenge
• Bias in reported research due to:
– Tendency for investigators or assistants to
give “a leg up ” to interventions they favour “expectation for improvement”
– Dependence on client self-report to measure
intervention outcomes which may reflect
social desirability, expectancy, & cognitive
dissonance effects
Small & Transient Effect Size
Challenge
• The rule is small effect sizes which may
not be clinically or social important
• Gains frequently do not persist, especially
for complex and chronic problems such as
addressed by social workers
Assessment Challenge
• Reliable, valid, & relevant assessment tools are
needed to support evidence-based social work
practice
• Rapid assessment measures have contributed
greatly (Corcoran & Fisher, 2000)
• Social work practitioners generally do not use
any standardized assessment procedures to
guide their practice (Mullen & Bacon, 2004)
Shortage of Evidence Challenge
• Evidence-based social workers must remember that
when they make decisions for which little or no evidence
exists, that they should exercise caution and perhaps be
even more vigilant in monitoring outcomes
• Knowledge gaps point the way to needed research
• Methodological corrections should be instituted to deal
with the inflation of evidence issue
• Where interventions are found to have small with
transient effects of limited importance this should be
acknowledged and addressed
• The underutilization of assessment instruments should
be corrected by enhance training in the use of
appropriate instruments & dissemination of information
about available instruments.
Challenge of Applying Results to
Individuals
• Results of RCTs are analyzed by comparing the mean
score of the experimental group against that of the
placebo or control group (or some comparable summary
statistic). This masks the fact that there is always
individual variability around the means, & overlap in the
distributions of scores for the two groups. The result of
this is that a proportion of people in the experimental
group actually do worse than some in the control group
and, conversely, some in the comparison group improve
more than some people in the active treatment group.
• Practitioners cannot blindly apply a “proven” procedure
and assume that a particular individual receiving that
procedure will benefit
Challenge of Applying Results to
Individuals
• We are at least able to quantify the probability
with which an individual person will respond to a
given procedure (Number Needed to Treat)
• Alternative to using evidence-based
interventions – with their known rate of failure –
is to use unproven procedures, based only on
the hope that they may work
• Practitioners can and should view each case or
situation as an “N = 1” study and collect data
Efficacy Versus Effectiveness
Challenge
• The efficacy versus effectiveness challenge requires a
careful consideration of the trade off between internal
and external validity
• Claims to evidence-based practice typically must be
grounded in random, controlled trials which usually
translate into efficacy studies
• Such studies do not address how effective such
interventions would be in real world contexts
• At the present time there is a paucity of evidence of
relevance to social work based on effectiveness studies
conducted in real world contexts
• it may be more efficient to begin with studies based on
conditions as realistic as possible so as to avoid the
misinformation that too frequently comes from
misinterpretations of efficacy studies
Adoption & Implementation
Challenge
• Two major
challenges:
(1) adoption &
effective
implementation of
evidence-based
practice in social
agencies
(2) educating for
evidence-based
practice
Sisyphus
Training, Time, Resources
Challenges
EBP requires:
• Training in search
techniques
• Training in critical
appraisal
• Computer
resources
• Electronic
resources
Killer B’s
• Low BASE rate
• Incompatible client or
community BELIEFS
• BAD BARGAIN in
terms of scarce
resources
• BARRIERS too high
Adoption & Implementation
Challenge
•
•
•
•
•
Motivate core group
Acceptance of EBP
Facilitate adoption
Implementation of effective services
Evaluate outcomes
2.5%
13.5%
34%
34%
16%
Innovators
Early
Adopters
Early
Majority
Late
Majority
Laggards
Gira, Kessler and Poertner (2004)
• Survey of reviews of
adoption of research
evidence medical practice
• Educational outreach
visits and audit and
feedback showed weak to
moderate effects
• Certain types of
continuing education and
the use of computers as
decision aids showed
moderate effects
• There are no ‘magic
bullets’ (Oxman et al.,
1995)
•“The literature from health
care suggests that
disseminating information
alone is insufficient. Many
interventions have been
designed to improve
practitioners’ adherence to
EBP guidelines and are
differentially effective. To
date, no intervention has
demonstrated powerful
effects” (p. 77-78).
Adoption & Implementation
Challenge
•
•
•
•
•
•
•
•
Can social agencies adopt EBP approach?
Can social workers become EB practitioners?
What are the barriers? Facilitators?
What resources are needed?
What training is required?
What systems need to be established?
How will EBP change client experience?
What outcomes will be achieved?
Challenges in Teaching
Evidence-based Social Work
• Fear of Loss of Human
Context
• Disconnect between
classroom and field
• Overwhelmed by new
skills and knowledge
needed
(Bilsker & Goldner, 2004)
A Challenge to Critics
• Knowledge and
information revolution
• Valuable resource
• Cannot be ignored
• Must be harnessed
A Measured Optimism
• Real world constraints
-limitation of approach
-social and political realities
• Great promise, but need for caution
-harnessing knowledge and information
-management of scarce resources
-unknowns abound
• Hold EBP to its own evidentiary standards
-the jury is still out
Knowledge needs to be managed much more
effectively than it has been in the past
The challenge is to
rethink our view of what
knowledge is & how best
to facilitate its rapid
generation, sharing, and
application in a manner
that closes the policyresearch-practice gap