SWK 7401. Evaluating Social
Work Practice
Alvin Mares, PhD, LSW
Autumn 2012
Week 1 (Aug 22nd & 24th)
• Introductions
– Instructor
– Students and “memorable client” from 1st yr field
placement
• Organization of Textbook
– Overview (Chpts. 1-2)
– Measurement (Chpts. 3-10)
– Evaluation Designs & Decision Making (Chpts. 11-15)
• Review of Syllabus
– Assignments
– Schedule
Introductions
• Instructor
– Practice, program & policy-level interventions for atrisk, TAY (18-29) to prevent chronic homelessness
• Emphasis on post-secondary education & vocational training
• Students
– Name
– BSSW/BSW Field Placement (OR) Work/Volunteer Exp.
• Memorable client
–
–
–
–
–
–
First name
Age
Gender
Presenting problem
Assistance provided by you (i.e., intervention)
Outcome/disposition
Wednesday Evening Section
Student
Agency
Aleta
Marion Area Counseling Center
Alisha
Brittany
North Central Mental Health Services, Inc.
Chelsea
United Methodist Children's Home
Emily
OSU-Hospital East-Talbot Recovery Svcs
Jane
Union County Dept Job and Family Svcs
Jon
Julie
Ohio Department of Youth Services
Memorable Client
Wednesday Evening Section (con’t)
Student
Agency
Kim
OSU-Hospital East-Talbot Recovery Svcs
Krystle
Directions for Youth and Families
Mara
Marissa
Muriel
Directions for Youth and Families
Nicole
Rosemont Center, Inc.
Shannon
Taylor
Franklin County Children Services
Memorable Client
Friday Morning Section
Student
Agency
Jason
Loving Care Hospice, Inc
Julie
Katie
Franklin County Children Services
Mandy
Amethyst Inc.
Michael
Marion Area Counseling Center
Michelle
North Central Mental Health
Molly
Columbus Public Health-Community
Outreach Assistance Team
Sarah
Tyler
Volunteers of America, Greater Ohio
Yohana
OSU Multicultural Center
Memorable Client
Organization of
Textbook
Review of Syllabus
Assignments
• Attendance ………. 15 pts.
– 1 pt./class
• Discussion ?’s …… 15 pts.
– 1 pt./chpt.
– Download from Carmen
• Quizzes …………….. 40 pts.
– 20 pts. each
– Drawn from Discussion ?’s
• Evaluation plan … 30 pts.
– Rubric forthcoming
Total ………………. 100 pts.
•
•
•
•
•
•
•
•
•
•
•
•
•
Schedule
Wk. 1: Intro/overview
Wk. 2: Chpts. 1 & 2
Wk. 3: Chpts. 3 & 4
Wk. 4: Chpt. 5
Wk. 5: Chpt. 6
Wk. 6: Chpts. 7 & 8 (Quiz 1)
Wk. 7: Chpts. 9 & 10
Wk. 8: Chpt. 11
Wk. 9: Chpts. 12 & 13
Wk. 10: Chpts. 14 & 15 (Quiz 2)
Wks. 11-13: Evaluation plan
Wk. 14: Off
Wk. 15: Wrap-up
Homework Due Aug. 29th & 31st
• Read Chapters 1 & 2
• Answer Discussion Questions for Chpts. 1 & 2
downloaded from Carmen
– Highlight & type answers to questions in MS Word
• Bold correct answer for T/F and MC questions
• Type answers for essay questions
– Brief answers (3-5 sentences) sufficient for most essay questions
– As much as possible, apply concepts to your work last year with
your memorable client when answering essay questions
– Bring printed copy of questions with answers to class
to share your thoughts with others and turn in at end
of class; pass/fail grading (0 pts. if late or incomplete)
Week 2 (Aug 29th & 31st)
• Chapter 1 Introduction to Client-Centered
Evaluation of Practice (C-CEP)
– 8 steps/components
• Case study (Phillip)
• Graph (Fig. 1.1, p.8)
• Chapter 2 Conceptualization: Naming What We
See in the Client Situation
–
–
–
–
Concepts
Propositions
Theories: General vs. Local
Intervention Design, C-CEP & Logic Model Elements
• Case study (Ben & Katherine)
CHAPTER 1:
Introduction to Client-Centered
Evaluation of Practice
Chapter 1 Introduction to ClientCentered Evaluation of Practice
• C-CEP framework: 8 steps in evaluating practice
– Case study example of Phillip, 10 y/o boy with
diabetes who is feeling depressed
1. Client defines goals
– “feel pretty good most of the time” [positive self-image]
– feeling ladder from 1 (worst) to 5 (best); 3=ok
2. Select intermediary objectives and their specific targets
– objective: weight loss
– targets (empirical proxies)
» exercise: riding bike around block 3 times
» healthy diet: snack on carots & fruit
Chapter 1 Introduction to ClientCentered Evaluation of Practice
• C-CEP framework (con’t)
3. ID evidence-based general practice from literature +
evaluation-informed specific practice from work with
client
– review of literature to find promising or best practices
– tailoring/application of general best practices to client’s
situation
– feedback from client over time on effectiveness of
practice/intervention
4. Collect and plot data on graphs
– Y-axis: goals, objectives, targets (e.g., 1-5 feeling about self)
– X-axis: time (e.g., days)
– pattern of targeted events (improvement, decline, no change)
during various periods (baseline (A), intervention (B),
maintenance (M), follow-up (F))
Chapter 1 Introduction to ClientCentered Evaluation of Practice
• C-CEP framework (con’t)
5. Select benchmarks defining “success”
– client-designated (e.g., “4=positive most of the time”) and/or
– norms from standardized scales, agency/admin standards, etc.
6. Review and discuss with client at end of intervention
phase (B) extent to which goals attained, helpfulness of
intervention
7. Transition to maintenance phase (M)
– Client assumes responsibility for ongoing intervention
his/herself; drives solo (rather than w/ you in passenger seat)
8. Analyze empirical data (graphs) & compare with
qualitative data (sharing during intervention sessions)
C-CEP Framework Summary
1. Identify the Client Who Defines the Goals in the Case
2. Select Intermediary Objectives and Their Specific Targets
3. Identify Evidence-Based General Practice From the
Literature and Evaluation-Informed Specific Practice From
Working With the Client
4. Collect Data in an Ongoing Basis and Plot Data on Graphs
5. Construct Clear Benchmarks to Know When the Results
Are (or Are Not) Successful
6. Engage the Client in Determining If the Client’s Goals Have
Been Attained During Intervention
7. Introduce a Maintenance Phase in Which the Client Is in
Complete Control of the Intervention on His or Her Own
8. Analyze Patterns of Data to Coordinate With the Client's
Statements
C-CEP Graph
Chapter 1 Introduction to ClientCentered Evaluation of Practice
1-1. Single-system designs are used only for individual
clients, especially in institutional settings.
A. True
B. False
1-2. Single-system designs can be used in many different
professional settings where clear targets can be defined and
clear actions can be taken to attain these targets.
A. True
B. False
1-3. Evaluation is built into the codes of ethical practice
of many helping professions.
A. True
B. False
Chapter 1 Introduction to ClientCentered Evaluation of Practice
1-5. Evidence-based general practice is:
A. The same as evaluation-informed specific practice, when
used with a given client.
B. The same as empirical research, using control groups and
experimental groups.
C. The truth, and so practitioners ought to use it exactly as
discussed in the literature.
D. The best available information on some general topic, but
it ordinarily cannot be used directly with your specific client.
Chapter 1 Introduction to ClientCentered Evaluation of Practice
1-6. When you use evaluation-informed specific
practice, it means:
A. That you have used have used an evidence-based model,
without varying from the protocol or trying to alter the
model to fit your specific client.
B. That the information obtained in your single-system
evaluation specifically fits this particular client, and you can
make meaningful statements as to how well that client is
doing.
1-7. It is easy to mix up goals and objectives in
evaluation. Take a case example from your own practice
experience (or make one up), and distinguish goals and
objectives in the case.
Chapter 1 Introduction to ClientCentered Evaluation of Practice
1-8. Identify at least three basic ingredients for all
single-system designs used in evaluation of practice
from this list of terms:
•
•
•
•
•
•
•
•
a specified target (problem or desired goal)
operational definitions of targets, interventions, benchmarks
establishing a baseline
different phases (baseline, intervention, maintenance,
follow-up)
take repeated measures in both baseline and intervention
phases
use of an evaluation design, at least the basic design
analyses of data
decision making
Chapter 1 Introduction to ClientCentered Evaluation of Practice
1-9. We defined targets as empirical proxies. Explain what
this means in your own language, but include in your
discussion the idea of operational definitions. (That is,
illustrate this phase, “empirical proxies,” by describing a
client situation in which such proxies are involved.)
1-10. Why do we assert that you need to collect data on a
repeated basis, even at those times when you are not even
doing any intervention?
1-11. You are working with a group of young women (ages
15-20 years), all of whom are pregnant with their first child.
They have determined, among themselves, to continue
their education. Give one example of a benchmark in these
situations. (Hint: is it measurable? How can we know
whether or not each client has achieved this benchmark?)
BREAK
CHAPTER 2:
Conceptualization: Naming What
We See in the Client Situation
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Concept
– Also known as a construct
– An arbitrarily constructed general term derived
from some class of events
– Serves as a building block for larger conceptual
terms (e.g., propositions, theories) and as a link to
the information network where you can find
evidence-based practice ideas (Britner & Bloom)
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Propositions
– Sets of concepts related in such ways as to
describe:
(1) what we assume to exist, so we can get to the
second proposition;
(2) what we hypothesize to exist and which we
can test in the empirical world; or
(3) something we prefer or value, for which there
is no empirical test.
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Theories
– Systems of concepts and propositions that focus
on, describe, explain, and predict a limited domain
of the world.
– General theories are abstract enough to apply to
broad topics (e.g., behaviorism)
– Local theories deal with specific situations (for a
given client and their context)
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Case study: Ben & Katherine (Kat)
– Young couple w/o children arguing about…
•
•
•
•
not having enough money (primary presenting problem)
having children
sex (secondary presenting problem)
religion
– “Assessment and Problem Formation” phase of the
Social Work Problem-Solving Process (Dr. Bean)
• Problem definition = primary presenting problem(s)
– A problem well-defined is a problem half-solved
The Social Work Problem-Solving Process
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Case study: Ben & Katherine (Kat)
– Given primary concerns: financial problems & sex
– C-CEP goals: financial solvency & sexual satisfaction
– Concepts and propositions
• Concepts relating to money
– work; earnings; recurring bills to pay; other expenses; part-time vs.
full-time employment
• Propositions
– Prolonged imbalanced of earnings compared to expenses leads to
psychological stresses that may affect the continued well-being or
survival of the group [marital relationship].
– When the stronger partner does not earn enough money to pay
the bills, that partner will compensate his or her weakened
situation by becoming more demanding in other domains (e.g.,
sex), which may be unwelcomed by the other partner.
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Case study: Ben & Katherine (Kat)
– Theories
• Local theory
– Therefore, only when former (economic) difficulty is resolved
will the latter (sexual conflict, etc.) be resolved.
• General theory(ies)
–
–
–
–
Family Life Model (establishment)?
Family Life Cycle for families (new couple)?
Erickson’s developmental stages (industry vs. inferiority)?
Levinson’s developmental phases (early adulthood)?
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Case study: Ben & Katherine (Kat)
– Action plan based on…
• Presenting problems of financial troubles (primary) & sex
(secondary)
• Goals of financial solvency & sexual satisfaction
• Local & general theories
• General theory(ies)
– Ben to find higher paying job to increase HH income
• Fig. 2.1: Graph of job status (old vs. new)
• Fig. 2.2: Graph of HH income ($ range per week)
– Kat to extend educational career
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
• Case study: Ben & Katherine (Kat)
– C-CEP and Logic Model Elements
Condition
Targets
Intermediate Outcomes
Long-term Outcome
Problem
Objective 1
Objective 2
Goal
Financial
insolvency
• New job
• HH income
>300/wk
Decreased
psychological
stress
• Operational
measure?
Increased
sexual
satisfaction
• Operational
measure?
Increased wellbeing of
marriage
relationship
• Operational
measure?
Propositions & Theories
Assumptions
Intervention-related activities
Inputs, Activities & Outputs
The Social Work Problem-Solving Process
Thoughts On Interventions
• Remember that good theory will have direct
relevance to practice
• There will be a translation process where you will
using concepts and propositions of the theory to
frame your practice model or intervention
• Being aware of your ‘theory of change’ and
communicating that to your client(s) is an essential
component of the helping process
• Objectives and targets, guided by local theory
• Empirical evidence + theories (informed by review of
literature & knowledge of client) are bases of plans
of action [and revisions to action plans over time]
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
2-1. A concept is a general and abstract term, for which
there may be many instances in the real world.
A. True
B. False
2-2. Two concepts combined with a linking term create
a proposition.
A. True
B. False
2-3. Theories are either true or false.
A. True
B. False
Chapter 2 Conceptualization: Naming
What We See in the Client Situation
2-4. What if, in the process of evaluating your client,
some concept gains local empirical support? Does this
make the concept “true”? Explain why or why not.
2-5. Explain in your own terms why propositions can be
tested in the real world and be found to be true or
false. How can we go from two or more abstract
concepts linked together (i.e., a proposition) to
generate some factual evidence with regard to the
client’s actions?
2-6. Clearly distinguish between local and general
theory, illustrated with one of your own cases, if
possible.
2-8. How are conceptual definitions related to
operational definitions, if at all?
“Caring Instructor” Intervention
Homework Due Sept 5th & 7th
• Read Chapters 3 & 4
• Answer Discussion Questions for Chpts. 3 & 4; print &
bring to next class
• Provide memorable client brief description (i.e., first
name, age, gender, presenting problem, assistance
provided by you & outcome/disposition at last contact)
• Develop C-CEP and Logic Model Elements (1-pg.
diagram/figure) for your memorable case, using Ben &
Kat case study as an example
• Draw or print memorable client description and CCEP/Logic Model elements diagram/figure and include
with Discussion Question answers for Chpts. 3 & 4
Week 3 (Sept 5th & 7th)
• Measurement issues (Chpt. 3)
– Validity
– Reliability
– Error
– Sustainability
• Graphing (Chpt. 4)
– Rules
– Interpretation
CHAPTER 3:
The Issues: Measurement Theory;
Validity; Reliability; Error
Messages; Sustainability
Validity
• Definition
– True, accurate measure of a concept
• Example: digital scale is an accurate measure of weight
• Types
– Face: appears to be accurate measure (“faith”)
– Content: comprehensive set of items/questions
– Criterion: correlation with gold standard measure or
behavior
– Construct: measure supports proposition or theory
– “Client”: achieved state = desired state
• Assessed at end of intervention and maintenance phases
Reliability
• Definition
– Consistency
• Example: digital scale also highly reliable measure of weight
• Types
– Interobserver: among two or more observers
• Agreements / (Agreements + Disagreements) = reliability
score
– Test-retest: same measurement under same
circumstances
– Alternate forms: different questions measuring same
concept
– Internal consistency: high correlation among set of
questions
Error & Sustainability
• Measurement Error
– Random: natural variability among people
– Systematic: consistent bias due to setting, time of
day/season, helping professional, etc.
• Sustainability
– Length of maintenance period “long enough” to
be sure that gains realized during intervention
period are retained post-intervention
Chapter 3 Discussion Questions
3-2. Validity means consistency in reporting what we see
in the world
A. True
B. False
3-3. Reliability means how truthful a measure is in
reflecting what is occurring in the world.
A. True
B. False
3-5. Which of the following is a true statement?
A. If a measure is reliable, it will also give valid information.
B. If a measure is valid, it will also give reliable information.
C. Neither A or B is true.
Chapter 3 Discussion Questions
3-9. We have introduced the idea of client validity,
meaning that whatever is true in a situation has to be
true for the client in that situation, and only the client can
tell us whether or not this is so…What are the limits or
risks in depending on a client’s assessment of his or her
own outcome at any given stage of practice?
3-11. Give an example from your client case, if possible,
or make up a client case, in which you illustrate random
and systematic errors.
3-12. Why is sustainability in client outcome important in
client-centered evaluation of practice?
BREAK
CHAPTER 4:
Graphing: The Basics; Exceptions to
the Rules
The Rules
• Draw a graph with horizontal and vertical axes
• Label the graph
– x-axis (time intervals); y-axis (target)
– Phases, divided by vertical lines
•
•
•
•
•
Reconstructed baseline (A) -- if retrospective data used
Baseline A
Intervention B
Maintenance M
Follow-up – as needed, at less regular intervals
• Add data points!
Exceptions
• Follow the rules unless there is a compelling
reason not to do so (e.g., missing data).
• Justify your reasoning for an exception and
document it.
• As appropriate, make an annotation on the
graph.
Case Study: Rick and his “walking
support group” (Figure 4.1)
Figures 4.2 and 4.3
Intervention Theory
Problem
Objective(s)
Shorter-term
goal(s)
Longer-term
goal(s)
Ben & Kat
1.
Financial
problems
Truancy
Increase
Ben’s
income
(Fig. 2.1 & 2.2)
1. Extend Kat’s
schooling
Reduced
stress
Improved
sex
Increased
marital wellbeing
Ricky
Increase
attendance
Improved
grades
(Fig. 4.1)
(Fig. 4.2)
Increased
self-efficacy
(Fig. 4.3)
Chapter 4 Discussion Questions
4-1. A graph is a means of representing client data
over time so as to observe the changes (positive or
negative or none at all) and to respond to these
changes with appropriate professional helping.
A. True
B. False
4-3. A target for intervention is described by naming
the concept referring to that client concern and
giving its operational definition.
A. True
B. False
Chapter 4 Discussion Questions
4-6. Describe an on-going progress of a case. At
what point do you decide:
– to begin intervention after a sufficient baseline
period?
– change the intervention, when either improvement or
deterioration appear to be occurring?
– terminate the case?
Explain each of the three decisions.
4-7. Look at Figure 4.1 (from the text; or substitute
another example). Find the annotation. Why is it
helpful (or not) to have annotations directly
written/printed on graphs?
Accessing Adobe Connect
iPhone & iPad App:
http://itunes.apple.com/us/app/adobe-connect-mobile-for-ios/id430437503?mt=8&ign-mpt=uo%3D2
Android App:
https://play.google.com/store/apps/details?id=air.com.adobe.connectpro
BlackBerry App:
https://appworld.blackberry.com/webstore/content/36989/
Homework Due Sept 12th & 14th
• Read Chapter 5 and answer Discussion
Questions
• Diagram Intervention Theory describing your
work with your memorable client
– See Ben & Kat and Ricky slide from today
• Develop 1-3 graphs summarizing your work
with your memorable client, including data as
best you can remember [postponed to Wk. 5]
– Should related to Intervention Theory diagram
and resemble Fig’s. in Chpts. 2 and 4
Week 4 (Sept 12th & 14th)
• Feedback on Homework from Wks. 2 & 3
• “Caring Instructor” Intervention Update
• Information Retrieval (Chpt. 5)
– Search Engines and Databases
– Boolean Search Strategies
• Overview of Client Evaluation Plan Assignment
[Postponed to Wk. 5]
– Sample Practice Evaluation Article/Report
– Guidelines and Rubric
Homework Feedback
(Chpts. 1-4)
• Great start! All +4/4 (for those who submitted)
• A few missing Client Descriptions and/or C-CEP/Logic
Model Diagrams … submit ASAP
• Memorable case data compiled and appended
throughout semester for reference & illustrative
purposes
– See ‘7401 memorable client case studies’ PP slides on
Carmen—Content—Other
• Next step for most: specify general theory(ies) of human
behavior or lifespan development providing
rationale/justification/support for (local) intervention
theory (recall Ricky, Ben & Kat, Caring Instructor slides)
“Caring Instructor” Intervention
• (Local) Intervention Theory
Problem
Objective(s)
Stress
Increased
social
support
• Beginning Data &
Graph
Section
Wed eve
Fri am
Wk 1 Wk 2 Wk 3
3.88 3.22 3.50
3.89 3.50 3.45
Increased
well-being
Mean values for groups (sections)
5.0
4.5
4.0
How are you 3.5
feeling now? 3.0
(5=Excellent;
2.5
1=Poor)
• Forthcoming
Start of
field
3-day
weekend
Wed eve
Fri am
2.0
1.5 A
• General Theory
Longer-term
goal(s)
Shorter-term
goal(s)
B
1.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Week
CHAPTER 5:
Information Retrieval:
Finding general evidence-based
practice information
Search Engines
• What search engines have you used in your
research and/or daily life?
• Why do you use that search engine? What are
its advantages and disadvantages?
• How would those advantages and
disadvantages apply to a search in which you
wanted to find help creating an intervention
for your client?
Source: Bloom & Britner (2012)
Academic Search Engines and
Databases
• Identify key words or search terms.
• Compare a search using a free, full-text search
engine (like Google Scholar) with one using a
subscription database (like Social Work
Abstract, PsycINFO, or MEDLINE).
• Try an appropriate (for the field or topic)
“what works?” clearinghouse site for leads on
valid (for your client) interventions.
Source: Bloom & Britner (2012)
Searching OSU Databases from Home
• Go to Library home page: library.osu.edu
• Click on “Off Campus Sign-in”; log on with OSU
username and password
• Click on “Subject Guides”, then “Social Work”
• Select database of interest…
– Academic Search Complete … journal articles
– Article Express … request journal articles not available
online
– OhioLINK … books
– Campbell Collaboration … systematic reviews
• Note: After logging on to OSU library network, able to
access many full-text publications from Google Scholar
Information Clearinghouses
• Child welfare … Child Welfare Information Gateway
– http://www.childwelfare.gov
• Youth development … HHS Administration for Children and
Families National Clearinghouse on Families and Youth
– http://ncfy.acf.hhs.gov/publications/ydfactsh.htm
• Substance abuse & mental health … SAMHSA Treatment,
Prevention & Recovery
– http://store.samhsa.gov/facet/Treatment-Prevention-Recovery
• Homelessness … U.S. Interagency Council on Homelessness
Research and Evaluation
– http://www.usich.gov/usich_resources/research_and_evaluation
• Aging … NIH National Institute on Aging A-Z Health Topics
– http://www.nia.nih.gov/health/topics
• Others?
Boolean Search Strategies
“AND” (intersection)
“OR” (union)
A … Self-concept
B … Academics
C … Practice
Source: Bloom & Britner (2012)
Fig. 5.1 Ricky case illustration
Searching the Literature
• A Pecking Order of Proof
– Systematic Reviews and meta-analyses
– Randomized controlled trial studies (true
experiments)
– Non-randomized comparison studies (quasiexperimental designs)
– Cross-sectional studies with retrospective
measurements (surveys)
– Case studies
Source: Bean (2012)
Caring Instructor Intervention Lit Review
• General theory article on social support
– Cohen, S., & Wills, T.A. (1985). Stress, social support, and the
buffering hypothesis. Psychological Bulletin, 98(2):310-357. Cited
by 6,571
• Examines whether the positive association between social support and
well-being is attributable more to an overall beneficial effect of support
(main- or direct-effect model) or to a process of support protecting
persons from potentially adverse effects of stressful events (buffering
model)…It is concluded that there is evidence consistent with both
models…Implications for theories of social support processes and for the
design of preventive interventions are discussed.
Caring Instructor Intervention Lit Review
• Local intervention theory article: social support among
graduate students
– Goplerud, E.N. (1980). Social support and stress during the first year of
graduate school. Professional Psychology, 11(2):283-290. Cited by 64
• Describes the effects of varying levels of social interaction on 22 1st-yr
graduate students' reports of stressful events and on their health and
emotional problems during the 1st 6 mo of graduate study…An inverse
relationship was found between frequency of social interactions with peers
and faculty during the 1st 10 wks of study and the incidence of stressful life
events and the number of reported physical and psychological disturbances
throughout the next 6 mo. The quality of faculty–student interactions also
emerged as an important factor that moderated the negative consequences
of the major life changes associated with beginning graduate school.
Conclusion
Social support appears to be a crucial variable that moderates the negative consequences of
the unavoidable life changes that occur during students' first year of graduate work. Frequency
of social interactions with other students and with faculty was found to be inversely related to
the intensity and duration of life disruptions and also to susceptibility to psychological and
physical disturbances. Developing faculty awareness of their critical influence on graduate
students' health and emotional well-being and helping new students expand their socially
supportive contacts appear to be important primary and secondary prevention strategies to
reduce graduate students' risk for stress-related problems.
(Local) Intervention Theory Diagram
for Caring Instructor Intervention
Increased
[faculty] social
support
Decreased
stress
Increased
well-being
Key intervention elements
1.
2.
3.
4.
5.
Express interest
Validate
feelings
Share thoughts,
suggestions
Encourage
contact with
field liaison
Improve
teaching, class
as-needed
1-5 rating of
overall wellbeing
Chapter 5 Discussion Questions
• 5-2. A search question puts together all of the major
concepts under consideration as entry points into
information space, recognizing that depending on
the number of hits resulting from this very specific
question, we may have to remove one or more terms
until we reach some useful answers.
A. True
B. False
• 5-3. A key term is to information retrieval as
concepts are to theory.
A. True
B. False
Chapter 5 Discussion Questions
• 5-5. What is the importance of a chapter on
information retrieval in this book on evaluating client
outcomes?
• 5-6. Let’s say we wanted information on depression,
children under the age of 10, and African Americans.
How would we create a Venn Diagram that included
just these ideas and excluded others? How would
you write that search, in Boolean terms?
• 5-8. What if you locate information through your
retrieval searches that indicate that the therapy or
intervention you are using has been shown to be
relatively ineffective with clients very much like your
own? How does this information influence your
practice methods?
BREAK
Demonstration of
Searching Literature to
Examine Level of Support for
Local Intervention Theory
Homework Due Sept 19th & 21st
• Read Chapter 6 and answer Discussion Questions
• Develop 1-3 graphs summarizing your work with your
memorable client, including data as best you can
remember
– Should related to Intervention Theory diagram and
resemble Fig’s. in Chpts. 2 and 4
• Search the literature to find 1-3 scholarly
articles/publications providing strongest level of
support for your local intervention theory for your
memorable client (ie, diagram included with Chpt. 5
Discussion Question answers)
– Include full citation(s) and abstracts only – NOT FULL TEXT
OF ARTICLES/PUBLICATIONS – next week with Chpt. 6
Discussion Question answers
Week 5 (Sept. 19th & 21st)
• Examine practice evaluation article
– Sample final report for capstone project due end of spring
semester
• Review rubric for Evaluation Plan paper due Nov. 16th (8
wks. from now)
– Background & Methods sections of capstone final report
– Results & Discussion sections added during spring semester
• Break
• Data collection options
– Individualized rating scales & structured logs (Chpt. 6)
– Observation (Chpt. 7)
– Standardized rating scales (Chpt. 8)
Practice Evaluation Article
• Carmen—Content—Other
– “Sample clinical eval report”
– Title: On the use of progressive relaxation in the
treatment of bulimia: A single-subject design study
• Background section
– Paragraph 1: Description of "negative affect" model
(i.e., theory of change explaining binge/purge
aspect of bulimia)
– P2: Description of “stress-reduction function” (i.e.,
intervention theory)
Practice Evaluation Article
• Background section (con’t)
– P3,4: Goal of study/project: examine effectiveness
of progressive relaxation (i.e., advanced practice
method)
• Methods section
– P1: Patient (client) description
– P2: Treatment (description of baseline phase and
data collection methods)
– P3: Description of intervention phases and
clarification of key outcome measures (e.g., # of
binge urges and binge episodes)
Practice Evaluation Article
Rubric for Evaluation Plan Paper
• Carmen—Content—Other
– “Rubric for eval plan paper”
• Highlights
– Timeline
• Memorable client & text readings … Wks 1-10
• Develop of eval plan for current clients … Wks 11-13
– Draft eval plan paper due Fri., Nov. 16th
• Receive feedback on draft from field instructor & myself
(over Winter/Christmas break)
• Revise plan as-needed, based on feedback, and obtain final
approval to begin collecting data by Jan. 28th
• Complete data collection by Apr. 5th (10 wks.)
• Submit final eval report by Apr. 19th & present findings next
week in class (2 wks. before Commencement)
Rubric for Evaluation Plan Paper
• Highlights (con’t)
– Sections and sub-sections
• Background section
–
–
–
–
Practice setting
Problem statement
Intervention
Theory of change
• Methods section
–
–
–
–
–
Clients
Measures
Design
Data collection
Analysis
– Format (e.g., body 10 pg. max., single-spaced)
– Checklist
BREAK
Data Collection Options
• Individualized rating scales & structured logs
(Chpt. 6)
• Observation (Chpt. 7)
• Standardized rating scales (Chpt. 8)
--- End of material covered on Midterm Quiz --• Midterm Quiz
– Chpts. 1-8 of text
– Take-home, completed on Carmen, opens Sept. 29th
at 10 AM and closes Oct. 2nd at 11:59 PM (4 days to
complete)
– T/F, MC questions mentioned in lecture
– Essay questions mentioned in lecture, applied to
memorable client
Data Collection Options (Con’t)
• Individualized rating scales & structured logs
(Chpt. 6)
– Co-constructed with client
– Single item, Likert 1-5 (or so) responses
– Unknown validity (beyond face) and reliability
– Most useful (in my view)
• Observation (Chpt. 7)
– Useful when able to observe client in setting
where and when problem presents (seldom)
Data Collection Options (Con’t)
• Standardized rating scales (Chpt. 8)
– Existing, multiple item measure with known
psychometric properties and available norms against
which client’s scores may be compared
– Administered and scored by helping professionals
(sans co-construction with client)
– Known validity and reliability, thus most rigorous and
scientific option
– However, of limited use post-intervention
(maintenance & follow-up periods) given unlikely selfadministration and scoring by client
• Homework due Sept 26th & 28th: Discussion
questions from chpt’s. 7 & 8
Week 6 (Sept. 26th & 28th)
• Individual consultations
– Return homework from chpts. 5 & 6
– Feedback on memorable case-related assignments
•
•
•
•
(Local) intervention theory figure -- – see Caring Instructor slides from this wk
Graph(s)
Lit review article citation(s) and abstract(s)
Revisions to Case Description and C-CEP/Logic Model Elements figure (asneeded)
– Wed eve
• In-person: Alisha (4:50); Aleta (5:00); Marissa (5:10); Eva (5:20); Chelsea (5:30);
Muriel (5:40); Krystle (5:50); Nicole (6:00); Jane (6:10); Brittany (6:20)
• Phone (Alvin @ 740-804-6275): Jon (6:50); Kim (7:00); Shannon (7:10); Julie (7:20);
Emily (7:30)
– Fri am
• 40-minute general Q&A/discussion on memorable clients (9:10-9:50)
• Break (9:50)
• Individual consultations (10:00-11:55)
Week 6 (Sept. 26th & 28th)
• Collect this week’s homework (chpts. 7 & 8)
• Homework due Oct 3rd & 5th
– Mid-term Quiz (Carmen—Quizzes; between 9/29 @
10a and 10/2 @ 11:59)
– Discussion Questions for Chpts. 9 & 10
Caring Instructor Intervention Lit Review
• General theory article on social support
– Cohen, S., & Wills, T.A. (1985). Stress, social support, and the
buffering hypothesis. Psychological Bulletin, 98(2):310-357. Cited
by 6,571
• Examines whether the positive association between social support and
well-being is attributable more to an overall beneficial effect of support
(main- or direct-effect model) or to a process of support protecting
persons from potentially adverse effects of stressful events (buffering
model)…It is concluded that there is evidence consistent with both
models…Implications for theories of social support processes and for the
design of preventive interventions are discussed.
Caring Instructor Intervention Lit Review
• Local intervention theory article: social support among
graduate students
– Goplerud, E.N. (1980). Social support and stress during the first year of
graduate school. Professional Psychology, 11(2):283-290. Cited by 64
• Describes the effects of varying levels of social interaction on 22 1st-yr
graduate students' reports of stressful events and on their health and
emotional problems during the 1st 6 mo of graduate study…An inverse
relationship was found between frequency of social interactions with peers
and faculty during the 1st 10 wks of study and the incidence of stressful life
events and the number of reported physical and psychological disturbances
throughout the next 6 mo. The quality of faculty–student interactions also
emerged as an important factor that moderated the negative consequences
of the major life changes associated with beginning graduate school.
Conclusion
Social support appears to be a crucial variable that moderates the negative consequences of
the unavoidable life changes that occur during students' first year of graduate work. Frequency
of social interactions with other students and with faculty was found to be inversely related to
the intensity and duration of life disruptions and also to susceptibility to psychological and
physical disturbances. Developing faculty awareness of their critical influence on graduate
students' health and emotional well-being and helping new students expand their socially
supportive contacts appear to be important primary and secondary prevention strategies to
reduce graduate students' risk for stress-related problems.
(Local) Intervention Theory Diagram
for Caring Instructor Intervention
Increased
[faculty] social
support
Decreased
stress
Increased
well-being
Key intervention elements
1.
2.
3.
4.
5.
Express interest
Validate
feelings
Share thoughts,
suggestions
Encourage
contact with
field liaison
Improve
teaching, class
as-needed
1-5 rating of
overall wellbeing
Week 7 (Oct. 3rd & 5th)
• Return Chpts. 7 & 8 homework; discuss quiz
• Case management intervention elements
• Chpt. 9: Qualitative Data: Self-Monitoring
• Chpt. 10: Measurement Cautions
--- (end Part II. Measurement of book) ----- (begin final Part III. Evaluation designs) ----- (begin transition memorable to current clients) --Upcoming…
• Next wk: Chpt. 11: Baselining
• Week after next: Chpts. 12 & 13: Basic & Advanced
Evaluation Designs
Mid-term Quiz Results & Feedback
• Scores …range: 15-20; mean: 18.0 (both sections)
• Overall, fine demonstration of ability to apply key
practice evaluation concepts discussed in class to
memorable client
• Essay questions subjective, with multiple reasonable
applications of concepts to practice, yet some responses
incorrect or incomplete
• Reminder to…
– Use complete sentences when answering final quiz
questions
– Review slides pertaining to concept being asked about (e.g.,
pros & cons of individualized vs. standardized rating scales)
• Questions? Comments? Suggestions?
Case Management: General Practice
Source
Rothman, J. (1991). A model
of case management:
Toward empirically based
practice. Social Work
36(6):520-528.
Case Management: Inpatient Health
Source
Taylor (1999). Comprehensive
nursing case management: An
advanced practice model.
Nursing Case Management
4(1):2-10
Case Management: Mental Health
Full Service attempted to provide all the clinical and support services needed by
the client through the in vivo efforts of a specially trained interdisciplinary team.
Brokering of services was minimized. Training in Community Living, Assertive
Community Treatment, Assertive Outreach and The Bridge program (Test, 1992; Bond,
McGrew & Fekete, 1995) are examples.
Broker provided very little direct service to patients. Rather, services needed by the
patient were arranged from among those available in the community mental health
system. These programs included Broker, Specialist, Generalist and Supportive (Rubin,
1992; Solomon, 1992; Chamberlain & Rapp, 1991).
Hybrid provided some services while other services were brokered. The distinction
between full service and hybrid was a matter of degree. For example, programs
characterized as PACT provided virtually all services, and were clearly full service.
Other programs such as Intensive Case Management and Strengths expected staff to
provide most services but would also broker some services. For the present review,
Intensive Case Management, Expanded Broker, Family Case Management, Personal
Strengths, Rehabilitation, and Clinical Case Management were considered hybrid
models (Rubin, 1992; Solomon, 1992; Chamberlain & Rapp, 1991).
Source
Bedell et al. (2000). Case management: The current best practices and the next
generation of innovation. Community Mental Health Journal 36(2):179-194.
CHAPTER 9:
Qualitative Data in Single-System
Designs: Self-Monitoring
Self-Monitoring (SM)
• Definition: client involved in observing &
measuring some aspects about him/herself
– Applicable when using structured logs and
individualized rating scales (Chpt. 6)
– Less/not applicable when using observation (Chpt. 7) or
standardized rating scales (Chpt. 8)
• Maximal use of SM – and ideal for C-CEP – client…
–
–
–
–
–
Defines goals of service
Participates in constructing targets of intervention
Involved in data collection
Participates in interpretation of graphs
Informs helping professional when results have attained
his/her goal(s) (“client” validity)
Self-Monitoring (con’t)
• Qualitative form (Fig. 9.1, p.85) capturing data
gathered using/following:
–
–
–
–
Narrative approach
Positive psychology / Strengths Perspective
Global assessment
Useful with memorable and/or current clients?
• “Clients know themselves far better than the
practitioner ever can. The role of the practitioner
is essentially to bring out other positive
alternatives for the client to use in resolving the
presenting problem and move ahead in life goals.
(p.83)” Agree or disagree?
Fig. 9.1: Illustrative (Qual) SM Data Form
Asset/strength #1
Asset/strength #2
Global:
Overall situation
CHAPTER 10:
Measurement Cautions
Issues
• Group practice/intervention measures
– Observed by practitioner
• Participation >> attendance - # and/or %
• Engagement >> 1-5 rating by practitioner for each
participant, averaged to compute single score for group
– Self-monitored/reported by clients
• Perceived satisfaction, helpfulness or other domain >>
mean 1-5 rating among participants using individualized
rating scale
• “Whenever we ask questions (or a person
knows they are being observed), there is always
some effect on the client.”
Reactivity, Ethics & Homework
• Reactivity and obtrusiveness
– “Whenever we ask questions (or a person knows they
are being observed), there is always some effect on the
client.”
• Clinical vs. research vs. practice evaluation pros & cons?
– Minimize using archival records and observation
• Ethics
–
–
–
–
–
Clients define own goals
Clients understand & agree on interventions
Clients involved in choosing measures & eval designs
Clients determine if changes reflect real goals
Clients determine whether changes are sustainable
• Hmwk: Chpts. 9 & 10 due today; Chpt. 11 due next wk.
Week 8 (Oct. 10th & 12th)
• Return assign’s; review updated Carmen—
Grades
--- (begin final Part III. Evaluation designs) ----- (begin transition memorable to current clients)
• Chpt. 11: Baselining
• Briefing field instructor(s) on capstone project
• Next week
– DUE: Chpts. 12 & 13 Discussion Questions
– Peer evaluation of teaching (Dr. Balaswamy via
Adobe Connect)
Grades Updated in Carmen
• Attendance
– Wks 1-7 (through last week); out of 7 pts.
• Discussion
– Chpts. 1-10 (received in class and via Carmen Dropbox, as
of 10/9/Tues. @ 2p); out of 10 pts.
– Zero pts. “placeholder” for chpts. not yet recv’d.
• Midterm quiz
– Chpts. 1-8; out of 20 pts.
– Note mean 18.0 overall (17.9 Wed.; 18.0 Fri)
• Thus, as of last Fri. (10/5): 7 + 10 + 20 = 37 possible pts.
(or 37% of grade) completed
• Remaining: 8 attendance + 5 discussion + 20 final quiz +
30 evaluation proposal paper = 63 pts.
CHAPTER 11:
Baselining and the Beginning of
Evaluated Practice
Chpt. 11: Baselining
• Critical assessment, presenting problem clarifying,
starting point in C-CEP
• Case example: 7 y/o Clare victim of “bullying” at
school
– Local theory, baseline assessment questions
• What forms does bullying take, who’s doing it, and to whom
is it directed?
• What are teachers & staff doing about it?
• Where in school does the bullying occur?
• Does it also occur beyond the school day?
• When does it occur?
• How often does bullying occur?
• Are others present when the bullying occurs?
Chpt. 11: Baselining
• “This is what the problem looked like when I
entered the case, and this how it looked at
several points during the intervention (or at
the end of the intervention) (p.98).”
– Key problem targets (i.e., y-axis measures) in
Clare’s case
• Graph 1, solid line: Extortion of lunch money (# times
occurrence per school day; 0=no, 1=yes)
• Graph 1, dashed line: Social exclusion instances (# times
per school day)
• Graph 2: Clare’s feelings of oppression (1-5 IRS; 1=very
good … 3=mixed … 5=very sad)
Chpt. 11: Baselining
• Application to first client in field this year…
– 11-9. Baselining is conducted to find clear patterns
of the presenting concerns and the contexts in
which they take place. How can we be sure we are
collecting: a) the right kind of information; b) in
the right amount; and, c) in the right way?
Illustrate these points with a client situation.
• How many graphs needed for your client?
• Data collection method(s) required?
– Observation, IRS and/or SRS
• How frequently, by whom, and by what means (e.g.,
logs) problem data will be collected?
Chpt. 11: Baselining
• Ideally “concurrent” baselining used
– Both (A) reconstructed, in conjunction with, A
current baseline (except when emergency
intervention required, then (A) -> B, without A)
– Purpose to establish an accurate picture of
presenting problem(s) pattern(s) before
intervention
– Rough rule of thumb: 5-7 time (x-axis) units
– Possibly more…however many time points needed
to feel confident in accurately depicting problem
pattern graphically during your initial baseline
problem assessment stage of working with client
Chpt. 11: Baselining
• Application to first client in field this year…
– 11-7. Why would we ever conduct reconstructed
baselines, when we are beginning work with a client
seated in front of us?
– 11-8. How long should you collect baseline
information?
– 11-10. What patterns are you looking for when you
collect baseline data? That is, what changes in these
data are of main concern to the helping professional?
– Openness to begin baselining with your current and/or
new clients starting now?
• Possibly helpful in practice…definitely helpful in writing
evaluation proposal (due 11/16)
Chpt. 11: Baselining
11-1. Which of the following is true of baselines?
A. Intervention may take place during baselining, as needed and depending on
the client’s readiness.
B. Systematic measures and/or observations are used on a regular and
repeated basis throughout the baseline phase and whenever you return of a
non-intervention phase.
C. Baselining criteria should be selected by the helping professional.
11-2. If you had good archival data on a client’s concern, you could use the
archival information as a kind of reconstructed baseline.
A. True
B. False
11-3. If collecting baseline information from the client is having a negative
impact on him or her, you should stop collecting baseline data and move into
intervention, using the best available information and making suitable
corrections as you go along.
A. True
B. False
Chpt. 11: Baselining
11-4. Baselining is an absolute essential in single-system
evaluation, and must be performed at the beginning of
every case.
A. True
B. False
11-5. Clients have to be directly involved in collecting
baseline data.
A. True
B. False
11-6. We should always share baseline data with clients so
that they can see the scope of their concerns and what is
to be changed.
A. True
B. False
BREAK
Field Instructor Briefing & Next Week
• Email and/or print capstone project briefing
packet and discuss with field instructor during
next supervision meeting
– Carmen – Content – Other – “Field instructor capstone
packet”
– Cover letter; 4 overview slides; proposal rubric;
syllabus; and, sample clinical practice evaluation
report (progressive relaxation and bulimia article)
• Emphasize NOT research, rather clinical practice
evaluation/field education project
• Next week: Chpts. 12 & 13 (Basic & Advanced
Evaluation Designs); peer eval teaching (Wed eve)
Week 9 (Oct. 17th & 19th)
•
•
•
•
Preview of next semester (SWK 7402)
Field instructor feedback on capstone project?
Evaluation designs (Chpts. 12 & 13)
Coming up…
Preview of Next Semester
• Common elements across capstone sections, tracks
– 9 meetings in class throughout term
• 3 at beginning; 3 near middle; and 3 at end
– Grading elements
• Attendance (10-15%); Assignment(s) (10-15%); Presentation (1015%); and Final Report (60%)
• Student presentations (using PowerPoint) most meetings, based on
assignments due that week of class
• Final report
– Background & Methods sections revised from evaluation proposal, possibly
throughout term if/as-needed
– Results & Discussion sections (including dissemination or decision-making
component)
– Comparable rubrics
– Final report due Apr. 8th
• Unique elements across sections, tracks
– Topics & assignments for each meeting dates
Mares Sections Schedule (Tentative)
Wk (Date)
Location
Topic & Assignment
1 (Jan. 8th)
Class
Overview of online revision & resubmission, data
collection monitoring system
2 (Jan. 15th)
Class
Revised proposals (RP) DUE [10 pts.]; RP presentations
3 (Jan. 22nd)
Class
RP presentations (con’t)
4 (Jan. 29th)
---
Begin data collection (most students)
5 & 6 (Feb. 5th & 12th)
---
Continued collection of baseline & early intervention data
7 & 8 (Feb. 19th & 26th)
Class
Baselining and design presentations
9 & 10 (Mar. 5th & 12th)
---
11 (Mar. 19th)
Class
12 (Mar. 26th)
---
13 (Apr. 2nd)
Class
End data collection (most students); Data Analysis (DA)
presentations
14 (Apr. 9th)
Class
Final report (FR) DUE (Apr. 8th) [60 pts.]; FR presentations
15 (Apr. 16th)
Class
Reflections on continued C-CEP; feedback on capstone
project, 7401/02
Draft Results & Discussion (R&D) sections DUE; R&D
presentations
Plus: Attendance (18 pts.; 9 classes @ 2 pts.); Presentation (12 pts.) = 30 pts.
Field Instructor Feedback on
Capstone Project?
Evaluation Designs
Basic Designs: client outcomes only; no causal
inference re/ effectiveness of intervention
1. “B Only” (my own term; variation of authors’
BABM Emergency Design from Chpt. 13)
2. AB*M** (Chpt. 12)
3. Multiple Baselines (from Chpt. 13)
Advanced Designs: both client outcomes &
causal inference
4. ABAM
5. ABABM
6. Changing Intensity
B-Only Design
• More realistic (in my view) version of authors’ “BABM
Emergency Design” from Chpt. 13)
• No practical opportunity for baseline or maintenance
period due to client risk and brief nature of intervention
• Applicable with “single-session” therapy and limited-time
crisis intervention work
– Healthcare: emergency departments; medical/surgical inpatient
units & outpatient clinics
– Child welfare: intake & investigation units
– Poverty and homelessness: emergency food, clothing & shelter
programs; public assistance & benefits application
– Education & training: single-session classes, seminars, groups
• Compare group mean (y-axis) to threshold for each day of
practice (x-axis), rather than comparing individual client or
group cohort change from A (baseline) to B (intervention)
periods…and ideally M (maintenance) period
AB*M** Design
• Builds on AB design.
• Adds maintenance phase M, in which practitioner
teaches the client to take full control over a successful
intervention B change since baseline A in the target.
• Asks client about goal attainment (indicated by *) after
B and again (indicated by **) after M.
• This is the fundamental design for client-centered
evaluation of practice, new because:
– The client is directly asked about how well her or his goals
are being achieved, to complement any objective
measurement.
– The clients takes charge of the measurement and the
maintenance of the intervention.
AB*M** Example:
Case Study (Jesse, runner; Figure 12.1)
AB*M** Steps
1. Identify the client and the client’s goals.
2. Break down goals into specific objectives and
measurable targets. Plot the data on a graph
through baseline A and intervention B phases.
3. When the helping professional and client agree
that positive change has occurred, go to a
maintenance M phase, led by the client.
4. Collect M phase data to see if positive results
persist. If not, reconsider B or training.
AB*M** Questions
• Definition
– 12-1. An evaluation design is a formal plan in which
something is done (the intervention) to something
else (the target) with the objective-to-be tested that
some change will occur approximating what the client
is seeking through the service.
A. True
B. False
• Key elements
• 12-3. The meaning of the symbols “A-B design” is that
an Action or intervention is introduced at A, and is
then compared to the Baseline indicated by B.
A. True
B. False
AB*M** Questions (con’t)
• Objective vs. subjective (client-validity) outcomes
– 12-6. What does “objective evaluation of services”
mean to you (in your own words)?
– 12-8. What do you see as the main advantage of using
the * in the basic AB*M** design? And what is the
distinctive advantage of the ** in the basic AB*M**
design?
– Why don’t we just accept what the client says about
his or her goal attainment at the end of the B phase,
and terminate the case?
– Why do we accept what the client says when he or
she is in charge of the intervention in the M phase?
Multiple Baselines (Figure 13.15)
Multiple Baselines Design
• Authors’ description
– “Ideal” Advanced Design for practitioners reluctant to
withdrawal a successful intervention from clients
– 3 scenarios
• 2 persons with same problem in same setting
• 2 different problems with same person in same setting
• 2 different settings with same person and same problem
• My view
– Falls under Basic Designs due to lack of causality (i.e.,
no “quasi-experimental” withdrawal of intervention)
– 4th scenario: 2 problems with differing amounts of
baseline data available (most likely reconstructed from
existing clinical records or other sources)
BREAK
Advanced Evaluation Designs
• Who’s Happy?
– Client and Practitioner
• If the client target behavior changes
• If the change is maintained by the client
– Researcher
• If we can show that the intervention caused the
change.
– Can’t everyone be happy?
• Yes, with some advanced designs:
– ABA and ABAM
– ABAB and ABABM
– Multiple Baseline Designs (and those with M)
Advanced Evaluation Designs
• Some evaluation designs remove an
intervention to see whether the target returns
to a prior (undesired) state, as part of the
process of investigating causality.
• To distinguish pre-intervention baselines from
those times when intervention is intentionally
removed, we will call the latter Observationonly Phases. The exact same data collection
methods are being applied to the same target.
Advanced Evaluation Designs
• Adding an M phase and * and **
– Conventional designs, such as the AB, the ABA,
the ABAB, and the multiple baseline design, all
may likewise benefit from a maintenance M phase
in which the client interprets results from his or
her point of view (that is, * and **).
• On some occasions, the changing intensity
designs and alternating intervention designs
may likewise benefit.
ABAM Design
ABABM Design
Advanced Designs Question
• 13-9. When you compare results from an
ABAB design, an ABAM design, and an ABABM
design, you may see quite different results.
Explain what differences might occur, and why
one of these designs -- identify it -- is the
strongest among the three patterns.
Changing Intensity Design
Changing Intensity Design
• Authors’ case example
– Developmentally delayed woman, Frida, in sheltered
workshop employment program
– Behavioral intervention: increasing periods of work (10
min’s -> 15 min’s -> 25 min’s) rewarded with 1 min break
talking with you (practitioner)
– Aside: chronic disease/problem (i.e., developmental
disability) reduces M (and F) period outcomes
expectations
• Applies also to other evaluation designs (basic & advanced)
– Nearly all practice may be considered changing intensity,
given that most work winds down (M period) and ends (F
period) at some point, leaving client to manage on his/her
own
• Ethical practice within constraints of limited time and resources
Advanced Design Question
• 13-7. What do you see as the ethical issues in
using advanced [“quasi-experimental”]
designs?
Consider how much stress one can impose on clients
by making changes in the situation vs. our interest in
scientifically evaluating the effectiveness of our
practice.
Why do we not just use the simplest scientific
design, the AB*M** design, that provides some
evidence of change due to the intervention?
Advanced Design Question
• 13-8. Alternative explanations are other possible
reasons why the results came out as they did. The
intervention is only one possible explanation for any
change in the target. We argue that advanced designs
reduce the impact of these alternative explanations by
controlling the logic of the situation.
Illustrate this with reference to either your memorable client
or your first client, where you describe the…
(i) Situation/presenting problem(s),
(ii) Intervention, and
(iii) Most appropriate advanced evaluation design to explain
why chance alone would be a less likely explanation than
the intervention
Coming up…
• Next week (Oct. 24th & 26th)
– DUE: Chpts. 14 & 15 Discussion Questions … last ones!!
– Final Quiz on Carmen
• Chpts. 11-14 (excluding Chpts. 9, 10 & 15)
– Topics: Baselining; Evaluation Designs; and, Data Analysis
• Opens 10/27/Sat @ 10a … closes 10/30/Tues @ 11:59p
• Week after next (Oct. 31st & Nov. 1st) >> NO CLASS
– DUE: Draft Background section of Eval Proposal to Carmen
Dropbox by 11/3/Sat @ noon
• See 7401 Practice Evaluation Proposal Rubric on Carmen
• Practice setting; problem statement; intervention; theory of
change
• Draft Figures 1 & 2 (C-CEP/Logic Model & Local Intervention
Theory)
Week 10 (Oct. 24th & 26th)
• Return homework; Carmen—Grades updated
• Analysis of Data (Chpt. 14)
• Decision Making (Chpt. 15) >> to be discussed
mid-Spring semester (ie, Wk 11/Mar. 20th; 8th
wk of 10-wk practice eval capstone project data
collection period)
• Coming up…
CHAPTER 14:
Analysis of Data: A Systematic and
Holistic Approach
Chpt. 14: Data Analysis Methods
Four methods of analysis (all valid, all limited)
– Some or all should all be performed to obtain a systemic or
holistic perspective on the situation
1. Trend analysis of “objective data” (Methods 1 & 3 in
chapter)
2. Goal attainment for “subjective” client validity data
(Method 2) (ie, * and ** in AB*M** design)
3. Non-overlapping data: baseline vs. intervention
periods (Method 4)
4. Sustained time analysis: Maintenance period
(Method 6)
5. Statistical analysis (Method 5): not discussed; beyond
scope of book/class
Trend Analysis of Objective Data
Trend Analysis of Objective Data
Goal Attainment for Subjective Data
• Global assessment (from Fig. 9.1)
•
Looking at your whole situation (problems, strengths,
resources), how would you judge the changes in the “complex
whole” since…
–
–
•
Objective rating (1-5 Likert scale)
–
–
–
–
–
•
…we began working together? (* or change from A -> B), AND
…you began doing what we discussed on your own (** or change
from B -> M)
1=Great Deterioration
2=Some Deterioration
3=About the Same as Before
4=Some Improvement
5=Great Improvement
Subjective, narrative explanation for objective rating (“Why?”)
• Domain-specific assessment (OPTIONAL)
Non-Overlapping Data: A vs. B Periods
Effect Size, or
“Improvement Rate
Difference (IRD)”
Chapter 14 Discussion Question
14-2. Effect size is one way to put into numbers
the extent of improvement that has occurred, if
any, between one phase and another phase.
A. True
B. False
Sustained Time Analysis: B vs. M Periods
• “Sustained time of successful (self-help)
intervention” = M period
• Recommended length of M period (ideal for
use in post-MSW practice … vs. possibly
contrived 10-wk capstone project data
collection period)
•
•
•
Low-severity target … equal to B period (most common)
Medium-severity target … twice as long as B period
High-severity target … three times longer than B period
Chapter 14 Discussion Question
14-1. Which of the following are correct with
regard to a desirable intervention pattern?
A. There should be real improvement in
intervention, compared to the baseline phase.
B. The client will feel like the intervention should
end.
C. Once the desired level of outcome has been
attained, the helping professional can begin
termination.
D. The desired condition should be sustained for an
indefinite period of time before termination of the
case begins.
Chapter 14-Related Question
What is the (a) primary strength and (b)
limitation for each of the data analysis methods
discussed?
Coming up…
• This weekend: Final Quiz (Chpts. 11-14): 10/27/Sat @
10a – 10/30/Tues @ 11:59p
• Wk 11 (Oct. 31st & Nov. 2nd): NO CLASS; Draft
Background section of paper to Carmen Dropbox by
11/3/Sat @ noon
• Wk 12 (Nov. 7th & 9th): Draft Methods section of paper
to Carmen Dropbox by 11/10/Sat @ noon; review final
quiz results & final paper rubric
• Wks 13 & 14 (Nov. 14th/16th & 21st/23rd): NO CLASS;
Individ consults as-needed (in-person, by phone and/or
by Adobe Connect)
– Final paper deadline extended from 11/16/Fri to
11/18/Sun at 11:59p
• Wk 15 (Nov. 28th & 30th): Individ meetings during class
to discuss final paper score and comments/feedback
Week 12 (Nov. 7th & 9th)
• Final paper rubric
• Caring instructor intervention review
• Final quiz results
----• Upcoming
– NO CLASS MEETING next 2 wks (during submission &
grading of complete final paper/evaluation proposal)
– Wk. 15 (Nov. 28th & 30th)
• 15-minute consultations re/ feedback on paper (for revision
after break) and final grade for 7401
• Feedback on course
• Last date to submit any assignment: Nov. 25th by 11:59 PM
Final Paper
• Rubric: Two sections; nine sub-sections
• Checklist: p.4 of Rubric
• Submission: Deadline extended to 11/18,
11:59 PM
• Revision: First 2 weeks of spring semester,
responding to written comments/feedback
received from me and field instructor; 10% of
grade in 7402
• Feedback: Provided by me during last class
Final Paper
• Fig. 1 C-CEP/Logic Model Example: Slide #32
Final Paper
• Fig. 2 (Local) Intervention Theory Ex: Slide #70
Caring Instructor Outcomes
General Well-being
Excellent
5.0
4.5
Good
4.0
3.9
3.3
3.5
Average
3.5
3.4
3.5
3
4
5
3.6
3.8
3.5
3.4
9
10
3.2
3.0
2.5
Fair
2.0
1.5
Poor
1.0
1
2
6
Week
7
8
Caring Instructor Review
• Having concluded the (lone vs. initial?) B (intervention)
phase, how helpful in life overall?
–
–
–
–
–
5=Very helpful
4=Somewhat helpful
3=Mixed/unsure
2=Minimally helpful
1=Not at all helpful
• Example of the initial asterisk (*) following B phase
• Shared decision-making; co-construction of next step…
– End intervention >> B-only design
– Continue, resuming after break >> custom B*MB design
• No change
• Change (Specify…)
Final Quiz Results
• Mean scores
– Wed. eve.: 18.2
– Fri. a.m.: 18.4
– Overall: 18.3
• Range (same for both sections): 15-20
• Discussion of specific quiz items
Wk. 15 (Nov. 28th & 30th) Consultations
• 15 minutes per student to…
– Deliver written comments on paper
– Discuss assignment scores & final grade in 7401
• 15 minute phone calls or in-persons meetings in
my office (Wed. eve. section on 11/28 & 11/30)
• 15 minutes x 10 students (Fri. a.m.) = 2.5 hrs.
(9:10-11:40 AM)
– In-class
– Random order
– Bring homework to work on while waiting