CiMH Palette of Measures
Evaluation Training: Skill Streaming
Checklists
Cricket Mitchell, PhD
CIMH Evaluation Consultant
Palette of Measures Evaluation:
What You Will Need (slide 1 of 2)
• Palette of Measures Data Entry Shell v2 (Excel file)
– Developed by CiMH and customized for each
participating agency
– Holds all data for clients served thru the Palette of
Measures project
– Demographics, service delivery information, pre- and
post- outcome measure data
• Palette of Measures Data Dictionary v2 (Word
document)
– A guide for using the associated data entry shell
– Defines each column in the excel file
2
Palette of Measures Evaluation:
What You Will Need (slide 2 of 2)
• Outcome measures from the two-pronged
approach
– General Outcome Measure
– Target-Specific Outcome Measure(s)
– For example…
• Skill Streaming Checklists
– Part of the Skillstreaming curriculum published by
Research Press
• Must be purchased by interested Palette of Measures
project participating agencies
3
Overview of Training
• Brief Overview of Palette of Measures evaluation
protocol
• Skill Streaming Checklists: Target-specific
measure for disruptive behavior
– Administration
– Scoring
– Clinical Utility
• Instructions for Palette of Measures data entry and
data submissions
– Data entry: Skill Streaming Checklists
– Data entry: Demographics & Services
– Data submissions to CiMH
4
Brief Overview of Palette of
Measures Evaluation Protocol
5
Outcome Assessment
• Palette of Measures providers will track outcomes
using data from pre- and post- administrations of
standardized measures of functioning
• Pre- and Post- a “dose” of treatment / an
intervention interval
– General measure of youth mental health functioning
(e.g., YOQ/YOQ-SR, CANS, Ohio Scales)
– Target-specific measure linked to focus of
treatment/intervention (e.g., AQ, ECBI, PHQ-9,
PTSD-RI, RCADS)
• Providers may choose to administer mid-course
assessments as well
– e.g., at 3-month intervals in usual care
6
A note about the use of standardized
assessment measures… (slide 1 of 2)
• Assessment is the beginning of developing
a relationship with the child and family
– Demonstrates a desire to know what the child
and family are experiencing
– By incorporating standardized assessment
measures of functioning, the efficiency and
thoroughness of assessment is enhanced
7
A note about the use of standardized
assessment measures… (slide 2 of 2)
• Using standardized assessment measures of
functioning…
– Assists in initial clinical impressions
– Provides valuable information to guide
treatment/interventions
– Assesses sufficiency of treatment delivered
– Demonstrates treatment-related improvements in
child functioning
*Although the Skill Streaming Checklists are not standardized,
they are structured assessments that provide information
specific to treatment planning and outcome assessment.
8
Skill Streaming Checklists
9
CiMH Skill Streaming Checklist
Training
• Information on the administration, scoring,
and clinical utility of the Skill Streaming
Checklists was obtained from the Aggression
Replacement Training™ book, written by
Arnold P. Goldstein, Barry Glick and John C.
Gibbs, and published by Research Press in
1998; and, from discussions with Education &
Treatment Alternatives, Inc.’s Master Trainers
in Aggression Replacement Training™
10
Skill Streaming Checklist
Description
• Target-specific measure for disruptive behavior
• Measures the extent to which a youth exhibits
pro-social skills (a strength-based measure)
• Elementary and Adolescent versions available
– Youth self-report
– Parent/caregiver report
– Teacher/staff report
• 5 – 15 minutes to complete
11
Skill Streaming Checklist
Description
• Part of the Skillstreaming curriculum
published by Research Press
– After purchase of Skillstreaming program
materials (~$45 per age version), there is no
per-use cost
• **20% discount if purchased through CIMH
– http://www.researchpress.com/skillstreaming/a
dolescent/product/item/6576/
12
Skill Streaming Checklist
Description
• 50 items (adolescent)
• 60 items (elementary)
• 5-point Likert scale
response options
–
–
–
–
–
Almost never (1)
Seldom (2)
Sometimes (3)
Often (4)
Almost always (5)
• Total Score
– Note: The SS Checklists
are not standardized
measures; therefore, a
subset of items can be
administered,
corresponding to the
specific set of pro-social
skills youth are being
taught. This is referred to
as a “targeted
assessment”.
13
Example: Items from the Skill
Streaming Checklist (adolescent)
• Does the youngster pay attention to someone
who is talking and make an effort to
understand what is being said?
– Skill: Listening
• Does the youngster request assistance when
he/she is having difficulty?
– Skill: Asking for Help
• Does the youngster let others know which
emotions he/she is feeling?
– Skill: Expressing Your Feelings
14
Skill Streaming Checklist
Administration
• Administer pre- and post- a dose of
treatment, or an intervention interval,
focused on disruptive behavior
– “Student” version of the Skill Streaming
Checklist completed by client
– “Parent” version completed by parent or
caregiver
– “Teacher” version completed by teacher or
staff member
15
Let’s take a look at the
Skill Streaming Checklist
(Student version)...
16
Administration and Clinical Utility of
the Skill Streaming Checklist
• Informs specific pro-social skills for which
youth have the greatest need for
intervention
– Can be used to determine the subset of skills
that are selected for skills training curriculum
• Administer checklist to program staff members and
ask them to complete it with the average client in
mind
• Administer to clients to determine actual skill
needs prior to intervention
17
Clinical Utility of the
Skill Streaming Checklist
• “Targeted Assessment”
– Pre- and Post- test only those skills that are taught in
your curriculum
– i.e., if 10 skills are taught – those same 10 skills are
selected for pre- and post- outcome assessment
• When all 50 skills are assessed, the gains youth make in 10
skills tend to be washed out by the 40 skills that youth have
not had an opportunity to learn and practice
– If the entire checklist is administered pre-treatment to
determine the subset that will be taught
• Score only those that are selected for the curriculum
• Administer only the subset at the post-assessment
18
Skill Streaming Checklist
Scoring
• Each item on the checklist has five response
options with values ranging from 1 – 5
• To obtain the Total Score, sum the values of
the responses circled by the respondent
– If all 50 skills have been assessed,
Total Scores will range from 50 – 250
– If a subset of skills are assessed in a “targeted
assessment”, the range will change
• e.g., if 10 skills have been assessed, Total Scores will
range from 10 – 50
• Missing items are not scored
19
Skill Streaming Checklist
Scoring
• Given the variability in number of skills that
can be assessed, Total Scores are not used
for outcome analyses
• For pre-/post- outcome analyses, a mean
item response is calculated (the average
item response)
– Total Score ÷ Number of Skills Assessed
– Scores for outcome analyses range from
1.0 to 5.0
20
Clinical Utility of the
Skill Streaming Checklist
• Assesses youth’s use of pro-social skills (a
strengths-based measure)
– Higher scores indicate better use of pro-social skills
– Lower scores indicate poorer use of pro-social skills
• Informs which pro-social skills youth have the
greatest need for intervention
• Comparisons of pre/post scores demonstrate
treatment-related outcomes in youth’s use of prosocial skills
21
Palette of Measures Data Entry
and Data Submissions
22
Palette of Measures Data Entry
Shell
• There is a separate spreadsheet in the excel
workbook (aka database) for each type of
data:
–
–
–
–
–
Demographics & Services
Pre- General outcome measure
Post- General outcome measure
Pre- Target-specific measure(s)
Post- Target-specific outcome measure(s)
• Specific outcome measure spreadsheets included in
each agency’s database varies across Palette of
Measures project participants
23
Palette of Measures Data Entry
Shell
• In addition to the spreadsheets that hold
data...
– There is an Instructions spreadsheet
• Basic data entry instructions
• Contact information for T.A. (Cricket Mitchell)
– There is a Data Lists spreadsheet at the end
of the workbook that you will not use
• Data Lists populate the pull-down menus in other
spreadsheets
24
Palette of Measures Data Entry:
Skill Streaming Checklist
PreSSChecklists
Client ID#
Skill Streaming
Date of Checklists
Self-Report:
Parent Report:
CCPPAssessment SSVersion1
SS#Skills1 SSScore1 SS#Skills1 SSScore1
Teacher/Staff
Report:
SSSS#Skills1 SSScore1
•There is a separate spreadsheet for Pre-SS data, Post-SS
data, and Mid-SS data
•In each spreadsheet, there is a field for the Version, # of
Skills Assessed, and Total Score for all possible informants
•In the event of missing data, leave the fields blank/empty.
Do not enter text into any of the fields.
25
Palette of Measures Data Entry:
Skill Streaming Checklist
MidSSChecklists
Client ID#
Self-Report:
Date of
Assessment
Skill Streaming
Assessment Checklists
Interval
SSVersionM
C-SS#SkillsM C-SSScoreM
•For agencies who will conduct mid-treatment assessments,
indicate the Assessment Interval in the Mid-SS spreadsheet
by selecting from the available pull-down menu
•e.g., 1st mid-treatment assessment, 2nd mid-treatment
assessment
26
But, before you enter any
outcome data, you’ll enter
Demographics & Service
Delivery Information...
27
Palette of Measures Data Entry:
Demographics & Services (1 of 2)
Client Information:
Primary (DSM-IV code) (DSM-IV code)
Client ID#
DOB
Gender Ethnicity Language Primary Axis I Secondary Axis I Therapist ID
•Use a unique identifier for Client ID#
•Categorical variables will have pull-down menus from which
you’ll select an option (e.g., gender, ethnicity, language)
•Dates should be entered as xx/xx/xxxx
•Axis I diagnoses s/b the numeric DSM-IV code
•Therapist ID is optional
28
Palette of Measures Data Entry:
Demographics & Services (2 of 2)
Focus 1 of Treatment
Focus1
Date of
First
Session
Date of Total # Completed
Last
of
Session Sessions Focus 1?
(if Focus 1 not
completed)
Completed
(if Services not
completed)
Reason
Services?
Reason
•Select Focus from pull-down menu (e.g., anxiety, depression)
•The shell will hold data for up to 4 foci, or treatment targets
•Enter Date of First Session
•The remaining fields are to be completed at the end of treatment
targeting this particular focus (e.g., Date of Last, # Sessions)
• 2 levels of “Completed?”
• Treatment targeting this particular focus
• Overall service delivery
29
Palette of Measures Data
Submissions (slide 1 of 3)
• Data submissions to CiMH will occur twice a year
throughout the duration of the project
– The end of each May (reflecting all clients served from the
initiation of the project through the end of that April)
– The end of each December (reflecting all clients served
from the initiation of the project through the end of that
November)
– Note that this is the anticipated schedule; actual data
submission dates may vary slightly
• An email notice will be sent to Palette of Measures
site leads approximately one month in advance of
each data submission deadline
30
Data Submissions (slide 2 of 3)
• Providers may choose from among the following
methods for submitting their Palette of Measures
Excel databases to CiMH:
– Use YouSendIt, or another secure web-based transfer
site, to submit data electronically
• YouSendIt (www.YouSendIt.com) is a vendor that supports the
secure transfer of electronic data (encrypted and passwordprotected)
– Mail a password-protected CD to CiMH and submit the
password separately (via email or phone)
– Email an encrypted, password-protected file(s) to CiMH
and submit the password separately (via email or phone)
31
Data Submissions (slide 3 of 3)
• After data are submitted, sites continue to
enter new data into the same database
– Always reflects an ongoing, historical record of
clients served through the Palette of Measures
project
• Every effort is made to distribute reports
within two months of each data submission
– Aggregate and site/agency-specific reports
32
Questions
33
The End
Contact Information
•Cricket Mitchell, PhD
•Email: cmitchell@cimh.org
•Cell phone: 858-220-6355
34