Case Study Evaluations for Field Based Practicum
EDC 514: Academic Assessment for Intervention
The case study rubric has been used for several years to evaluate student’s skill in
conducting case studies using the Response to Intervention methodology. Prior to and
including Winter 2004 evaluations were limited to case study implementation integrity.
Case Study Integrity Rating: Evaluation conducted by faculty using the Case Study
Rubric to determine how well the school psychology graduate student followed the case
study evaluation procedure. Ratings were based on the following:
5 = Outstanding:
All components in the
Competent and
Outstanding categories
are checked
4 = Substantially
Developed: All
components in the
Competent category
plus some components
in the Outstanding
category are checked
3 = Competent: All
components in the
competent category are
checked
2 = Threshold
Development: Some
components in the
competent category are
checked
1 = Needs
Development: Only
components in the
Needs Development
category are checked
Winter 2004
Student
T.B.
K.B.
J.B.
A.B.
D.P.
L.S.
T.S.
P.S.
A.L.
Case Study
Integrity Rating
5/5
5/5
5/5
4/5
5/5
5/5
5/5
5/5
5/5
The problem with the data obtained in Winter 2004 was that the outcome of the
procedure was not formally evaluated to determine the impact of the intervention on the
target behavior. The case study rubric was therefore, revised to include information to
indicate the impact of this method on the K-12 students served by our practicum students
and interns. In addition, to the case study integrity rating, effect sizes, and percent of nonoverlapping data points were reported.
Effect size: Effect sizes were calculated for each case study to determine the impact of
the intervention on the target behavior as follows:
Mean Intervention Score – Mean Baseline Score / Standard Deviation of All Scores
Large effect size = .8
Moderate effect size = .5
Small effect size = .2
Percent of Non-Overlapping Data Points: This is a measure to determine the impact of
the intervention by comparing data points during baseline with data points after
intervention implementation. 100% of non-overlapping data points indicates that the
intervention made a difference, while 0 non-overlapping data points indicates that there
was no difference across phases or that the data points were very unstable.
Winter 2005
Student
Case Study
Integrity Rating
Effect Size
S.K.
J.T.
S.A
K.B.
S.C.
K.H.
S.H.
H.H.
S.K.
K.S.
L.Y.
MEAN
SCORES
(STANDARD
DEVIATION)
5/5
5/5
5/5
5/5
4/5
5/5
5/5
5/5
5/5
4/5
5/5
1.47
6.67
% of NonOverlapping
Data Points
21%
100%
.70
75%
3.30
3.90
1.41
.11
100%
100%
100%
100%
100%
50%
Case Study Integrity Rating: Evaluation conducted by faculty using the Case Study
Rubric to determine how well the school psychology graduate student followed the case
study evaluation procedure. Ratings were based on the following:
5 = Outstanding:
All components in the
Competent and
Outstanding categories
are checked
4 = Substantially
Developed: All
components in the
Competent category
plus some components
in the Outstanding
category are checked
3 = Competent: All
components in the
competent category are
checked
2 = Threshold
Development: Some
components in the
competent category are
checked
1 = Needs
Development: Only
components in the
Needs Development
category are checked
Effect size: Effect sizes were calculated for each case study to determine the impact of
the intervention on the target behavior as follows:
Mean Intervention Score – Mean Baseline Score / Standard Deviation of All Scores
Large effect size = .8
Moderate effect size = .5
Small effect size = .2
Percent of Non-Overlapping Data Points: This is a measure to determine the impact of
the intervention by comparing data points during baseline with data points after
intervention implementation. 100% of non-overlapping data points indicates that the
intervention made a difference, while 0 non-overlapping data points indicates that there
was no difference across phases or that the data points were very unstable.
Goal Attainment Scale: Goals for improvement were established by the student and
team through collaborative data-based problem solving, and research prior to intervention
implementation. This is an indicator of both the success of the intervention, and the
accurate establishment of a goal and/or expected rate of improvement. A general
guideline is presented below:
-2 = much less improvement than expected
-1 = somewhat less improvement than expected
0 = improvement occurred at the level expected without intervention
+1 = somewhat more improvement than expected
+2 = much more improvement than expected
Winter, 2006
Student
Case Study
Integrity Rating
Effect Size
A.A.
K.B.
M.B.
S.C.
D.D.
S.E.
M.F.
S.G.
S.J.
P.M.
C.M.
J.M.
D.S.
5/5
5/5
4/5
5/5
5/5
5/5
5/5
5/5
5/5
5/5
5/5
5/5
5/5
1.85
.79
1.36
1.04
.7
1.51
.89
1.6
.4
1.74
1.69
2.54
.4
% of NonOverlapping
Data Points
100%
50%
71.4%
57%
14%
100%
33.33%
57%
37.5%
100%
100%
100%
30%
Goal
Attainment
Scaling
+2
+2
+2
-2
+2
+2
+1
+2
+2
+1
+2
+2
0
Z.T.
MEAN
SCORES
(STANDARD
DEVIATION)
5/5
1.58
100%
1.29 (.61)
67.87 (31.88)
+2
Aggregated Data Chart
Goal Attainment Scales for Ten 05-06 Interns on Four Outcome-Based Assignments
Intern
Academic Behavioral
Classwide
Individual
Classwide
Initials
Case Study Case Study Intervention Counseling
Description
TB
2
1
2
0
Acad
AL
0
0
1
0
Beh
PS
1
1
1
2
Acad
JT
0
0
1
0
Beh
JB
-1
1
1
1
Beh
AMB
-1.7
0.7
-1
1
Acad
DP
-1
1
1
1
Beh
TS
1
2
1
1
Beh
LS
2
2
0
0
Beh
KB
2
1
2
2
Beh
0.43
1.07
0.9
0.6
0.75
2007
Student
Rubric
Fidelity
AK
BS
CK
CB
CE
DP
LG
MA
MS
RC
TV
TM
TML
5/5
4/5
5/5
5/5
4/5
4/5
4/5
5/5
5/5
5/5
3/5
5/5
5/5
Goal
Attainment
Scaling
+2
+2
+1
+1
0
+2
+2
+2
0
+1
+2
+1
+2
Effect Size
.71
.40
1.4
1.63
1.67
1.11
1.5
1.01
1.53
1.63
1.19
1.41
.05
Nonoverlapping
Date Points
100%
100%
100%
84%
60%
100%
78%
42%
77%
100%
57.14%
25%
13%
Treatment
Integrity
99.7%
98%
100%
100%
100%
100%
100%
100%
100%
79.5%
93.75%
98.15%
91.5%
For the past several years, the PND tended to be somewhat inconsistent with the other
outcome data. Further study regarding outcomes assessment indicated that the PND data
were only appropriate to use under certain conditions. Therefore the aggregated data
charts reflect the elimination of this indicator.
2008
Student
Case Study
Integrity Rating
AW
AR
DM
EF
GA
JM
5/5
5/5
5/5
5/5
0/5
Incomplete
Effect Size
g-index = .5
g-index = 1
g-index = .12
g-index = .6
NA
Goal
Attainment
Scaling
+2
+2
+2
+2
NA