Small-Scale RCTs in
Clinical Practice:
Word Finding, Grammar and Inferencing
Susan Ebbels
Moor House School, Hurst Green, Surrey, UK
and
Division of Psychology and Language, University College London
Four small-scale RCTs
1. Verb semantic therapy and ‘Shape Coding’ therapy
for verb argument structure (Ebbels, van der Lely & Dockrell,
2007)
2. ‘Shape Coding’ therapy for grammatical
comprehension (Ebbels, Marić, Murphy & Turner, submitted)
3. Therapy for word finding difficulties (Ebbels, Nicoll, Clark,
Eachus, Gallagher, Horniman, Jennings, McEvoy, Nimmo & Turner, 2012)
4. Inferencing
Four small-scale RCTs
• Design
 ‘Blind’ assessment (student SLTs / other researchers /
SALTA)
 therapy group(s) vs. control group
 Control group receive same amount of intervention, but
on a different topic (usually unspecified)
• Participants
 From one special school for children with primary
language impairments
 severe language difficulties
 >9 years old (mostly >11 years)
Four small-scale RCTs
• Therapy
 1:1
 With SLT
• Statistical analyses
 First checked groups similar pre-therapy
 Compared progress (or normalised gain) between
groups - reduces variability
 For each group, compared progress to zero
Verb argument structure
27 participants
(11;0-16;1)
9
9
Verb
semantic
therapy
Shape Coding
therapy
9
Control
therapy
(inferencing)
Ebbels et al., 2007, JSLHR, 50, 1330-1349
Therapy
• 1 x 30 minutes per week for 9 weeks (4.5 hours
total)
• 1:1 with SLT (me!)
• Normal therapy and classroom teaching continued
in addition on different topics
Ebbels et al., 2007, JSLHR, 50, 1330-1349
Verb semantic therapy
• SLT and child jointly devise a definition of verb with as
much detail as possible
“pour”
• e.g.,
“fill”
 make something full to the top
 put something in it
liquid or lots of things
go down together
to a new place
container is tipping
• SLT / child act out event and the other decides if event
fits all the criteria (i.e., can be described by verb or not)
Ebbels et al., 2007, JSLHR, 50, 1330-1349
Shape Coding therapy
• Teach verbs in groups according to their broad meaning
• Teach matching constructions/argument structures using
Shape Coding templates
• E.g:
You
are
pouring the water
in the glass
MOVES
CHANGES
You
are
filling
the glass
with water
Ebbels et al., 2007, JSLHR, 50, 1330-1349
Normalised gain
Normalised gain score
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
• Difference in gain
between therapy
groups and
controls sig
(p<0.04, d>1.2)
• Two therapy
groups equal
gain
• Gains
significantly
greater than zero
at both timepoints
for both therapy
groups (p<0.02,
d>0.7)
to post-therapy
to follow-up
Semantic
Shape Coding
Therapy Group
Control
• Therefore
progress
maintained
Ebbels et al., 2007, JSLHR, 50, 1330-1349
Conclusions
• 4.5 hours of therapy can improve verb argument
structure
• Both therapy methods equally effective
• (Progress generalised to other verbs, but not to
morphology)
• Progress maintained for 3 months
Comprehension of conjunctions
14 participants
(11;3-16;1)
7
Phase 1
Phase 2
Shape
Coding
Therapy
Other
therapy
7
Other
therapy
Shape
Coding
Therapy
Ebbels et al., 2012, submitted
Therapy
• 1x 30 mins per week for 8 weeks (4 hours total)
• 1:1 with own SLT
• Using Shape Coding as a visual support
 Start by comparing ‘and’ and ‘but not’
 Then add in ‘neither nor’
 Then add in ‘not only but also’
• SLTs given sequence of steps to work through at
child’s own pace, stopping after 8 sessions,
regardless of step reached
Ebbels et al., 2012, submitted
Change in score
0.50
Phase 1
0.40
• Therapy group
more progress
than controls
after Phase 1
(p=0.01, d=1.64)
Phase 2
• After Phase 2,
both groups
equal progress
0.30
0.20
• Both groups
made sig
progress with
therapy (p<.04,
d>0.8)
0.10
0.00
-0.10
-0.20
-0.30
Original therapy group
Waiting controls
• Progress
maintained by
original group
Ebbels et al., 2012, submitted
Conclusions
• 4 hours of therapy improved comprehension of
conjunctions
• (Progress led to significant change in standard
score on TROG)
• (no significant predictors of progress, including
 Pre-therapy language scores
 NV IQ (Matrices)
 Visual-perceptual skills
 Age)
• Progress maintained for 4 months
Word finding
21 participants
(aged 9;9-15;11)
Phase 1
Phase 2
8
Semantic
therapy
6
Phonological
therapy
7
Other
therapy
Semantic
therapy
Ebbels et al., 2012, IJLCD, 47: 35-51
Unfortunately……
• Pupils allocated to groups before testing
completed
• When analysed, phonological group different from
semantic and control groups.
 4/6 were within normal limits on the TAWF (0/8 in
semantic and control groups)
• Therefore, unwise to compare all three groups
• Therefore, all analyses only on the semantic and
control groups.
 Can tell us whether the semantic therapy effective,
 but can’t tell us anything about the phonological therapy
Semantic therapy
• 2 x 15 mins per week for 8 weeks (4 hours total)
• 1:1 with own SLT
• Sort photo cards of one category (animals, clothes, food)
into semantic sub-categories (broad, then narrower)
• Discuss attributes of pictures
• Compare attributes of different pictures
• 20 questions giving cues
• Dice game – throw dice, pick up picture, give the cue on
dice for that picture
Ebbels et al., 2012, IJLCD, 47: 35-51
Change in Standard Score
25
Phase 1
20
• Therapy group
more progress
than controls
after Phase 1
(p=0.04, d=1.0)
Phase 2
• After Phase 2,
both groups
equal progress
15
10
• Both groups
made sig
progress with
therapy (p<.02,
d>0.8)
5
0
-5
-10
Original therapy group
• Progress
maintained by
Waiting controls
original group
Ebbels et al., 2012, IJLCD, 47: 35-51
Conclusions
• 4 hours of semantic therapy improved word finding
on a standardised test
• (no significant predictors of progress, including
 Pre-therapy word finding ability
 Pre-therapy language scores
 “Fast” vs. “slow namers”)
• (effects of therapy did not generalise to discourse)
• Progress maintained for 5 months
Inferencing
54 participants
(aged 10;7-16;0)
Phase 1
Phase 2
18
Lexical
inference
therapy
18
Mental
imagery
therapy
18
Other
therapy
CANCELLED
Therapy
• 1 x 30 mins per week for 9 weeks (4.5 hours total)
• 1:1 with own SLT
• Lexical inference, based on Yuill & Oakhill (1988)
• Mental Imagery - based on Visualising and
Verbalising (Bell, 1991)
• Waiting controls – other therapy, but class work
supported by SLT in English continued, which
often included work on inferencing
Change in score
0.16
• No sig dif in
progress
between groups
(p=0.9)
0.14
0.12
0.10
0.08
• All groups
(including
controls) made
sig progress
(p<.02, d>0.6)
0.06
0.04
0.02
0.00
-0.02
-0.04
Lexical
inferencing
Mental Imagery Waiting controls
Conclusions
• On average, all groups improved (although lots of
variability).
• Progress could have been due to:
1. Practice effect (doing the test twice) AND/OR
2. The work which continued elsewhere in the school
curriculum which often included work on inferencing
• Cannot distinguish between these as no control
group who were not in the school.
• However, adding 4.5 hours of 1:1 therapy on
inferencing to the rest of the school curriculum did
not lead to significantly greater progress
Summary of four RCTs
Verb
argument
structure
Cn of
Word
conjunctions finding
Inferencing
Participants
27
14
21 → 15
54
Number of
groups
3
2
3→2
3
Group size
9
7
7 or 8
18
Therapy >
control?
yes
yes
yes
no
Effect size of
therapy vs.
control
>1.2
1.64
1.0
<0.2
Progress >0
yes
yes
yes
yes
Effect size of
progress vs. 0
>0.7
>0.8
>0.8
>0.6
What I’ve learned!
• Check groups are equivalent pre-therapy (see WFD
study)
• Have a proper control group who do not have any
access to any other teaching / therapy on the topic
(see inferencing study)
• The waiting controls get the best deal – only
receive therapy if proven effective (no ethical
issue!)
• The effect size is more important than the group
size
 Both for finding a significant effect
 And for clinical value
Effect sizes and
significance
• Effect size (d) is:
 difference between groups, or between progress and
zero (bigger differences increase effect sizes) DIVIDED
BY
 standard deviation (greater variability decreases effect
sizes)
• Significance depends on
 Effect size (larger effect sizes increase significance)
 Number of participants in a group (more participants
increases chance of finding a significant effect, if there
is one to find – increases “power”).
So….
• If your effect size is big enough, you don’t need
large groups to find a significant effect
• I found significant effects in three studies for
 Group sizes of 7 or more
 With effect sizes of d>0.8
To conclude…
• Small-scale RCTs are not too hard to carry out as
part of clinical practice
• lack of significance in small-scale RCTs is hard to
interpret (can’t be used to disprove effectiveness
due to insufficient power)
• BUT, a significant effect is likely to be clinically
important
• Therefore, the effort of doing small-scale RCTs in
clinical practice is worthwhile to find clinically
important effects.
Questions?