Doctorate in Educational and Child Psychology
Tamara Powell
Case Study 1: An Evidence Based Practice Review Report
Theme: Parent Interventions
How effective is Parent Child Interaction Therapy (PCIT) at reducing symptom severity
in young children with internalising disorders?
1. Summary
Parent-Child Interaction Therapy (PCIT) is an empirically supported treatment
designed to help parents manage problematic behaviour and build nurturing
relationships with their child. Parents receive live coaching via a bug in the ear
receiver from a therapist situated behind a one-way mirror throughout the course of
treatment. Psychoeducational support that explains the treatment rational, nature of
the disorder and effective parenting strategies is also interleaved into the various
treatment modules.
This literature review aimed to evaluate the effectiveness of PCIT and its
neighbouring protocols in reducing symptom severity in young children with
internalising disorders. A systematic literature search conducted over three
databases revealed six studies, which were subsequently coded (adapted protocol
from Kratochwill, 2003) and appraised using Harden and Gough’s (2012) ‘Weight of
Evidence Framework.’ This review found moderate evidence for PCIT – Emotion
Development (PCIT-ED) and preliminary evidence to support the effectiveness of
Coaching Approach Behaviour Leading by Modeling (CALM) in reducing symptoms
related to depression and anxiety. Traditional PCIT programmes, however, were
found to be less effective in managing separation anxiety disorder (SAD) symptoms
in children ages 3 to 8. Research that utilizes larger sample sizes and greater
methodological rigour are warranted.
2. Introduction
2.1 Description of PCIT
PCIT is an empirically validated psychotherapeutic intervention originally
developed to treat young children (2-8 years) with disruptive behaviour problems.
This intensive 8-14 week manual-based training programme is comprised of two
Doctorate in Educational and Child Psychology
Tamara Powell
distinct phases: Child-Directed Interaction (CDI) and Parent-Directed Interaction
(PDI). The former is focused on improving the parent child relationship and the latter
in reducing disruptive behaviour. Both phases of treatment typically begin with a
didactic training session that introduces key principles and skills and proceeded by
individualised coaching sessions that provide parents live in vivo feedback.
The CDI phase teaches parents PRIDE (Praise, Reflection, Imitation,
Description, and Enthusiasm) coaching skills to elicit cooperative play behaviour,
whereas the PDI phase focuses exclusively on firm, non-punitive disciplinary
practices. During PDI, parents are trained to consistently praise desired behaviour
(e.g., self-confidence, independence) and selectively ignore undesired behaviour
(e.g., aggressive outbursts) through labelled praise and systematic ‘time-outs’. Daily
homework is given throughout treatment and the programme is ceased once parents
have mastered the CDI and PDI skills and their child’s internalising behaviour is
brought within normal limits.
PCIT can be administered by any mental health professional who has
obtained at least a master’s degree in their field of work and undergone the required
year-long training programme (Eyberg, 1999). In most cases, however, the
programme is delivered by a clinical psychologist in a clinic-based setting.
2.2 Adaptations of PCIT
2.2.1 CALM
CALM is an intensive 12 week intervention that treats young children with
social phobia, generalized anxiety disorder (GAD), SAD and specific phobias. Similar
to PCIT, CALM is a mastery based protocol that uses live, bug in the ear technology
to coach parents in using positive attending and developmentally sensitive play skills
(PRIDE). The programme begins with a teaching session and CDI phase of
treatment which focuses on strengthening the parent-child relationship. Unlike PCIT
however, the PDI phase is replaced by exposure sessions (brave behaviours
module) that require parents to use directive parenting skills (Figure 1) to coach their
child through increasingly difficult situations. Parents work alongside therapists to
develop individualised fear hierarchies which children gradually achieve throughout
the course of treatment.
Doctorate in Educational and Child Psychology
Tamara Powell
Figure 1. Sequence of DADS steps (Puliafico, Comer & Albano, 2013).
2.2.2 PCIT-ED
PCIT-ED is a developmentally lateral extension of PCIT for young children
with early-onset depression. The programme includes three modules: CDI, PDI and
the novel emotion development (ED) module. Throughout the ED module, children
are encouraged to identify and label personal triggers as a way to consciously
regulate precipitating stressors. Parents are also concurrently taught behavioural
techniques to reinforce adaptive behaviour and strategies to manage their own
negative affect.
2.3 Psychological Basis
PCIT is built on the tenets of social learning theory (Bandura, 1977), operant
conditioning (Skinner, 1938) and attachment theory (Bowlby, 1978). Although
attachment theory forms part of the theoretical framework that guides PCIT, the
intervention is primarily rooted in behaviourism and social learning theory (Allen,
Timmer & Urquiza, 2014). The application of behavioural principles is arguably the
most integral component of PCIT, as moment by moment functional analyses
informs all treatment related decisions.
Behavioural principles support the re-structuring of dysfunctional parent-child
interactions, which according to social learning theory inadvertently develop in
response to child behaviour problems. Coercive interactions between parent and
child develop in a cyclical manner and are maintained through negative
Doctorate in Educational and Child Psychology
Tamara Powell
reinforcement strategies (Stormont, 2002). Children’s maladaptive behaviour is
therefore learned and preserved by parents (Patterson, Capaldi & Bank, 1991). With
this in mind, PCIT attempts to specifically target critical, over-bearing parenting,
found in anxious parents (Whaley, Pinto & Sigman, 1999) and inconsistent lowaccepting mothers in children with major depressive disorder (MDD; Gonzales, Pitts,
Hill & Roosa, 2000). Consistent disciplinary practices in PDI that differentially
reinforces adaptive behaviour is accordingly well suited for this population who
exhibit dysfunctional, and often, reciprocal patterns of behaviour.
Parent-child attachment style affects a child’s ability to effectively cope and
regulate emotions during distressing or fearful situations (Cassidy & Kobak, 1988).
That is, children who have parents that are sensitive and responsive to their needs
(secure attachment) will come to consistently expect this treatment and view
negative affect as a temporary state, which they will eventually master. The CDI
phase of PCIT is thought to underpin changes in children’s internal working models,
as parents are taught how to interpret their child’s cues and use positive parenting
practices to facilitate warm and mutually responsive relationships (Timmer et al.,
2011).
2.4 Rational
Forty six percent of reception teachers now report that more than half of the
pupils in their class were not ready for school (West, Denton, & Reaney, 2001).
School readiness, defined as, emotional self-regulation, social competence, the
absence of behaviour problems and parental involvement, plays a pivotal role in
children’s future social and academic success (Ladd, Birch, & Buhs, 1999;
Normandeau & Guay, 1998). Unfortunately, children with internalising disorders are
at a heightened risk of exhibiting maladaptive emotional, social and behavioural
difficulties that impede their ability to function productively within social and
academic settings at school.
The preschool period offers an early window of opportunity to offset potential
long term and more intractable forms of psychopathology later in life. In fact, the
neuroplasticity of the brain during this developmental period may serve to enhance
robust treatment effects in this population (Luby, 2013). Early childhood
interventions, such as PCIT, may also hold the potential to ameliorate long term
Doctorate in Educational and Child Psychology
Tamara Powell
suffering, symptom impairment and reliance on psychotropic medication that carry
risks and dangerous side effects for young children (Wagner, 2005).
There is a growing body of evidence that highlights the importance of
prevention and early intervention for childhood psychiatric disorders (DfES, 2001). In
applied educational psychology practice, early interventions, including PCIT, can
help reduce the number of permanent and fixed term exclusions and incidence of
peer victimisation in schools. By building children’s resilience at a young age through
secure early attachments and clear, firm and consistent disciplinary practices (DfE,
2015), students will also be more likely to establish healthy social relationships with
peers and adults and consequently show higher levels of academic engagement and
attainment.
Psychosocial treatments for school aged children has been well-researched
over the past 20 years, (Kendell, Furr, & Podell, 2010; Silverman, Pina &
Viswesvaran, 2008), however, there is a paucity of work examining the treatment of
anxiety and depression in early childhood. This may be due, in part, to the fact that
empirically validated approaches to treating older children rely on higher cognitive
capacities, such as thought monitoring, cognitive restructuring and mood
identification, which have not yet developed in young children. PCIT circumvents this
particular issue by focusing on reshaping the primary context within which
development occurs (parent-child relationship), rather than working directly with
children who lack the key metacognitive skills needed for treatment.
Minor adjustments will likely be needed before PCIT can be implemented in
the UK. Due to the intrusive nature of the intervention, it would be worthwhile to
discuss with parents any personal boundaries that should be respected during
treatment. Likewise, professionals should be mindful of their tone and directiveness
when instructing parents, as it is imperative they feel like active agents in the
problem solving process.
2.5 Review question
How effective is PCIT and its various adaptations in reducing symptom severity in
young children with internalising disorders?
Doctorate in Educational and Child Psychology
Tamara Powell
3. Critical Review of Evidence
3.1 Literature Search
A comprehensive literature search was carried out between the dates of
December 5th and 30th, 2014 to locate studies that were relevant to the research
question. The literature search spanned three databases (PsycINFO, ERIC,
MEDLINE) and used the search terms outlined in Table 1.
The initial search, which was limited to the inclusion criteria summarised in
Appendix 1, yielded 381 results (PsycINFO, n = 177; ERIC, n = 149; MEDLINE,
n = 55). After removing 242 duplicates, the titles and abstracts of 127 studies were
screened for eligibility, and a further 12 articles were examined through full text. On
closer inspection, four papers were found to be review studies that did not report
original empirical data and two focused on children with externalising behaviour
problems (Appendix 2). Figure 2 depicts the entire screening process from the initial
database searches to full article review and Table 2 identifies the six studies that
were included in the final review.
Table 1
Search Terms Applied
Multifield Search Terms
Parent child interaction therapy
(abstract) and internal* (all fields)
Parent child interaction therapy
(abstract) and anxiety (all fields)
PCIT (abstract) and anxiety (abstract)
Parent child interaction therapy
(abstract) and depression (all fields)
PCIT (abstract) and depression
(abstract)
Parent child interaction therapy
(abstract) and mood disorder* (all
fields)
Parent child interaction therapy
(abstract) and inhibit* (all fields)
Parent child interaction therapy (title)
and internal* (all fields)
Limits
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
Doctorate in Educational and Child Psychology
Multifield Search Terms
Limits
Parent child interaction therapy (title)
and depression (all fields)
Parent child interaction therapy (title)
and mood disorder* (all fields)
Parent child interaction therapy (title)
and inhibit* (all fields)
Parent child interaction therapy (title)
and behavio*(title)
PsycINFO n = 177
ERIC
n = 149
Participants
MEDLINE n = 55
TotalResearch
n Design
= 381
Control Group
Tamara Powell
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
English language, peer reviewed, 2005current, human research
Articles
Identified
Using Search
Terms
Papers Excluded by abstract:
Inclusion criteria not met n = 127
Duplicates n = 242
Papers Full
Text
Reviewed
Papers Excluded:
Inclusion criteria 1b n = 4
Inclusion criteria 2b n = 2
Total Excluded n = 6
Included
Papers n = 6
Figure 2. Application of inclusion and exclusion criteria.
Doctorate in Educational and Child Psychology
Tamara Powell
Table 2
Studies Eligible for Review
Included Studies
Chase, R. M., & Eyberg, S. M. (2008). Clinical presentation and treatment outcome
for children with comorbid externalizing and internalizing symptoms. Journal
of Anxiety Disorders, 22(2), 273–82.
Choate, M. L., Pincus, D. B., Eyberg, S. M., & Barlow, D. H. (2005). Parent-child
interaction therapy for treatment of separation anxiety disorder in young
children : A pilot study. Cognitive and Behavioral Practice,12(1),126–135.
Comer, J. S., Puliafico, A. C., Aschenbrand, S. G., McKnight, K., Robin, J. A.,
Goldfine, M. E., & Albano, A. M. (2012). A pilot feasibility evaluation of the
CALM program for anxiety disorders in early childhood. Journal of Anxiety
Disorders, 26(1), 40–9.
Lenze, S. N., Pautsch, J., & Luby, J. (2011). Parent-child interaction therapy emotion
development: A novel treatment for depression in preschool children.
Depression and Anxiety, 28(2), 153–9.
Luby, J., Lenze, S., & Tillman, R. (2012). A novel early intervention for preschool
depression: findings from a pilot randomized controlled trial. Journal of Child
Psychology and Psychiatry and Allied Disciplines, 53(3), 313–22.
3.2 Evaluation Process
This review is based on six papers that are delineated in Table 2. Included
studies were critically evaluated using Harden and Gough’s (2012) ‘Weight of
Evidence Framework’ (WoE; Table 3). This framework appraises studies on four
dimensions: methodological quality, methodological relevance, relevance to the
review question and overall assessment. Overall assessment, computed by
averaging the WoE A, B and C, is particularly important because it provides a
comprehensive measure that facilitates comparative inferences to be drawn (Table 4
and 5).
Doctorate in Educational and Child Psychology
Tamara Powell
Table 3
Weight of Evidence Framework (Harden & Gough, 2012)
Weight of
Evidence A Methodological
Quality
The study’s
methodological
quality is judged
against a generally
accepted criteria.
Weight of Evidence
B - Methodological
Relevance
Weight of Evidence
C- Relevance to the
Review Question
Review specific
judgement about the
appropriateness of
the study design in
answering the
review question.
Review specific
judgment about the
relevance of the
type of evidence
provided in relation
to the review
question.
Weight of
Evidence D Overall
Assessment
Averaged weight
of evidence from
sources A, B and
C.
Each study was coded using an adapted protocol from the Task Force on
Evidence-Based Intervention in School Psychology (Kratochwill, 2003). An overall
summary of each included paper and corresponding coding protocols can be found
in Appendix 5 and 7, respectively. The weighting criteria and outcome scores for
group and single based designs are also detailed in Appendix 3 and 4. Finally, Table
4 reports the results from WoE A, B, C and D for all six included studies.
Table 4
Weighting of Evidence across the Four Scoring Dimensions
Reviewed
Study
Chase &
Eyberg
(2008)
Choate et
al. (2005)
Comer et al.
(2012)
Lenze et al.
(2011)
Luby et al.
(2012)
Pincus et al.
(2008)
WoE A
WoE B
Methodological Methodological
Quality
Relevance
WoE C Relevance
to the Review
Question
1.8
1.0
1.0
0.8
1.0
1.0
1.2
1.0
3.0
1.8
1.0
2.0
1.8
3.0
3.0
1.0
1.0
1.0
WoE D
Overall
Assessment
Low
1.3
Low
0.9
Med
1.7
Med
1.6
High
2.6
Low
1.0
Doctorate in Educational and Child Psychology
Tamara Powell
Table 5
Overall Weight of Evidence
Overall Assessment Score (WoE D)
High
Medium
Low
Averaged Scores
2.5 or higher
1.5 to 2.4
1.4 or less
3.3 Critique and Appraisal of Studies
3.3.1 Participants
A total of 131 children ages 3 to 8 years participated in the six reviewed
studies. Children with depression were generally younger in age (preschool period),
while children with anxiety related disorders spanned both the preschool and school
age period. Gender was slightly skewed in favour of males (61%), and a much
higher proportion of White children were included in four (Chase & Eyberg, 2008;
Choate, Pincus, Eyberg & Barlow, 2005; Lenze, Pautsch, & Luby, 2011; Luby, Lenze
& Tillman, 2012) out of the five studies that reported ethnicity and race. Families
were also more likely to have high annual incomes (>60,000) and educational
attainment (Lenze et al., 2011; Luby et al., 2012).
With the exception of one study, all participants met criteria for a principle
diagnosis of MDD (Egger & Angold, 2006) or a specific anxiety related disorder as
judged by various DSM-IV screening measures (Appendix 5). Participants in Chase
and Eyberg’s (2008) study displayed clinical levels of oppositional defiant disorder
(ODD) and SAD, however, the primary diagnosis was not reported. Although
participants were not concurrently receiving any other psychotherapies, several
children in the PCIT-ED group were taking stable doses of psychotropic medication.
Sample size varied across studies, as both group and single-based designs
were included in the present review. In the group-based studies, stringent inclusion
criterion and accessibility of low incidence populations impeded recruitment efforts.
Consequently, only one (Chase & Eyberg, 2008) of the four papers employed a
sufficiently large sample. Test sensitivities were therefore compromised in the
remaining three papers (Lenze et al., 2011; Luby et al., 2012; Pincus, Santucci,
Ehrenreich & Eyberg, 2005), which were all underpowered.
Doctorate in Educational and Child Psychology
Tamara Powell
The level of parental involvement differed across papers. However, there was
no evident group differences between single and co-parenting couples who
participated in treatment. As seen in Table 4, overall WoE varied between low
(Choate et al., 2005) and medium (Comer et al., 2012) ratings for studies that
included both parents in all facets of treatment. Lastly, although the reviewed studies
were carried out in an English speaking country (United States), there has been
none to date conducted in the UK. Results should therefore be interpreted with
caution, as cultural norms, legislation and educational systems differ between the
two countries.
3.3.2 Research Design
Two of the reviewed papers (Choate et al., 2005; Comer et al., 2012) used
single-case experimental designs (SCED) to explore PCIT and CALM treatment
effects. This design, however, was found to be poorly suited for the research
question at hand. When examining methodological quality across both multiple
baseline designs, it is apparent that symptom severity is ill-defined, and for this
reason, measured at only 3 single time points across treatment. The inadequate
amount of data is likely due to the fact that symptom severity, a broad concept that
encompasses adverse changes in physical, emotional and social functioning
(Kessler et al., 2002), is a very difficult operationalise, and therefore, overtly observe
through time. Choate et al. (2005) consequently received an unclassified score of 0
for the quality of baseline and primary outcome measures, as there was an
insufficient number of data points to judge the stability, overlap and trends in scores
during the baseline and intervention time period.
Comer et al. (2012) reported group averages for 9 participants at three
different time points (baseline 1, baseline 2, and posttest) instead of showing three
demonstrations of an experimental effect (Horner et al., 2005). The authors did not
compute any statistical tests or provide sufficient data to allow independent
calculations of significance. As such, the primary outcome rating could not be
estimated using SCED or group experimental design (GED) guidelines, and it was in
turn awarded no numerical ranking. Similar to Choate et al. (2005), this study also
received an unclassified score of 0 for the quality of baseline because its short length
precluded any conclusions regarding the level of the symptom severity in participants
prior to treatment.
Doctorate in Educational and Child Psychology
Tamara Powell
Within the GEDs, one randomised control trial study (RCT; Luby et al., 2012)
offered the highest scores on all four weights of evidence. This strong weighting
reflects the fact that Luby et al. (2012) employed an active comparison group,
utilized multiple methods to assess symptom severity and randomly assigned
participants to different experimental conditions. Two (Chase & Eyberg, 2008; Lenze
et al., 2011) of the three remaining one-group pretest-posttest GEDs were also found
to be methodologically sound (WoE A). In summary, GEDs methodology quality
(WoE A) was found to be superior to SCEDs in measuring differences in symptom
severity. Despite some efforts to increase experimental control through staggered
start times (Choate et al., 2005; Comer et al., 2005), both SCED and GEDs received
consistently low ratings on WoE B. Without a comparison group or within-participant
control, one-group pretest posttest and multiple baseline designs face the inherent
risk of having confounding variables influence participants’ symptom severity scores.
3.3.3. Intervention
3.3.3.1 PCIT
Three studies (Chase & Eyberg, 2008; Choate et al., 2005; Pincus et al.,
2008) examined the use of traditional PCIT in treating young children with SAD. The
overall rating for each PCIT study was judged to be low and this ranking was shown
to be largely consistent across WoE A, B and C. This low ranking can be attributed
to a number of factors, including the lack of multi-sourced and multi-method data,
poor methodological rigour and comparatively more participants with comorbid
diagnoses. Further, children in Choate’s et al. (2005) study only participated in a total
of 6 to 7 sessions, which is substantially shorter than the average 13-session length
(Bell & Eyberg, 2002). Choate et al. (2005) and Pincus et al. (2008) also failed to
stipulate whether neurological impairment, cognitive delay, psychotropic medication
and family discord had been examined during the initial screening phase.
Although, Chase and Eyberg’s (2008) study was appraised as being low on
WoE B, C and D, it did show moderate evidence for methodological quality. A higher
score on this dimension is probably due to longer treatment sessions (14 sessions),
stringent inclusion criterion, high treatment fidelity, positive changes in diagnostic
criteria and sufficiently large sample size.
Doctorate in Educational and Child Psychology
Tamara Powell
3.3.3.2 CALM
Although the preliminary results of Comer’s et al. (2012) pilot study showed a
moderate effect of treatment on children’s clinical outcome scores, the study failed to
isolate the unique effects of the novel brave behaviour module (WoE A). There was,
however, a high level of overall care taken to ensure positive outcomes, which
included, a sufficiently long course of treatment (12 sessions), involvement of both
parents and the administration of multiple assessment measures (WoE C). Such
efforts resulted in a medium overall weight of evidence (WoE D).
3.3.3.3 PCIT-ED
The strongest evidence in this review comes from Luby’s et al. (2012) RCT
study examining the effects of PCIT-ED on preschool children with MDD. This study
built on the previous work of Lenze et al. (2011) who examined PCIT-ED in a smaller
sample of children using a one-group pretest posttest design. Both studies
demonstrated a high fidelity to treatment, employed multiple assessment methods
and offered clinically significant results (WoE A).
3.3.4 Measures
Most studies (Choate et al., 2005; Comer et al., 2012; Pincus et al., 2008)
utilised the Anxiety Disorder Interview Schedule – Child and Parent Version (ADISC/P) (Silverman & Albana, 1996) instrument to measure symptom severity in
children with anxiety related disorders. The ADIS-C/P is a semi-structured diagnostic
interview that uses the DSM-IV criteria to produce a clinician severity rating (CSR)
score between 0 and 8 (> 4 indicates a clinical diagnosis). The inter-rater reliability
for ADIS-C/P ranges between 0.65 and 0.88 (Silverman & Ollendick, 2005;
Silverman, Saavedra & Pina, 2001), which is comparably higher than other more
generic measures of SAD, such as the Diagnostic Interview Schedule for ChildrenParent Version (DISC-IV-P; retest reliability = 0.58; Shaffer, Fisher, Lucas, Dulcan, &
Schwab-Stone, 2000). Comer et al. (2012) also employed the Children’s Global
Assessment Scale (CGAS; Shaffer et al., 1983) and Clinical Global Impression –
Severity Scale (CGI-S) scores to measure the overall severity of child disturbance.
However, the inter-rater reliability for CGAS is quite variable (0.62 to 0.99; Lavigne et
al., 1995), and at present, reliability and validity for CGI-S has only been evaluated in
samples of adults with dementia (Dahlke et al., 1992).
Doctorate in Educational and Child Psychology
Tamara Powell
Preschool Age Psychiatric Assessment (PAPA Egger & Ascher, 1999), which
assesses the intensity, frequencies, durations and first onsets of behaviour relevant
to Axis 1 diagnoses on the DSM-IV, was the preferred diagnostic tool for preschoolers with MDD (Lenze et al., 2011; Luby et al., 2012). This is presumably
because PAPA (test-retest reliability = 0.71; inter-rater reliability = 0.72) and other
similar screening measures, such as Preschool Feelings Checklist (PFC; Cronbach’s
alpha = 0.92; test-retest reliability = 0.82; Luby, Heffelfinger, Koenig-McNaught,
Brown & Spitznagel, 2004) and Health and Behavior Questionnaire – Parent Version
(HBQ-P; test-retest reliability = 0.88; Essex et al., 2002) show favourable
psychometric properties.
Aside from Comer’s et al. (2012) CALM study, there were no other anxiety
related interventions that employed more than one measure to evaluate symptom
severity in children. As a result, all traditional PCIT interventions scored low on WoE
C and generally received low ratings (Chase & Eyberg, 2008; Choate et al., 2005) on
the measurement item of WoE A. Notably poor reliability on Comer’s supplementary
measures (CGAS and CGI-S) also effected this intervention’s (CALM) rating on WoE
A (measurement).
Interventions tailored for children with depression (PCIT-ED) that conversely
used two (Lenze et al., 2011) or more (Luby et al., 2012) relevant outcome measures
received medium (Lenze et al., 2011) to high (Luby et al., 2012) ratings on WoE C.
Both PCIT-ED interventions were also awarded medium ratings on the measurement
scale of WoE A, indicating such studies utilised sound evaluative tools.
3.3.5 Findings
3.3.5.1 Anxiety
For the PCIT intervention, the results were mixed. Choate et al. (2005) and
Chase and Eyberg (2008) found positive treatment effects, with high responder rates
of 100% (3 out of 3) and 73% (11 out of 15), respectively. The majority of
participants in Pincus et al. (2005) study, however, still met criteria for SAD at
posttest (M = 4.2), and only marginal reductions (M = 1.6) were observed in
participants CSR scores. Although this reduction resulted in a medium effect size,
Doctorate in Educational and Child Psychology
Tamara Powell
there was no evidence to suggest PCIT made clinically relevant changes in
participants’ symptom severity. Chase and Eyberg (2008) on the other hand,
effectively demonstrated both statistical (large effect size) and clinical significance for
most children with SAD and ODD in their sample. A large effect size was also found
for all three participants in Choate’s et al. (2005) study.
In Comer’s et al. (2012) intent to treat sample, there was an overall mean
reduction of 2.8 (SD = 2.0) CSR points, which equated to a large effect size. Results
indicate that children with SAD were the most responsive to treatment (75%),
followed by social anxiety disorder (67%), GAD (50%) and specific phobia (0%).
Large effect sizes and high response rates (e.g., improvement of 10 points on GCAS
or scores lower than 3 on CGI-S) were similarly observed in the completers group for
the CGAS (85.7%) and CGI-S (100%) measures.
3.3.5.2 Depression
Positive changes in diagnostic profiles were reported in Lenze’s et al. (2011)
study. Indeed, 5 out of the 7 participants (71%) showed subclinical levels of MDD at
post-treatment and large effect sizes were noted for the PAPA MDD sum severity
score and HBQ-P internalising symptom subscale. Luby et al. (2012) found a large
effect size for PFC-S, however, the study reported a medium effect size for PAPA
MDD and no effect for HBQ-P. Given the differences in sample size and design
between the two studies, it’s more likely that the conservative estimates of PAPA
MDD and HBQ-P provided by Luby et al. (2012) is closer to the population
parameter.
Table 6 provides an overview for the aforementioned effect sizes. Caution
needs to be exercised, however, when interpreting these results. Importantly, the low
overall weights of evidence for all traditional PCIT interventions (e.g., only covering
the CDI and PDI modules) undermine the large (Choate et al., 2005) and medium
effect sizes (Chase & Eyberg, 2008; Pincus et al., 2008) reported in the current
review. Greater weight should instead be assigned to those interventions such as
CALM (Comer et al., 2012) and PCIT-ED (Lenze et al., 2011; Luby et al., 2012) that
received higher overall weights of evidence. Further, small sample sizes (Choate et
al., 2005; Comer et al., 2012; Lenze et al., 2011; Luby et al., 2012; Pincus et al.,
2008), which carry higher error rates and lower power may bias the obtained results.
It should also be noted that effect sizes were calculated using different evaluative
Doctorate in Educational and Child Psychology
Tamara Powell
frameworks (Becker, 1988; Morris, 2007). Comparative inferences, should therefore,
be avoided between studies with and without a comparison group.
Doctorate in Educational and Child Psychology
Tamara Powell
Table 6
Effect Size and Overall Quality Ratings
Study
Intervention &
Programme
Modules
Design
Outcomes
Measure(s)
Population
Effect Size
Descriptors
Effect Size
n
Overall
Weight of
Evidence
Chase &
Eyberg
(2008)
PCIT – CDI
and PDI
GED –
One
group
pretest
posttest
design
Group x time
interaction, with
significant
decreases in
symptom
severity in the
SAD only group
at posttest.
Diagnostic Interview
Schedule for
Children was
administered to
child’s female
caregiver.
(DISC-IV-P)
Anxiety
Medium
PPC SMD = 1.48
58
Low
Choate
et al.
(2005)
PCIT– CDI
and PDI
SCED –
Multiple
baseline
design
Clinical severity
ratings for SAD
decreased to
subclinical
levels at
posttest and
remained in the
normative range
at follow up.
Anxiety Disorders
Interview Schedule
for DSM-IV – Child
and Parent
Versions. (ADISC/P)
Anxiety
Large
Participant 1de
Posttest:
PP SMD = 1.29
Follow up:
PP SMD = 1.29
3
Low
Large
Participant 2de
Posttest
PP SMD = 1.71
Follow up:
PP SMD = 1.71
Large
Participant 3de
Posttest
PP SMD = 1.29
Follow-up:
PP SMD = 0.86
Doctorate in Educational and Child Psychology
Study
Comer et
al.
(2012)
Intervention &
Programme
Modules
Design
Outcomes
Measure(s)
CALM – CDI
and brave
behaviour
(exposure
therapy)
SCED –
Multiple
baseline
design
A diagnostic
response was
observed in
75% of SAD,
67% of social
anxiety, 50% of
GAD and 0% of
specific phobia
participants.
Anxiety Disorders
Interview Schedule
for DSM-IV – Parent
Version
(ADISC-P)
On average,
global severity
scores improved
from
“moderately”
and “markedly”
ill to “normal”
“not at all ill” and
“borderline ill.”
Lenze et
al.
(2011)
Tamara Powell
PCIT-ED–
CDI, PDI and
Emotion
Development
GED –
One
group
pretest
posttest
design
Significant
decline in
depression
severity scores
following
treatment.
Effect Size
n
Overall
Weight of
Evidence
Largea
PP SMD = 1.26de
9
Medium
Children’s Global
Assessment Scale
(CGAS)
Largeb
PP SMD = 1.64de
7
Clinical Global
Impression Severity
and Improvement
Scales (CGI-S)
Largeb
PP SMD = 2.26de
7
Large
PP SMD =1.19d
8
Large
PP SMD =0.82d
8
Preschool Age
Psychiatric
Assessment (PAPA
MDD)
Health and Behavior
Questionnaire –
Internalising
Symptoms Subscale
(HBQ-P)
Population
Anxiety
Depression
Effect Size
Descriptors
Medium
Doctorate in Educational and Child Psychology
Study
Luby et
al.
(2012)
Pincus et
al.
(2008)
Intervention &
Programme
Modules
Design
PCIT-ED –
CDI, PDI and
Emotion
Development
GED –
Between
subjects
randomis
ed
pretest
posttest
design
PCIT– CDI
and PDI
GED –
One
group
pretest
posttest
design
Outcomes
Tamara Powell
Measure(s)
Effect Size
Descriptors
Effect Size
Medium
Preschool Feelings
Checklist-Scale
(PFC-S)
Health and Behavior
Questionnaire –
Internalising
Symptoms Subscale
(HBQ-P)
Significant
decreases in
symptom
severity as
measured by
the PFC-S and
HBQ-P in both
experimental
conditions.
PAPA MDD
conversely only
found significant
changes in the
PCIT-ED group.
Preschool Age
Psychiatric
Assessment (PAPA
MDD)
SAD
symptomatology
decreased after
treatment,
however, most
scores
remained in the
clinical range.
Anxiety Disorders
Interview Schedule
for DSM-IV – Child
and Parent Versions
(ADIS-C/P)
Population
Depression
Anxiety
n
Overall
Weight of
Evidence
PPC SMD = 0.50
42
High
Large
PPC SMD = 0.83
43
No effect
PPC SMD = 0.00
40
Mediumc
PP SMD = 0.73d
10
Low
Notes:
Effect size, referred to as Pretest-Posttest Standard Mean Difference (PP SMD; Becker, 1988), was calculated by dividing the difference in means by the pretest standard deviation score (Choate et al., 2005; Comer et al., 2012; Lenze et al., 2011; Pincus et al., 2008). In Chase and Eyberg (2008) and Luby’s et al.
(2012) study, effect size (Pretest-Posttest-Control Standard Mean Difference; PPC SMD) was calculated by dividing the difference in means (pre and post
treatment) by the pooled standard deviation (Hedge’s g; Morris, 2007).
Doctorate in Educational and Child Psychology
Tamara Powell
a) To reduce potential sampling biases, statistical analyses was performed on the intent to treat sample.
b) Data could not be obtained for the intent to treat sample, so effect size was computed using the completers sample.
c) Standard deviation for measure was not reported, it was, therefore, computed using a sample of children with comparable demographics (Comer et
al., 2012).
d) Effect sizes were corrected in accordance to Hedges’ (1981) guidelines for small samples (e.g., groups with 10 or less participants; Becker, 1988).
e) Choate et al. (2005) examined the impact of treatment on children’s symptomatology on an individual basis, whereas Comer et al. (2012) explored
pre-post treatment differences using group means. To maintain consistency, the current review reported individual effect sizes at the two phases of
treatment in Choate’s et al. (2005) study, and group level effect sizes between baseline and post-treatment for participants in Comer’s et al. (2012)
SCED paper.
Key:
ODD- oppositional defiant disorder; SAD- separation anxiety disorder; ADHD- attention deficit hyperactive disorder; CD- conduct disorder; MDD- major
depressive disorder; GAD- generalised anxiety disorder; OCD-obsessive compulsive disorder.
Effect Size Descriptors (Cohen, 1988
Effect Size Descriptor
Small
Medium
Large
Cohen’s d
0.20
0.50
0.80
Doctorate in Educational and Child Psychology
Tamara Powell
3.4 Conclusions
Interventions that assign individual modules to problem specific behaviour
were found to be stronger than treatment that targets symptomatology within a
broader and widely applicable framework, such as traditional PCIT. For instance, in
CALM children face personally relevant fears in a continually graded format during
the brave behaviour module. The ED module in PCIT-ED similarly focuses on
concurrently improving parent and child emotional competence through exercises
that centre on emotional regulation and recognition. It is therefore not surprising that
interventions, such as PCIT, that rely solely on Child-Directed Interaction (CDI) and
Parent-Directed Interaction (PDI), consistently scored low on all weights of evidence.
In fact, there were no detrimental effect found from removing the PDI module all
together from treatment (Comer et al., 2012). This module may be more applicable
to externalising disorders for which it was originally intended or better situated within
designs that focus more extensively on the child’s underlying symptoms (e.g., PCITED).
The first pilot study of CALM showed promising results in reducing symptom
severity in children with a host of anxiety related disorders. However, a larger RCT
study is required prior to drawing any conclusions with regard to treatment efficacy.
A number of methodological constraints limit the extent to which results arising from
this study and others within the PCIT tradition can be generalised to other children
presenting similar issues. It is for these reasons, PCIT-ED is the only intervention
that is being recommended for use in a clinical pre-school population with MDD.
In practice, educational psychologists (EP) could work closely alongside other
agencies, such as CAMHS, to deliver, support and maintain progress achieved
through PCIT-ED. In schools, EPs could also liaise with staff to teach the
behavioural approaches provided to parents in sessions to ensure consistency
between the two settings. Finally, EPs may work in homes with parents to evaluate
short and long term gains of the programme and to troubleshoot issues as they
arise.
3.5 Limitations
The findings from this review should be interpreted in light of its limitations.
Firstly, the majority of reviewed studies lacked sufficient power to detect changes in
Doctorate in Educational and Child Psychology
Tamara Powell
treatment because of inadequate sample sizes. Secondly, all studies primarily rely
on parent reports, which carry many inherent biases. Although it was not appropriate
or feasible to obtain supplementary self-report data because of the child’s age,
observational data and teacher reports could have triangulated the obtained results.
Thirdly, experimental control was low across multiple baseline and one-group pretest
posttest study designs. Therefore, changes in symptom severity cannot be attributed
to treatment alone, as regression to the mean, maturation and or expectancy effects
could have influenced the outcome variable. Lastly, interventions took place in a
contrived and highly controlled setting (university clinic) and despite some small
scale efforts to generalise obtained results to the home environment (e.g., daily
homework assignments) ecological validity was low.
3.6 Recommendations for Future Research
Future research should examine the different dimensions of symptom severity
in greater detail. Specifically with respect to what items on the DSM-IV criteria are
the most and least resistant to treatment. An in depth analysis of symptoms and age
should also be examined in tandem, as there is some preliminary evidence to
suggest older children did not benefit from treatment to the same extent as younger
children (Comer et al., 2012; Lenze et al., 2011; Luby et al., 2012). Higher response
rates in children with SAD relative to other anxiety related disorders also signals a
strong need for research to focus on underrepresented subpopulations that have not
garnered much attention to date. Knowledge gained from this research can in turn
guide more targeted treatment efforts for children who present different diagnostic
profiles.
As a final note, it would be remiss not to further examine the individual
contributions of different programme components (e.g., CDI, PDI, ED, brave
behaviours), which were largely neglected in the current review. Future work should
evaluate which programme modules, and in which order and combination, have the
greatest influence in improving both short term and long term symptomatology.
Doctorate in Educational and Child Psychology
Tamara Powell
References
Allen, B., Timmer, S. G., & Urquiza, A. J. (2014). Parent–child interaction therapy as
an attachment-based intervention: Theoretical rationale and pilot data with
adopted children. Children and Youth Services Review, 47, 334-341.
American Psychiatric Association. (2000). Diagnostic and statistical manual of
mental disorders (4th ed., text rev.). Washington, DC: Author.
Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice Hall.
Becker, B. J. (1988). Synthesizing standardized mean change measures. British
Journal of Mathematical and Statistical Psychology, 41, 257-278.
.
Bell, S. K., & Eyberg, S. M. (2002). Parent-child interaction therapy: A dyadic
intervention for the treatment of young children with conduct problems.
Innovations in Clinical Practice: A Source Book, 20, 57-74.
Bowlby, J. (1978). Attachment theory and its therapeutic implications. Adolescent
Psychiatry, 6, 5-33.
Brinkmeyer, M. Y., & Eyberg, S. M. (2003). Parent-child interaction therapy for
oppositional children. In A. E. Kazdin (Eds.), Evidence-based psychotherapies
for children and adolescents (pp. 204-223). New York: The Guilford Press.
Cassidy, J., & Kobak, R. R. (1988). Avoidance and its relation to other defensive
processes. In J. Belsky, & T. Nezworski (Eds.), Clinical implications of
attachment (pp. 300-323). Hillsdale, NJ: Erlbaum.
Chase, R. M., & Eyberg, S. M. (2008). Clinical presentation and treatment outcome
for children with comorbid externalizing and internalizing symptoms. Journal of
Anxiety Disorders, 22(2), 273–82.
Choate, M. L., Pincus, D. B., Eyberg, S. M., & Barlow, D. H. (2005). Parent-child
interaction therapy for treatment of separation anxiety disorder in young
children : A pilot study. Cognitive and Behavioral Practice,12(1),126–135.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).
Hillsdale, NJ: Lawrence Earlbaum Associates.
Doctorate in Educational and Child Psychology
Tamara Powell
Comer, J. S., Puliafico, A. C., Aschenbrand, S. G., McKnight, K., Robin, J. A.,
Goldfine, M. E., & Albano, A. M. (2012). A pilot feasibility evaluation of the
CALM program for anxiety disorders in early childhood. Journal of Anxiety
Disorders, 26(1), 40–9.
Dahlke, F., Lohaus, A. & Gutzmann, H. (1992). Reliability and clinical concepts
underlying global judgments in dementia: Implications for clinical research.
Psychopharmacological Bulletin, 28, 425–432.
Department for Education and Skills (DfES). (2001). Promoting children’s mental
health within early years and school settings. London: DfES.
Department for Education. (2015). Mental health and behaviour in schools:
Departmental advice for school staff. London: DfE.
Egger, H., & Angold, A. (2006). Common emotional and behavioral disorders in
preschool children: Presentation, nosology, and epidemiology. Journal of
Child Psychology and Psychiatry, 47, 313–337.
Essex, M. J., Boyce, W. T., Goldstein, L. H., Armstrong, J. M., Kraemer, H. C., &
Kupfer, D. J. (2002). The confluence of mental, physical, social, and academic
difficulties in middle childhood. II: Developing the macarthur health and
behavior questionnaire. Journal of the American Academy of Child and
Adolescent Psychiatry, 41(5), 588–603.
Eyberg, S. M. (2005). Tailoring and adapting parent–child interaction therapy to new
populations. Education and Treatment of Children, 28(2),197–201.
Eyberg, S. M., & Members of the Child Study Laboratory. (1999). Parent-child
interaction therapy integrity checklists and session materials. Retrieved
October 20, 2005, from University of Florida, PCIT Laboratory Website:
http//www.pcit.org.
Gallagher, N. (2003). Effects of parent-child interaction therapy on young children
with disruptive behavior disorders. Bridges: Practice-Based Research
Syntheses, 1(4), 1–17.
Doctorate in Educational and Child Psychology
Tamara Powell
Gonzales, N. A., Pitts, S. C., Hill, N. E., & Roosa, M. W. (2000). A mediational model
of the impact of interparental conflict on child adjustment in a multiethnic, lowincome sample. Journal of Family Psychology, 14, 365–379.
Guyatt, G. H., Sackett, D. L., Sinclair, J. C., Hayward, R., Cook, D. J., Cook, R. J.,
Bass, E., Gerstein, H., Haynes, B., Holbrook, A., Jaeschke, R., Laupacls, A.,
Moyer, V., & Wilson, M. (1995). Users' guides to the medical literature: IX. A
method for grading health care recommendations. Journal of the American
Medical Association, 274(22), 1800-1804.
Harden, A. & Gough, D., 2012. Quality and relevance appraisal. In D. Gough, S.
Oliver & J. Thomas (Eds.), An introduction to systematic reviews (pp. 153178). London: Sage.
Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and
related estimators. Journal of Educational Statistics, 6,107–128.
Hembree-Kigin, T. L., & McNeil, C. B. (1995). Parent-child interaction therapy. New
York: Plenum Press.
Herschell, A. D., Calzada, E. J., Eyberg, S. M., & McNeil, C. B. (2002). Clinical
issues in parent-child interaction therapy. Cognitive and Behavioral Practice,
9, 16-27.
Horner, R., Carr, E., Halle, J., Mcgee, G., Odom, S., & Wolery, M. (2005). The use of
single-subject research to identify evidence-based practice in special
education. Exceptional Children, 71(2), 165-179.
Jacobson, N.S., Follette, W.C., & Revenstorf, D. (1984). Psychotherapy outcome
research: Methods for reporting variability and evaluating clinical significance.
Behavior Therapy, 15(4), 336-352.
Kendall, P. C., Furr, J. M., & Podell, J. L. (2010). Child-focused treatment of anxiety.
In J. R. Weisz A.E. Kazdin (Eds.), Evidence-based psychotherapies for
children and adolescents second edition (pp. 45-60). New York, NY: Guilford
Press.
Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L.
T., Walters, E. E. & Zaslavsky, A. M. (2002). Short screening scales to
Doctorate in Educational and Child Psychology
Tamara Powell
monitor population prevalences and trends in non-specific psychological
distress. Psychological Medicine, 32, 956-976.
Kratochwill, T. R. (2003). Evidence-Based Practice: Promoting Evidence-Based
Interventions in School Psychology. School Psychology Quarterly, 18(4), 389408.
Ladd, G. W., Birch, S., & Buhs, E. (1999). Children’s social and scholastic lives in
kindergarten: Related spheres of influence? Child Development, 70, 1373–
1400.
Lavigne, J. V., Arend, R., Rosenbaum, D, Sinacore, J., Cincchetti, C., Binns, H. J.,
Kaufer Christoffel, K., Hayford, J. R., & McGuire, P. (1994), Interrater
reliability of the DSM-III-R with preschool children. Journal of Abnormal Child
Psychology, 22, 679-690.
Lenze, S. N., Pautsch, J., & Luby, J. L. (2011). Parent-child interaction therapy
emotion development: a novel treatment for depression in preschool children.
Depression and Anxiety, 28(2), 153–9.
Luby, J. L. (2013). Treatment of anxiety and depression in the preschool period.
Journal of the American Academy of Child and Adolescent Psychiatry, 52(4),
346-358.
Luby, J. L., Lenze, S. N., & Tillman, R. (2012). A novel early intervention for
preschool depression: Findings from a pilot randomized controlled trial.
Journal of Child Psychology and Psychiatry and Allied Disciplines, 53(3), 313–
22
Normandeau, S., & Guay, F. (1998). Preschool behavior and first-grade school
achievement: The mediational role of cognitive self-control. Journal of
Educational Psychology, 90, 111–121.
Morris, S. B. (2007). Estimating effect sizes from the pretest-posttest-control group
designs. Organizational Research Methods, 11, 364-386.
Patterson, G. R., Capaldi, D., & Bank, L. (1991). An early starter model for predicting
delinquency. Paper presented at the Earlscourt Conference on Childhood
Aggression, Toronto, Canada, Jun 1988. Lawrence Erlbaum Associates, Inc.
Doctorate in Educational and Child Psychology
Tamara Powell
Pincus, D. B., Santucci, L. C., Ehrenreich, J. T., & Eyberg, S. (2008). The
implementation of modified parent-child interaction therapy for youth with
separation anxiety disorder. Cognitive and Behavioural Practice, 15(2), 118125.
Puliafico, A. C., Comer, J. S., & Albano, A. M. (2013). Coaching approach behavior
and leading by modeling: Rationale, principles, and a session-by-session
description of the CALM program for early childhood anxiety. Cognitive and
Behavioral Practice, 20(4), 517-528.
Shaffer, D., Fisher, P., Lucas, C.P., Dulcan, M. K., & Schwab-Stone, M. E. (2000).
NIMH diagnostic interview schedule for children version IV (NIMH DISC-IV):
Description, differences from previous versions, and reliability of some
common diagnoses. Journal of American Academy Child Psychiatry, 39, 28–
38.
Silverman, W. K., & Albano, A. M. (1996). The Anxiety Disorders Interview Schedule
for DSM- IV-Child and Parent Versions. London: Oxford University Press.
Silverman, W. K., & Nelles, W. B. (1988). The anxiety disorders interview schedule
for children. Journal of the American Academy of Child and Adolescent
Psychiatry, 7, 772-778.
Silverman, W. K., & Ollendick, T. H. (2005). Evidence-based assessment of anxiety
and its disorders in children and adolescents. Journal of Clinical Child and
Adolescent Psychology, 34, 380–411.
Silverman, W. K., Pina, A. A., & Viswesvaran, C. (2008). Evidence-based
psychosocial treatments for phobic and anxiety disorders in children and
adolescents. Journal of Clinical Child and Adolescent Psychology, 37, 105130.
Silverman, W. K., Saavedra, L. M., & Pina, A. A. (2001). Test–retest reliability of
anxiety symptoms and diagnoses with the anxiety disorders interview
schedule for DSM-IV: Child and parent versions. Journal of the American
Academy of Child and Adolescent Psychiatry, 40, 937–944.
Doctorate in Educational and Child Psychology
Tamara Powell
Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. New
York: Appleton-Century.
Stormont, M. (2002). Externalizing behavior problems in young children: Contributing
factors and early intervention. Psychology in the Schools, 39, 127-138.
Timmer, S. G., Ho, L. K. L., Urquiza, A. J., Zebell, N. M., Garcia, E. F., & Boys, D.
(2011). The effectiveness of parent–child interaction therapy with depressive
mothers: The changing relationship as the agent of individual change. Child
Psychiatry and Human Development, 42,406–423.
Wagner, K.D. (2005). Pharmacotherapy for major depression in children and
adolescents. Progress in Neuro-Psychopharmacology and Biological
Psychiatry, 29, 819–826.
West, J., Denton, K., & Reaney, L. M. (2001). The kindergarten year: Findings from
the early childhood longitudinal study, kindergarten class of 1998–1999
(Publication No. NCES2001-023). Washington, DC: Department of Education,
National Center for Education Statistics.
Whaley, S.E., Pinto, A., & Sigman, M. (1999). Characterizing interactions between
anxious mothers and their children. Journal of Consulting and Clinical
Psychology, 67, 826-836.
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 1
Inclusion and Exclusion Criteria
Inclusion Criteria
1. Publication Type
a) Full length papers in
a peer-reviewed
journal.
Exclusion Criteria
Rational
Material not in a peer reviewed
journal (e.g., abstracts, book
chapters, conference
proceedings, dissertations).
The peer review process
ensures a higher level of
methodological rigour.
b) Studies containing
primary empirical data.
Studies that do not contain
primary empirical data (e.g.,
literature reviews, metaanalysis and opinion papers).
To ensure originality of findings.
c) Studies written in
English.
Studies written in a language
other than English.
Translation services were not
available.
2. Participants and
Setting
a) Children between
the ages of 3 and 8
years.
Infants younger than 3 years of
age, and children older than 8
years of age.
There is empirical evidence to
support a clinical depression
syndrome in children as young
as 3 years of age (Egger &
Angold, 2006; Luby et al., 2013).
Similarly, various forms of
childhood anxiety begin to
emerge after 2.5 years of age
(APA, 2000). This cohort of
young children also require
different types of treatment than
older children with higher
cognitive abilities.
b) Children with
internalising disorders,
including anxiety and
other mood disorders.
-Children with major medical or
neurological disease;
pervasive developmental
disorder; who have been
adopted or subjected to and/or
at risk of abuse.
Participants exclusively
presenting externalising
behaviour problems in
absence of any internalising
disorders.
-Family discord can place
children at a heightened risk for
an attachment disorder and
cause socioemotional delays
that could impact the efficacy of
treatment. -Externalising
behaviour problems are often
present in children with
internalising disorders, however,
the former group represents a
distinct population with different
clinical needs.
-Children with medical issues
and/or developmental delays
may have neurobiological
differences from the target
population.
c) Children not
undergoing concurrent
intervention(s).
Children engaged in more than
one form of psychotherapy or
on unstable doses of
psychotropic medication.
These confounding variables
can influence the effect
treatment has on symptom
severity.
Doctorate in Educational and Child Psychology
Inclusion Criteria
3. Dependent
Variables
The study measures
the efficacy of PCIT in
relation to the child’s
internalising
symptoms.
4. Analysis
The study reports
means, standard
deviations, test
statistic(s) and/or data
that allows for these
measures to be
computed.
5. Year of Publication
Study was published
after 2005.
Exclusion Criteria
The study does not measure
the efficacy of PCIT in relation
to the child’s internalising
symptoms.
Tamara Powell
Rational
Ensures papers directly relates
to the research question at
hand.
The study does not report data
that allows for means,
standard deviations and/or test
statistic(s) to be computed.
These measures are required in
the computation of effect size.
Study was published prior to
2005.
PCIT adaptions for internalising
disorders were published in
2005 (Eyberg, 2005).
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 2
Excluded Articles from PsycINFO, ERIC and MEDLINE
1
Studies
Carpenter, A. L., Puliafico, A. C., Kurtz, S. M. S.,
Pincus, D. B., & Comer, J. S. (2014). Extending
parent–child interaction therapy for early childhood
internalizing problems: New advances for an
overlooked population. Clinical Child and Family
Psychology Review, 17(4), 340-356.
Rational for Exclusion
Criteria 1b – Paper
does not report
primary empirical
data.
2
Kohlhoff, J., & Morgan, S. (2014). Parent-child
Criteria 2b –
interaction therapy for toddlers: A pilot study. Child & Participants have
Family Behavior Therapy, 36(2), 121–139.
been selected on the
basis of externalising
behaviour problems.
3
Pincus, D. B., Eyberg, S. M., & Choate, M. L.
(2005). Adapting parent-child interaction therapy for
young children with separation anxiety disorder,
Education and Treatment of Children, 28(2), 163–
181.
Criteria 1b – Paper
does not report
primary empirical
data.
4
Puliafico, A. C., Comer, J. S., & Albano, A. M.
(2013). Coaching approach behavior and leading by
modeling: Rationale, principles, and a session-bysession description of the CALM program for early
childhood anxiety. Cognitive and Behavioral
Practice, 20(4), 517–528.
Criteria 1b – Paper
does not report
primary empirical
data.
5
Puliafico, A. C., Comer, J. S., & Pincus, D. B.
(2012). Adapting parent-child interaction therapy to
treat anxiety disorders in young children. Child and
Adolescent Psychiatric Clinics of North America,
21(2), 607-619.
Criteria 1b – Paper
does not report
primary empirical
data.
6
Timmer, S. G., Ho, L. K. L., Urquiza, A. J., Zebell,
N. M., Garcia, E. F., & Boys, D. (2011). The
effectiveness of parent–child interaction therapy with
depressive mothers: The changing relationship as
the agent of individual change. Child Psychiatry and
Human Development, 42(4), 406-423.
Criteria 2b –
Participants have
been selected on the
basis of externalising
behaviour problems.
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 3
Weighting of Studies
1. Weight of Evidence A: Methodological Quality
A. Group Based Design: Randomised Control Studies and Quasi Experimental Designs
Key Features
Measurement
High Rating (3)
Instruments produce a reliability of
0.85 or higher and data is collected
using multiple methods and from
multiple sources for all primary
outcomes.
Medium Rating (2)
Instruments produce a reliability
of 0.70 or higher and data is
collected either using multiple
methods and/or from multiple
sources for at least 75% of the
primary outcomes.
Low Rating (1)
Instruments produce a
reliability of 0.50 or higher for
at least 50% of the primary
outcomes.
No Rating (0)
Instruments produce
reliability lower than
0.50 and/or data was
not collected using
multiple methods
and/or from multiple
sources.
Comparison
Group
Inclusion of at least one “active”
comparison group, group
equivalency, counterbalanced
change agents, equivalent mortality
and low attrition at post-treatment.
Inclusion of a “no intervention
group” comparison group, as well
as at least two of the following
criteria:
1) Counterbalancing change
agents
2) Group equivalence
3) Equivalent mortality with low
attrition (Intent to Intervene must
have been conducted in cases
where this is not demonstrated).
No efforts were made
to ensure group
equivalence.
Primary
Outcomes
Appropriate units of analyses used,
familywise/ experiment-wise error
rate (if applicable) is controlled and
there is a sufficiently large N. Also,
significant outcomes must be shown
for at least 75% of the total primary
outcome measures for each key
construct, with moderate effect sizes.
Appropriate units of analyses
used, familywise/ experimentwise
error rate (if applicable) is
controlled and there is a
sufficiently large N. Also,
significant outcomes must be
shown for at least 50-75% of the
total primary outcome measures
for each key construct.
At least one of the following:
1) Counterbalancing change
agents
2) Group equivalence
3) Equivalent mortality with low
attrition (Intent to Intervene
must have been conducted in
cases where this is not
demonstrated).
OR a comparison group for
studies with no group
equivalence procedures.
Appropriate units of analyses
used and familywise/
experimentwise error rate (if
applicable) is controlled. Also,
significant outcomes must be
shown for at least 25-49% of
the total primary outcome
measures for any key
construct.
None of the outlined
criteria were met.
Doctorate in Educational and Child Psychology
Tamara Powell
Key Features
Educational/
Clinical
Significance
High Rating (3)
At least 3 of the 4 criteria is met at
post-treatment or follow-up:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
Medium Rating (2)
At least 2 of the 4 criteria is met
during post or follow-up:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
Low Rating (1)
At least 1 of the 4 criteria is
met during post or follow-up:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
No Rating (0)
None of the outlined
criteria were met.
Identifiable
Components
Study shows strong evidence (score
= 3) for significant primary outcomes,
uses a design that isolates specific
programme components and an
analyses that demonstrates all
components were necessary to
produce change in the primary
outcome variable.
Study shows promising evidence
(score = 2) for significant primary
outcomes, uses a design that
isolates specific programme
components and an analyses that
demonstrates at least 50% of the
components were necessary to
produce change in the primary
outcome variable.
Study shows weak evidence
(score = 1) for significant
primary outcomes, use a
design that isolates specific
programme components and
an analyses that demonstrates
at least 25% of the
components were necessary
to produce change in the
primary outcomes.
No evidence of which
programme
components were
necessary to produce
change.
Implementation
Fidelity
At least 2 of the 3 criteria is met:
1) Ongoing supervision/consultation
2) Coding sessions
3) Audio/video tapes
AND the use of a manual.
If any adaptations are made they
should be described in detail.
At least 1 of the 3 criteria is met:
1) Ongoing
supervision/consultation
2) Coding sessions
3) Audio/video tapes
AND the use of a manual.
At least 1 of the 3 criteria is
met:
1) Ongoing
supervision/consultation
2) Coding sessions
3) Audio/video tapes
OR the use of a manual.
No efforts were made
to ensure
implementation fidelity.
Doctorate in Educational and Child Psychology
Tamara Powell
B. Single-Participant Design: Multiple Baseline Design Across Participants
Key Features
Measurement
High Rating (3)
For primary outcomes, instruments
produce a reliability of 0.85 or
higher, data is collected using
multiple methods and from multiple
sources and a case for the validity
is made.
Medium Rating (2)
For primary outcomes,
instruments produce a reliability of
0.85 or higher and data is
collected using multiple methods
and from multiple sources.
Low Rating (1)
For primary outcomes,
instruments produce a
reliability of 0.70 or higher.
No Rating (0)
Instruments produce
reliability lower than
0.70 and/or data
was not collected
using multiple
methods and/or from
multiple sources.
Overall Rating for
Quality of Baseline
At least 4 of the 5 criteria is met for
each primary outcome:
1) Minimum of 3 baseline points
2) Stability of scores across
baseline phase(s)
3) Baseline and intervention scores
do not overlap
4) Behaviour problem(s) is evident
5) Dependent variable does not
systematically increase or decrease
At least 3 of the 5 criteria is met
for each primary outcome:
1) Minimum of 3 baseline points
2) Stability of scores across
baseline phase(s)
3) Baseline and intervention
scores do not overlap
4) Behaviour problem(s) is evident
5) Dependent variable does not
systematically increase or
decrease
At least 2 of the 5 criteria is
met for each primary outcome:
1) Minimum of 3 baseline
points
2) Stability of scores across
baseline phase(s)
3) Baseline and intervention
scores do not overlap
4) Behaviour problem(s) is
evident
5) Dependent variable does
not systematically increase or
decrease
No evidence or
unknown/insufficient
data reported.
Visual analysis shows a large
change in behaviour during the
intervention stage, AND 3 of the 4
following criteria:
1) No-to-minimal overlap between
the baseline and intervention
phases
2) Clear trend reversal
3) Adequate phase length
4) Stable data
Visual analysis shows a moderate
to large change in behaviour
during the intervention stage, AND
2 of the 4 following criteria:
1) No-to-minimal overlap between
the baseline and intervention
phases
2) Clear trend reversal
3) Adequate phase length
4) Stable data
Visual analysis shows a
moderate change in behaviour
during the intervention stage,
AND 1 of the 4 following
criteria:
1) No-to-minimal overlap
between the baseline and
intervention phases
2) Clear trend reversal
3) Adequate phase length
4) Stable data
No change in
behaviour and either
one or none of the
listed criteria were
met.
Mean Rating Across
Participants
Overall Rating for
Primary Outcomes;
Mean Rating Across
Participants
Doctorate in Educational and Child Psychology
Tamara Powell
Key Features
Educational/ Clinical
Significance
High Rating (3)
At least 3 of the 4 criteria is met
during post-treatment or follow-up:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
Medium Rating (2)
At least 2 of the 4 criteria is met
during post- treatment or followup:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
Low Rating (1)
At least 1 of the 4 criteria is
met during post- treatment or
follow-up:
1) Categorical diagnosis data
2) Outcomes assessed by
continuous variables
3) Subjective evaluation
4) Social comparison
No Rating (0)
None of the outlined
criteria were met.
Identifiable
Components
Study shows strong evidence
(score = 3) for significant primary
outcomes, uses a design that
isolates specific programme
components and an analyses that
demonstrates all components were
necessary to produce change in the
primary outcome variable.
Study shows promising evidence
(score = 2) for significant primary
outcomes, uses a design that
isolates specific programme
components and an analyses that
demonstrates at least 50% of the
components were necessary to
produce change in the primary
outcome variable.
Study shows weak evidence
(score = 1) for significant
primary outcomes, use a
design that isolates specific
programme components and
an analyses that
demonstrates at least 25% of
the components were
necessary to produce change
in the primary outcome
variable.
Components were
not provided or none
of the criteria was
met.
Overall Rating for
Fidelity
At least 2 of the 3 criteria is met:
1) Ongoing
supervision/consultation
2) Coding sessions
3) Audio/video tapes
AND the use of a manual.
If any adaptations are made they
should be described in detail.
At least 1 of the 3 criteria is met:
1) Ongoing
supervision/consultation
2) Coding sessions
3) Audio/video tapes
AND the use of a manual.
At least 1 of the 3 criteria is
met:
1) Ongoing
supervision/consultation
2) Coding sessions
3) Audio/video tapes
OR the use of a manual.
No efforts were
made to ensure
implementation
fidelity.
Mean Rating Across
Participants
Doctorate in Educational and Child Psychology
Tamara Powell
Overall Methodological Quality Score
Overall methodological quality was computed by adding the weight of evidence from the aforementioned variables (e.g., assigning
3 for “high”, 2 for “medium”, 1 for “low” and 0 for “no evidence”) and dividing by a sum of six (or less in cases with no numerical
rating). Averaged scores equate to the following ranking:
Overall Methodological Quality
(WoED)
High
Medium
Low
Averaged Scores
2.5 or higher
1.5 to 2.4
1.4 or less
A. Group Based Design: Randomised Control Studies and Quasi Experimental Designs
Study
Chase &
Eyberg
(2008)
Lenze et
al. (2011)
Luby et
al. (2012)
Pincus
et al.
(2008)
Measures
Comparison
Group
Primary
Outcomes
Educational/
Clinical
Significance
Identifiable
Components
Fidelity
1.0
NNR
3.0
2.0
0.0
3.0
2.0
NNR
1.0
3.0
0.0
3.0
2.0
2.0
1.0
2.0
0.0
3.0
2.0
NNR
NNR
1.0
0.0
1.0
Overall
Quality
Med
1.8
Med
1.8
Med
1.7
Low
1.0
Doctorate in Educational and Child Psychology
Tamara Powell
A. Single-Participant Design: Multiple Baseline Design Across Participants
Study
Choate et
al. (2005)
Comer et
al. (2012)
Measures
Quality of
Baseline
Primary
Outcomes
Educational/
Clinical
Significance
Identifiable
Fidelity
Components
1.0
0.0
0.0
2.0
0.0
2.0
1.0
0.0
NNR
2.0
0.0
3.0
Overall
Quality
Low
0.8
Low
1.2
Key: NNR- no numerical value
2. Weight of Evidence B: Methodological Relevance
Weighting
Description
High Rating (3)
GEDs: Random assignment of participants to a control or experimental group to establish group
equivalence. Study also compares programme outcomes with an “active” comparison group pre and
post treatment.
SCEDs: High level of experimental control (more than three demonstrations of the experimental
effect and controls for common threats to validity) and strong baseline data (minimum of five
measurements).
Medium Rating (2)
Low Rating (1)
GEDs: Non-randomised group design with a “no intervention” comparison group.
SCEDs: Adequate experimental control (three demonstrations of the experimental effect and
controls for common threats to validity) and baseline data (minimum of three measurements).
GED: Quasi experimental designs with no control group. Data is obtained pre and post treatment
across primary outcome variables.
SCED: Weak experimental control (less than three demonstrations of the experimental effect and
some between subject comparisons) and inadequate baseline data (less than three measurements).
Doctorate in Educational and Child Psychology
Tamara Powell
Rational: Methodological relevance was weighted in relation to the guidelines for single case studies (Homer et al., 2005) and the
hierarchies of evidence proposed by Guyatt et al. (1995).
GED: Hierarchies of evidence (Guyatt et al., 1995) classify different study designs in relation to methodological rigour, which in turn
guide the strength of recommendations for clinical practice. This framework is accordingly well suited for clinical decisions, such as,
whether PCIT and its neighbouring protocols improve symptom severity in young children with anxiety and depression.
SCED: According to Homer and colleagues (2005), threats to internal validity and experimental control are amongst the most important
variables to consider when assessing SCED. A functional relationship between manipulations to the independent variable (PCIT) and
changes in the dependent variable (symptom severity) can only be inferred once studies have collected an adequate amount of data
across both the baseline and intervention time periods (within participant control). In line with Homer’s et al. (2005) recommendations,
studies must demonstrate at least three experimental effects and three baseline points in order to achieve a medium rating.
3. Weight of Evidence C: Relevance to the Research Question
Weighting
High Rating (3)
Medium Rating (2)
Low Rating (1)
Description
Multiple methods (more than 2 measures) are used to assess symptom severity in a child
population with a primary diagnoses of anxiety or depression. PCIT is administered by a
trained therapist, and treatment is carried out for at least 12 sessions.
Multiple methods (at least 2 measures) are used to assess symptom severity in children
with a primary diagnosis of anxiety or depression. PCIT is administered by a trained
therapist and treatment is carried out for at least 9 sessions.
A single method is used to assess symptom severity in a sample of children with a comorbid diagnoses of anxiety or depression. PCIT is administered by a trained professional
and treatment is carried out for less than 9 sessions.
Doctorate in Educational and Child Psychology
Tamara Powell
Rational: Number of Assessment Measures: Kratochwill (2003) recommends using at least two measures to assess the primary outcome
variable. Due to the young age of participants who may not to be registered in school or be able to voice their own opinions, only multimethod rather than multi-sourced data were considered. Further, symptom severity is a multidimensional construct that is difficult to
measure. Thus, reliability and validity of findings are enhanced if several outcome measures are examined simultaneously. Diagnosis:
Children presenting a principle diagnosis of anxiety or depression will show more severe symptomatology than children with a comorbid
diagnosis. A number of confounding variables will also be introduced to the analyses if the child’s primary diagnoses falls outside the
internalising domain. Programme Length: PCIT manual recommends that treatment be carried out for at least 9 sessions, with CDI lasting
between 4 and 7 sessions and PDI 5 to 7 sessions (Hembree-Kigin & McNeil, 1995). Treatment Administrator: Therapists with clinical
experience and expertise (graduate level) will be more adept in understanding the unique needs of clients relative to trained professionals
will minimal or no clinical experience.
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 4
Justification for Weights of Evidence
Study
Chase &
Eyberg
(2008)
Methodological Quality
(WoE A)
1.0 – Measures
NNR – Comparison Group
3.0 – Primary Outcomes
2.0 – Edu/Clinical Significance
0.0 – Identifiable Components
3.0 – Fidelity
1.8 = Medium
Methodological Relevance
(WoE B)
Quasi experimental design with
no control group. Data was
obtained pre and post treatment
for primary outcome variable.
Relevance of Evidence to the
Review Question (WoE C)
Overall
Weight of
Evidence
(WoE D)
DISC-IV-P was the only measure used to
assess symptom severity.
Participants had a comorbid diagnosis of
ODD and SAD.
1.0 = Low
1.3 = Low
Therapists’ credentials were not specified.
Treatment lasted for 14 sessions.
Choate et
al. (2005)
1.0 – Measures
0.0 – Quality of Baseline
0.0 – Primary Outcomes
2.0 – Edu/Clinical Significance
0.0 – Identifiable Components
2.0 – Fidelity
0.8 = Low
Weak experimental control (one
data point at posttest and followup) and inadequate baseline data
(single measurement).
1.0 = Low
1.0 = Low
ADIS-C/P was the only measure used to
assess symptom severity.
Children’s primary diagnosis was SAD.
Lead therapist was a professor with 2
years of training on PCIT.
Treatment was carried out for 6 to 7
sessions.
1.0 = Low
0.9 = Low
Doctorate in Educational and Child Psychology
Study
Comer et
al. (2012)
Methodological Quality
(WoE A)
1.0 – Measures
0.0 – Quality of Baseline
NNR – Primary Outcomes
2.0 – Edu/Clinical Significance
0.0 – Identifiable Components
3.0 – Fidelity
1.2 = Low
Tamara Powell
Methodological Relevance
(WoE B)
Relevance of Evidence to the
Review Question (WoE C)
Weak experimental control (group
means reported at posstest only)
and inadequate baseline data
(two measurements).
Symptom severity was assessed by:
ADIS-C/P, CGAS and CGI.
Subjects met diagnostic criteria for a
principle anxiety disorder.
1.0 = Low
CALM was administered by five Ph.D level
therapists who had clinical experience
working with children with anxiety related
disorders.
Overall
Weight of
Evidence
(WoE D)
1.7 = Medium
Treatment was conducted over 12
sessions.
Lenze et
al. (2011)
2.0 – Measures
NNR – Comparison Group
1.0 – Primary Outcomes
3.0 – Edu/Clinical Significance
0.0 – Identifiable Components
3.0 – Fidelity
1.8 = Medium
Quasi experimental design with
no comparison group. Data was
obtained pre and post treatment
for primary outcome variable.
1.0 = Low
3.0 = High
PAPA MDD severity score and HBQ-P
internalising subscale were used to assess
symptom severity.
All children had an MDD principle
diagnosis.
Three graduate level therapists
administered PCIT-ED.
Treatment was completed over 14
sessions.
3.0 = Medium
1.6 = Medium
Doctorate in Educational and Child Psychology
Tamara Powell
Study
Methodological Quality
(WoE A)
Methodological Relevance
(WoE B)
Luby et al.
(2012)
2.0– Measures
2.0 – Comparison Group
1.0 – Primary Outcomes
2.0 – Edu/Clinical Significance
0.0 – Identifiable Components
3.0 – Fidelity
1.7 = Medium
Participants were randomly
assigned to PCIT-ED
(experimental group) or a didactic
training (active control group)
condition.
Relevance of Evidence to the
Review Question (WoE C)
Overall
Weight of
Evidence
(WoE D)
PAPA MDD severity score, PFC-S and
HBQ-P internalising subscale were used
to assess symptom severity.
All children had an MDD principle
diagnosis.
3.0 = High
2.6 = High
Five graduate level therapists
administered treatment.
Treatment was conducted over 12 weeks
(14 sessions).
Pincus et
al. (2008)
2.0– Measures
NNR – Comparison Group
NNR – Primary Outcomes
1.0 – Edu/Clinical Significance
0.0 – Identifiable Components
1.0 – Fidelity
1.0 = Low
Quasi experimental design with
no comparison group. Data was
obtained pre and post treatment
for primary outcome variable.
3.0 = High
ADIS-C/P was the only instrument used to
assess symptom severity.
Participants met diagnostic criteria for
SAD.
1.0 = Low
1.0 = Low
No information was provided on therapists’
credentials.
Length of treatment was not reported.
1.0 = Low
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 5
Summary of Reviewed Studies
Study Aims
Intervention
PCIT
To examine
treatment effects
of PCIT in
children with
comorbid SAD
and ODD in
relation to
children with
ODD only on
measures of
internalising and
externalising
symptomatology.
Therapists adhered to
the PCIT principles
outlined in the Eyberg
(1999) treatment manual
for the CDI and PDI
phases of treatment.
Once proficiency was
achieved in CDI, parents
moved onto PDI which
interleaved aspects of
the initial CDI training.
Treatment was
completed when CDI
and PDI skills were
mastered and when
parents reported their
child’s behaviour to be
within ½ SD of the
normative mean for
ECBI.
Families participated in
a mean of 14.05
sessions (SD = 7.2)
over the span of 18.9
weeks (SD = 11.4).
Design
Chase & Eyberg (2008)
Participants
One group
pretest
posttest
design
Administrator n = 1
SAD severity
scores are
derived from
the DISC-IV-P
interview with
the child’s
female
caregiver
before and
after
treatment.
Results are
examined
across time
and group
(SAD+ODD
versus ODD
only).
Parents n = 1-2 per participant
One lead therapist coached parents via a bug in the
ear microphone.
At least one caregiver was required to participant in
the study, however, the sex of the parent and
parent-child ratio was not reported.
Children n = 64
Participants were part of a larger study examining
maintenance of treatment following standard PCIT.
To be included, participants required an ODD
diagnosis and both the child and parent needed to
demonstrate normative cognitive abilities, as
assessed by the PPVT and WPT (minimum St.
score of 75). Children with neurological impairments
or in out of home placements were excluded.
SAD+ODD group included 15 children, and the
ODD only group 49 children between the ages of 3
and 6 years (M = 4.48; SD = 1.13; 22F, 42M).
Children in the SAD+ODD group did not differ from
the ODD group on any demographic variables.
Measures
(relevant to
review)
DISC-IV-P
DISC-IV-P is a
structured
psychiatric
diagnostic
interview
administered to
parents to
assess over 30
psychiatric
diagnoses that
occur in
childhood and
adolescents,
including SAD
and ODD.
Diagnoses are
based on the
DSM-IV criteria
and the
interview is
conducted by a
trained graduate
research
assistant before
and after
treatment.
Outcomes
(relevant to
review)
Two way repeated
measures ANOVA
compared SAD
symptoms in the
SAD+ODD group to
the ODD only
group. Results
showed a
significant group x
time interaction
(F (1, 55) = 18.93,
p < .001, Cohen’s d
= 1.48), with a
significant reduction
in SAD symptoms
for the SAD+ODD
group between pre
and post treatment
(t (13) = 7.27, p
<.001). The
majority of
participants (11 out
of 15; 75%)
diagnosed with
SAD at pretreatment no longer
met diagnostic
criteria.
Doctorate in Educational and Child Psychology
Study Aims
This pilot
study aimed
to describe
the effects
PCIT has on
the treatment
of SAD. SAD
behaviours,
clinical
symptoms,
and
oppositional
behaviour
was
examined.
Intervention
Design
PCIT
PCIT followed the
standard procedures
outlined by Herschell et
al. (2002), which includes
the CDI and PDI phases.
Parents only proceeded
onto the PDI phase once
mastery was
demonstrated in the CDI
phase. The mastery
criteria required parents to
use no more than 3
questions, commands, or
criticisms, and to use at
least 10 behavioural
descriptions, 10
reflections and 10 labelled
praises. Parents received
a teaching session during
the PDI phase, followed
by 2-3 sessions where
they implemented the PDI
procedures. The
intervention terminated
when parents
demonstrated effective
CDI and PDI skills, and
when their child’s
separation anxiety
decreased to less than 2
incidents per week.
Multiplebaseline
experimental
design
PCIT ranged from 6 to 7
sessions for each family.
Start times
were
staggered
across the
three families,
with treatment
beginning after
1, 2 and 4
weeks
respectively.
Due to
logistical
constraints,
follow up
interviews
were
conducted at 3
(2 families)
and 6 months
(1 family).
Tamara Powell
Choate et al. (2005)
Participants
Measures
(relevant to review)
Administrator n = 2
ADIS-C/P
The lead therapist was a professor with 2
years of training on the administration of
PCIT, and the co-therapist was a second
year doctoral student.
Semi-structured
diagnostic interview
designed to specifically
assess child and
adolescent anxiety and
accompanying mood
disorders, in
accordance with the
DSM-IV criteria. A
CSR (ranging between
0 and 8) of 4 or greater
indicates a clinical
diagnosis of anxiety.
Parents n = 6
Both parents participated in all facets of
treatment. All three families had annual
incomes that exceeded $75,000 and were of
Caucasian decent.
Children n = 3
All participants were recruited through the
Child and Adolescent Fear and Anxiety
Programme at the Centre for Anxiety and
Related Disorders in Boston, Massachusetts.
Children between the ages of 4 and 8 years,
who were on stable doses of psychotropic
medication, not at risk of harming themselves
or others and whose primary diagnoses was
SAD were invited to participate in the study.
The sample included:
1) 5 year old male; principle diagnosis SAD
and an additional provisional diagnoses of
panic disorder
2) 8 year old female; principle diagnosis SAD
and an additional diagnosis of ODD.
3) 7 year old male; principle diagnosis SAD
ADIS-C/P is not
standardised for
children younger than
7 years (Silverman &
Nelles, 1988),
therefore the measure
was not administered
to the 5 year old
participant.
Outcomes
(relevant to
review)
Descriptive
statistics were
carried out for the 3
participants at pretreatment, posttreatment and 3 or
6 month follow up
time points. Sub
clinical levels of
SAD were found at
post-treatment and
maintained during
the follow up period
for all participants.
Effect sizes at
follow-up were
1.29, 1.71 and
0.86 for participant
1, 2 and 3,
respectively.
Doctorate in Educational and Child Psychology
Study Aims
Intervention
CALM
To
investigate
the
preliminary
feasibility
and efficacy
of a modified
version of
PCIT to treat
youth with
SAD, social
anxiety
disorder,
GAD, and/or
specific
phobias.
CALM follows the
framework outlined in
PCIT, including live and
individualised coaching
using unobtrusive
measures. It also includes
a CDI phase that focuses
on strengthening the
parent-child relationship.
Unlike PCIT however,
there are exposure
sessions whereby parents
teach children brave
behaviours through
increasingly challenging
tasks.
CALM is carried out for 12
sessions (4 devoted to
CDI and 8 to brave
behaviours) and the
average completing time
was 10.3 sessions (2 drop
outs completed 4.5
sessions; completers took
part in 12 sessions).
Design
Multiplebaseline
experimental
design
The primary
measure was
administered
across to two
baseline
assessments,
then again at
post-treatment.
The length of
time between
Baseline 1 and
Baseline 2 was
randomly
determined
(ranged
between 1 and
4 weeks)
The ADIS-P
(parent
version) was
completed by
an IE who was
blind to all
treatment
related data.
Tamara Powell
Comer et al. (2012)
Participants
Measures
(relevant to review)
Administrator n = 5
ADIS-C/P
Five Ph.D level clinical psychologists with
specialised knowledge in child anxiety
disorders administered this manual-based
adaptation of PCIT. Each therapist
underwent 4 months of training on CALM,
and received video-review supervision
through the principle investigator.
Semi-structured
diagnostic interview
designed to specifically
assess child and
adolescent anxiety and
accompanying mood
disorders, in
accordance with the
DSM-IV criteria. A
CSR (ranging between
0 and 8) of 4 or greater
indicates a clinical
diagnosis of anxiety.
Parents n = 17
With the exception of one participant, whose
mother only took part, both parents
completed the intervention with their child.
Children n = 9
Children were recruited through community
outreach efforts (e.g., mailings to
paediatricians and media advertisements).
Interested families were then screened to
ensure they met the study inclusion criteria
(child between the ages of 3 to 8 years;
principle diagnosis is an anxiety disorder;
English speaking).
Participants were from racially and ethnically
diverse backgrounds, ranging in age from 4
to 8 years (M = 5.4, SD = 1.3; 6F, 3M). At
intake, 66% of children met criteria for a
principle or co-principle diagnosis of SAD,
33% for a principle or co-principle diagnosis
of social anxiety disorder and 11.1% for a
principle diagnosis of specific phobia.
CGAS
CGI-S measures the
overall severity of child
disturbance (scores
range between 0 and
100).
CGI-S
Illness severity is rated
on a 7 point scale
(1”normal” to 7
“amongst the most
severely ill patients”).
Outcomes
(relevant to
review)
Descriptive
statistics revealed
no changes in
outcome scores on
the ADIS-P
between baseline
measures 1 and 2.
Post-treatment
scores however,
showed an average
CSR reduction of
2.8 (SD = 2.0),
Cohen’s d = 1.26,
amongst the intent
to treat group
(treated as the
primary analysis).
Overall, 66.7% of
participants were
found in the
normative range
following treatment.
A mean
improvement of
21.0 CGAS (SD =
11.0; Cohen’s d =
1.64) and 2.6 (SD =
1.0; Cohen’s d =
2.26) CGI-S
severity points was
observed between
baseline and posttreatment.
Doctorate in Educational and Child Psychology
Study Aims
Intervention
PCIT-ED
To report the
preliminary
findings from
a novel
adaptation of
PCIT for
preschool
children with
MDD.
PCIT-ED covers three
modules: CDI, PDI and
the novel emotional
development module.
Treatment is conducted
over 14 sessions, lasting
approximately 18 weeks
(+ 3.92). In contrast to
standard PCIT, CDI and
PDI is limited to 6
sessions and
demonstrated proficiency
is not required prior to
proceeding onto the next
module.
The novel component
includes one teaching
session followed by seven
coaching sessions, which
are intended to help
parents assist their child
in managing and
regulating intense
emotion. Homework was
assigned within each
module to support gains
made in sessions.
Design
One group
pretest
posttest
design
Pre-treatment
assessments
were
conducted
prior to the
onset of
treatment and
post-treatment
2 weeks
following the
final 14th
session.
Tamara Powell
Lenze et al. (2011)
Participants
Measures
(relevant to review)
Administrator n = 3
PAPA
Three graduate level clinicians delivered the
intervention. One co-therapist sat in every
session and completed a treatment integrity
checklist that was later reviewed alongside
videotapes by the lead author.
PAPA is a parent
interview for
diagnosing psychiatric
disorders in preschool
children ages 2 to 5
years. This measure
assesses intensity,
frequencies, durations,
and first onsets of
behaviour relevant to
Axis 1 diagnoses on
the DSM-IV.
Parents = 9
Six mothers, one father and one couple
completed treatment. Parental income
ranged between $15,000 to more than $60,
000 per year.
Children = 8
Participants were recruited through primary
care, mental health and pre-school settings.
Of the nine eligible participants, one refused
entry and one dropped out due to child birth
after 11 sessions. Children who did not meet
a diagnostic criteria of MDD, with any
neurological impairment, cognitive delays
and or living with an adult other than their
primary caregiver (> 6 months) were not
permitted to participate.
Final sample included seven Caucasian and
one biracial child (4F, 4M). Five children
were 5 years of age, one was 4 years of age
and two were 3 years of age.
HBQ-P
HSQ-P is a parent
questionnaire that
provides information
regarding the child’s
level of mental health
symptoms in the
internalising and
externalising domain.
Outcomes
(relevant to
review)
Wilcoxon Signed
Rank tests showed
significant
differences
between
participants’ pre (M
= 13.57, SD = 4.39)
and posttest (M =
7.57, SD = 6.00)
severity scores on
the PAPA (S =
2.38, p <.01,
Cohen’s d = 1.19).
Overall, mean
depression severity
scores decreased
by 44% and the
majority of
participants no
longer meet
diagnostic criteria
for MDD (5 out of
7).
Significant
reductions in
internalising
symptoms was also
reported on the
HBQ-P (Cohen’s d
= 0.82)
Doctorate in Educational and Child Psychology
Study Aims
Intervention
PCIT-ED
To build on the
previous work of
Lenze et al.
(2011) by
adding more
methodological
rigour in the
interpretations
of PCIT-ED and
its efficacy in a
preschool POMDD
population.
PCIT-ED covers three
modules: CDI, PDI and the
novel emotional development
module. Treatment is
conducted over 14 sessions,
lasting approximately 12
weeks. In contrast to
standard PCIT, CDI and PDI
is limited to 4 sessions each
and demonstrated
proficiency is not required
prior to proceeding onto the
next module.
The novel component, which
spans six sessions, directly
teaches parents emotional
competence and ways to
apply this in practice to
support their child’s
emotional development.
Psycho-educational
control condition (DEPI)
Parents are taught in groups
of 2 to 6 concepts that are
relevant to emotional and
social development, in the
absence of any individualised
coaching or practice.
Tamara Powell
Luby et al. (2012)
Design
Between subjects
randomised
pretest posttest
design
Using simple
random sampling,
eligible participants
were assigned to
one of the
intervention
conditions.
Therapists blind to
the treatment
condition and
treatment process
carried out the pre
and post treatment
assessments.
Treatment
commenced within
7 days of the initial
assessment, and
post-treatment
measures were
taken within 14
days following
treatment (median
11 days).
Participants
Administrators n = 3
Five graduate level therapists (Master’s
and Doctoral level) served as primary and
co-therapist pairs. The treatment integrity
checklist was filled out by each cotherapist, with results indicating a high
degree of adherence (average 97%).
Parents n = 1-2 per participant
Both parents were encouraged to attend
treatment sessions, however, mothers
generally participated for most families.
Children n = 43
Community paediatrician offices assisted
with recruitment efforts for children
between the ages of 3 to 7 years with
MDD. Children with neurological
impairments, cognitive or developmental
delay, and or adopted after 6 months of
age were excluded from the study.
Higher dropout rates were observed in the
control condition (ITT n = 18; completers n
= 10) relative to the experimental
condition (ITT n = 25, completers n = 19).
Sample included children ages 3 -6 who
were largely of Caucasian decent and
from married families with high annual
incomes and educational attainment (16F,
27M).
Measures
(relevant to
review)
PAPA
PAPA is a parent
interview for
diagnosing
psychiatric
disorders in
preschool children
aged 2 to 5 years.
This measure
assesses intensity,
frequencies,
durations, and first
onsets of behaviour
relevant to Axis 1
diagnoses in the
DSM-IV.
PFC-S
PFC-S is a 20 item
checklist scale that
assess depression
severity in
preschool children.
Responses are
recorded using a
Likert scale,
ranging from 0
(never) to 4 (most
of the time).
Outcomes
(relevant to
review)
Paired
sample t-tests
Significant
reductions in
depression
severity
measured
using the
PAPA MDD
was only
observed in
the PCIT-ED
condition
(Cohen’s d =
0.50).
Significant
declines in
depression
severity scores
on the PFC-S
and HBQ-P
were found in
in both the
PCIT-ED
(PFC-S
Cohen’s d =
0.83; HBQ-P
Cohen’s d =
0.0) and DEPI
condition.
Doctorate in Educational and Child Psychology
Study Aims
Intervention
PCIT
The
shortcomings
identified in this
study assisted in
the development
of a modified
version of PCIT
(brave
behaviour
module).
.
Standard PCIT contained
two phases: CDI and PDI
(Brinkmeyer & Eyberg,
2003). CDI focuses on
improving the quality of
the parent-child
relationship, while PDI
aims to reduce disruptive
behaviour in children.
Each respective section
begins with a teaching
session followed by
weekly coaching sessions
that run for approximately
1 hour. Mastery needs to
be displayed in CDI
before proceeding onto
the PDI phase (minimum
of 3 sessions is required
in each phase).
Tamara Powell
Design
One group pretest
posttest design
Behavioural
measures were
recorded before
and after treatment.
Pincus et al. (2008)
Participants
Administrators n = not specified
Parents n = 1 to 2
At least one parent or caregiver was
required to accompany the child to
treatment.
Children n = 10
Ten children (4F, 6M) ages 4 to 8 years
(M = 6.2 years) were recruited through
the Child and Adolescent Fear and
Treatment Anxiety Programme at the
Centre for Anxiety and Related
Disorders at Boston University. Inclusion
criteria required children to have a
principle diagnosis of SAD.
Measures
(relevant to
review)
ADIS-C/P
Semi-structured
diagnostic interview
designed to
specifically assess
child and
adolescent anxiety
and accompanying
mood disorders, in
accordance with
the DSM-IV criteria.
A CSR (ranging
between 0 and 8) of
4 or greater
indicates a clinical
diagnosis of
anxiety.
Outcomes
(relevant to
review)
Descriptive
statistics showed
a reduction in SAD
severity scores
before (M = 5.8)
and after (M = 4.2)
PCIT (Cohen’s d =
0.73). However,
there was no
clinically relevant
changes.
Key:
DISC-IV-P – diagnostic interview schedule for children; PCIT – parent-child interaction therapy; SAD – separation anxiety disorder; CDI – child directed interaction;
PDI – parent directed interaction; PPVT – peabody picture vocabulary test; WPT – wonderlic personnel test; ADIS-C/P – anxiety disorders interview schedule for
DSM-IV; child and parent versions; ODD – oppositional defiant disorder; CSR – clinical severity rating; CALM – coaching approach behaviour and leading and
modelling; GAD – generalized anxiety disorder; M – mean; SD – standard deviation; CGAS – the children’s global assessment scale; CGI-S – clinical global
impression-severity; IE- independent evaluator; ECBI – eyberg child behaviour inventory; PO-MDD – preschool onset major depressive disorder; PAPA – preschool
age psychiatric assessment; HBQ-P; health behaviour questionnaire-parent version; MDD – major depressive disorder; DEPI- developmental education and parenting
intervention; ITT – intent to treat sample; PFC-S – preschool feelings checklist scale; BDI – brave directed interactions.
Doctorate in Educational and Child Psychology
Tamara Powell
Appendix 6
Effect Size and Overall Quality Ratings
Study
Outcomes
Measure(s)
Chase &
Eyberg
(2008)
Group x time interaction,
with significant decreases
in symptom severity in
the SAD only group at
posttest.
Diagnostic Interview
Schedule for Children was
administered to child’s
female caregiver.
(DISC-IV-P)
Choate
et al.
(2005)
Clinical severity ratings
for SAD decreased to
subclinical levels at
posttest and remained in
the normative range at
follow up.
Anxiety Disorders
Interview Schedule for
DSM-IV – Child and Parent
Versions. (ADIS-C/P)
Participants
Effect Size
Descriptors
n
Overall
Weight of
Evidence
58
Low
Calculated from given data
ODD + SAD
Comorbid
diagnoses:
ADHD, CD,
MDD
Effect Size
Medium
PPC SMD = 1.48
Calculated from given data
SAD + Panic
Disorder
Large
Participant 1de
Posttest:
PP SMD = 1.29
Follow up:
PP SMD = 1.29
3
Low
SAD + ODD
Large
Participant 2de
Posttest
PP SMD = 1.71
Follow up:
PP SMD = 1.71
3
SAD
Large
Participant 3de
Posttest
PP SMD = 1.29
Follow-up:
PP SMD = 0.86
3
Doctorate in Educational and Child Psychology
Study
Outcomes
Comer et A diagnostic response
al. (2012) was observed in 75% of
SAD, 67% of social
anxiety, 50% of GAD and
0% of specific phobia
participants.
On average, global
severity scores improved
from “moderately” and
“markedly” ill to “normal”
“not at all ill” and
“borderline ill.”
Lenze et Significant decline in
al. (2011) depression severity
scores following
treatment.
Luby et
Significant decreases in
al. (2012) symptom severity as
measured by the PFC-S
and HBQ-P in both
experimental conditions.
PAPA MDD conversely
only found significant
changes in the PCIT-ED
group.
Tamara Powell
Measure(s)
Participants
Anxiety Disorders
Interview Schedule for
DSM-IV – Parent Version
(ADISC-P)
SAD
Social
Anxiety
Disorder
GAD
Specific
Phobias
Comorbidity:
OCD, ODD &
selective
mutism
Children’s Global
Assessment Scale
(CGAS)
Clinical Global Impression
Severity and Improvement
Scales (CGI-S)
Preschool Age Psychiatric
Assessment (PAPA MDD)
Health and Behavior
Questionnaire –
Internalising Symptoms
Subscale (HBQ-P)
Preschool Age Psychiatric
Assessment (PAPA MDD)
Preschool Feelings
Checklist-Scale (PFC-S)
Health and Behavior
Questionnaire –
Internalising Symptoms
Subscale (HBQ-P)
Effect Size
Descriptors
Effect Size
n
Overall
Weight of
Evidence
Calculated from given data
Largea
PP SMD = 1.26de
9
PP SMD = 1.64de
7
PP SMD = 2.26de
7
Largeb
Medium
Largeb
Calculated from given data
MDD
Large
PP SMD =1.19d
8
Medium
Large
PP SMD =0.82d
8
Calculated from given data
MDD
Medium
PPC SMD = 0.50
42
High
Large
PPC SMD = 0.83
43
No effect
PPC SMD = 0.00
40
Doctorate in Educational and Child Psychology
Tamara Powell
Study
Outcomes
Measure(s)
Pincus
et al.
(2008)
SAD symptomatology
decreased after
treatment, however, most
scores remained in the
clinical range.
Anxiety Disorders
Interview Schedule for
DSM-IV – Child and Parent
Versions (ADIS-C/P)
Participants
Effect Size
Descriptors
Effect Size
n
Overall
Weight of
Evidence
10
Low
Calculated from given data
SAD
Mediumc
PP SMD = 0.73d
Notes:
Effect size, referred to as Pretest-Posttest Standard Mean Difference (PP SMD; Becker, 1988), was calculated by dividing the difference in means by the pre-test
standard deviation score (Choate et al., 2005; Comer et al., 2012; Lenze et al., 2011; Pincus et al., 2008). In Chase and Eyberg (2008) and Luby’s et al. (2012) study,
effect size (Pretest-Posttest-Control Standard Mean Difference; PPC SMD) was calculated by dividing the difference in means (pre and post treatment) by the pooled
standard deviation (Hedge’s g; Morris, 2007).
f) To reduce potential sampling biases, statistical analyses was performed on the intent to treat sample.
g) Data could not be obtained for the intent to treat sample, so effect size was computed using the completers sample.
h) Standard deviation for measure was not reported, it was, therefore, computed using a sample of children with comparable demographics (Comer et al., 2012).
i) Effect sizes were corrected in accordance to Hedges’ (1981) guidelines for small samples (e.g., groups with 10 or less participants; Becker, 1988).
j) Choate et al. (2005) examined the impact of treatment on children’s symptomatology on an individual basis, whereas Comer et al. (2012) explored pre-post
treatment differences using group means. To maintain consistency, the current review reported individual effect sizes at the two phases of treatment in
Choate’s et al. (2005) study, and group level effect sizes between baseline and post-treatment for participants in Comer’s et al. (2012) SCED paper.
Key:
ODD- oppositional defiant disorder; SAD- separation anxiety disorder; ADHD- attention deficit hyperactive disorder; CD- conduct disorder; MDD- major
depressive disorder; GAD- generalised anxiety disorder; OCD-obsessive compulsive disorder.
Effect Size Descriptor
Small
Medium
Large
Cohen’s d
0.20
0.50
0.80
52
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Appendix 7
Coding Protocol: Group Based Design
Domain:
School-and community-based intervention programs for social and behavioural
problems
Academic intervention programmes
Family and parent intervention programmes
School-wide and classroom-based programmes
Comprehensive and coordinated school health services
Name of Coder: ____TP_____
Date: January 16th, 2015
Full name of Study in APA format: Chase, R. M., & Eyberg, S. M. (2008). Clinical
presentation and treatment outcome for children with comorbid externalizing and
internalizing symptoms. Journal of Anxiety Disorders, 22(2), 273–82.
Intervention Name (description from study):Parent-Child Interaction Therapy
(PCIT)
Study ID Number (Unique Identifier): 1
Type of Publication: (Check one)
Book/Monograph
Journal article
Book Chapter
Other (specify):
53
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
A. General Characteristics
Quasi-Experimental Designs Without Control Groups
The One-Group Posttest-Only Design
The One-Group Posttest-Only Design with Multiple Substantive Posttests
The One-Group Pretest-Posttest Design
The One-Group Pretest-Posttest Design Using a Double Pretest
The One-Group Pretest-Posttest Design Using a Non-equivalent Dependent
Variable
The Removed-Treatment Design
The Repeated-Treatment Design
B. Statistical Treatment/ Data Analysis
Yes No N/A
B1. Appropriate unit of analysis
B2. Familywise error rate controlled
B3. Sufficiently large N
B.3.1 Statistical Test: ANOVA
B3. 2 Level: 2
ES: Large
N required: 52
B4. Total size of sample (start of the study): 64
B5. Intervention group sample size: 58 (15 with and 49 without SAD)
B6. Control group sample size: 0
C. Type of Programme
C.1 Universal prevention programme
C.2 Selective prevention programme
C.3 Targeted prevention programme
C.4 Intervention/Treatment
C.5 Unknown
D. Stage of the Programme
D.1 Model/demonstration programmes
D.2 Early stage programmes
D.3 Established/institutionalised programmes
D.4 Unknown
54
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
E. Concurrent or Historical Intervention Exposure
E.1 Current exposure
E.2 Prior Exposure
E.3 Unknown
II. Key Features for Coding Studies and Rating Levels of Evidence/Support
(3 = Strong Evidence; 2 = Promising Evidence; 1 = Weak Evidence; 0 = No Evidence)
A. Measurement
A1. Use of outcome measure produces reliable sources for the majority of primary
outcomes.
A1.1Yes
A1.2 No
A1.3 Unknown/Unable to code
A2. Multi-method
A2.1 Yes
A2.2 No
A2.3 N/A
A2.4 Unknown/Unable to code
A3. Multi-source
A3.1 Yes
A3.2 No
A3.3 N/A
A3.4 Unknown/Unable to code
A4. Validity of measures reported
A4.1 Yes validated with specific target group
A4.2 In part, validated for general population only
A4.3 No
A4.4 Unknown/Unable to code
Rating for Measurement
3
2
1
0
55
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
B. Comparison Group
No control group – all participants received the intervention
B1. Type of Comparison Group
B1.1 Typical contact
B1.2 Typical contact (other) specify:
B1.3 Attention placebo
B1.4 Intervention elements placebo
B1.5 Alternative intervention
B1.6 Pharmacotherapy
B1.7 No intervention
B1.8 Wait list/ delayed intervention
B1.9 Minimal contact
B1.10 Unable to identify comparison group
Rating for Comparison Group
NNR- no control group
3
2
1
0
B2. Overall confidence rating in judgement of type of comparison group
B2.1 Very low (little basis)
B2.2 Low (guess)
B2.3 Moderate (weak inference)
B2.4 High (strong inference)
B2.5 Very high (explicitly stated)
B2.6 N/A
B2.7 Unknown/unable to code
B3. Counterbalancing of Change Agents
B3.1 By change agent
B3.2 Statistical
B3.3 Other
One lead therapist coached all parents who had children with ODD+SAD and ODD only.
High adherence to programme standards were evidenced through treatment integrity
checklist (M = 90%).
B4. Group Equivalence Established
B4.1 Random assignment
B4.2 Post hoc matched set
B4.3 Statistical matching
B4.4 Post hoc test for group equivalence
56
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Not Applicable – No control group
B5. Equivalent Mortality
B5.1 Low attrition (less than 20% for Post)
B5.2 Low attrition (less than 30% for follow-up)
B5.3 Intent to intervene analysis carried out
Findings:
Attrition rate was 36%, which is similar to other PCIT treatment programmes
(Gallagher, 2003). ITT was not carried out, however, attrition rates were similar for
children with and without SAD.
C. Primary Outcomes Are Statistically Significant
C.1 Evidence of appropriate statistical analysis for primary outcomes
C1.1 Appropriate unit of analysis
C1.2 Familywise/experimenterwise error rate controlled when applicable
C1.3 Sufficiently large N
C2. Percentage of primary outcomes that are statistically significant
C2.1 Significant primary outcomes for at least 75% of the total primary
measures for each key construct
C2.2 Significant primary outcomes for between 50% and 74% of the total
primary measures for each key construct
C2.3 Significant primary outcomes for between 25% and 49% of the total
primary measures for each key construct
Rating for Primary Outcomes Statistically Significant
3
2
1
0
C3. Evidence of appropriate statistical analysis for primary outcomes
Yes
C3.1 Main effect analyses conducted
C3.2 Moderator effect analyses conducted specify results:
C3.3 Mediator analyses conducted, specify results:
No
57
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology Association, Kratochwill,
T.R. (2003)]
D. Educational/Clinical Significance
Outcome Variable
Pretest
D1. Categorical Diagnosis
Data
Diagnostic information
regarding inclusion into the
DISC-IV-P; Diagnostic Interview
study presented:
Schedule for Children – Parent
Edition
Yes
No
Unknown
D2. Outcomes Assessed via
continuous Variables
DISC-IV-P; Outcome Score
Posttest
Positive change in diagnostic
criteria from pre to posttest:
Positive change in diagnostic
criteria from posttest to follow
up:
Yes No Unknown
Positive change in the
percentage of participants
showing clinical improvement
from pre to posttest:
Yes No Unknown
Positive change in the
percentage of participants
showing clinical improvement
from posttest to follow up:
Yes
D3. Subjective Evaluation:
The importance of behaviour
change is evaluated by
individuals in direct contact with
the participant.
Importance of behaviour
change is evaluated:
Yes No Unknown
D4. Social Comparison:
Behaviour of participant change Participant’s behaviour is
is evaluated by individuals in
compared to normative data:
direct contact with the
participant
Yes
No
Unknown
Follow-up
No
Unknown
Importance of behaviour
change from pre to posttest is
evaluate positively by
individuals in direct contact with
the participant:
Yes
No
Unknown
Participant’s behaviour has
improved from pre to posttest
when compared to normative
data:
Yes
No
Unknown
Yes
No
Unknown
Importance of behaviour
change from posttest to follow
up is evaluate positively by
individuals in direct contact with
the participant:
Yes
No
Unknown
Participant’s behaviour has
improved from posttest to
follow up when compared to
normative data:
Yes
No
Unknown
58
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Rating for Educational/Clinical Significance:
3
2
1
0
E. Identifiable Components
E1. Evidence for primary outcomes:
3
2
1
0
Yes No
E2. Design allows for analysis of identifiable components
E3. Total number of components: 2
E4. Number of components linked to primary outcomes: 0
Additional criteria to code descriptively:
Yes No
E5. Clear documentation of essential components
E6. Procedures for adapting the intervention are described
in detail
E7. Contextual features of the intervention are documented
Rating of Identifiable Components
3
2
1
0
F. Implementation Fidelity
F1. Evidence of Acceptable Adherence
F1.1 Ongoing supervision/consultation
F1.2 Coding intervention sessions/lessons or procedures
F1.3 Audio/video tape implementation
F1.3.1 Entire intervention
F1.3.2 Part of intervention
59
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
F2. Manualization
F2.1 Written material involving a detailed account of the exact procedures
and the sequence in which they are to be used
F2.2 Formal training session that includes a detailed account of the exact
procedures and the sequence in which they are to be used
F2.3 Written material involving an overview of broad principles and a
description of the intervention phases
F2.4 Formal or informal training session involving an overview of broad
principles and a description of the intervention phases
Yes No
Unknown
F3. Adaptation procedures are specified
Rating for Implementation Fidelity
3
2
1
0
III. Other Descriptive or Supplemental Criteria to Consider
A. External Validity Indicators
Yes No
A1. Sampling procedures described in detail
Specify rationale for selection: Participants were part of a larger study
Specify rationale for sample size: No rational provided
Yes No
A1.1 Inclusion/exclusion criteria specified
A1.2 Inclusion/exclusion criteria similar to school practice
A1.3 Specified criteria related to concern
A2. Participant Characteristics Specified for Treatment and Control Group (see
Appendix 5).
A3. Details are provided regarding variables that:
Yes No
A3.1 Have differential relevance for intended outcomes
Specify: SES, comorbid diagnoses
A3.2 Have relevance to the inclusion criteria
Specify: Scores on screening measure (DISC-IV-P)
60
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
A4. Receptivity/acceptance by target participant population
No data on receptivity/acceptance reported
Participants from
Treatment Group
Results (What person
reported to have gained
from participation in
programme)
General Rating
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
A5. Generalisation of Effects:
A5.1 Generalisation over time
Yes No
A5.1.1 Evidence is provided regarding the sustainability of outcomes
after intervention is terminated. Specify:
A5.1.2 Procedures for maintaining outcomes are specified
A5.2 Generalisation across settings
Yes No
A5.2.1 Evidence is provided regarding the extent to which outcomes
are maintained in contexts that are different from the intervention
context. Specify:
A5.2.2 Documentation of the efforts to ensure application of
intervention to other settings. Specify: Daily homework
A5.2.3 Impact on implementers or context is sustained. Specify:
61
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Unable to code A5.2.3 because there is insufficient information provided.
A5.3 Generalisation across persons
Yes No
A5.3.1 Evidence is provided regarding the degree to which outcomes
are manifested with participants who are different than the original
group of participants for with the intervention was evaluated. Specify:
B. Length of Intervention
B1. Unknown/insufficient information provided
B2. Information provided (if information provided, specify one of the
following):
B2.1. Weeks: 19
B2.2 Months: __
B2.3 Years: __
B2.4 Other: __
C. Intensity/dosage of Intervention
C1. Unknown/insufficient information provided
C2. Information provided (if information provided, specify one of the
following):
C2.1. Length of intervention session: unknown
C2.2 Frequency of intervention sessions: 14 sessions
D. Dosage Response
D1. Unknown/insufficient information provided
D2. Information provided (if information provided, answer D2.1):
D2.1 Describe positive outcomes associated with higher dosage:_______
E. Programme Implementer
E1. Research Staff
E2. School Speciality Staff
E3. Teachers
E4. Educational Assistants
E5. Parents
E6. College Students
E7. Peers
E8. Others
E9. Unknown/insufficient information provided
62
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
F. Characteristics of the Intervener
F1. Highly similar to target population on key variables (e.g., race,
gender, SES)
F2. Somewhat similar to target participants on key variables
F3. Different from target participants on key variables
Unknown/insufficient information provided.
G. Intervention Style or Orientation
G1. Behavioural
G2. Cognitive-behavioural
G3. Experimental
G4. Humanistic/Interpersonal
G5. Psychodynamic/insight oriented
G6. Other, specify:
G7. Unknown/insufficient information provided
H. Cost Analysis Data
H1. Unknown/insufficient information provided
H2. Information provided (if information provided, answer H2.1):
H2.1 Estimated Cost of Implementation:______________________
I. Training and Support Resources
I1. Simple orientation given to change agents
I2. Training workshops conducted
Parents received two training sessions that preceded the CDI and PDI phases.
# of workshops provided: _2 (parents)_
Average length of training: _unknown_
Who conducted training:
I2.1 Project Director
I2.2 Graduate/project assistants
12.3 Other, specify:
12.4 Unknown
I3. Ongoing technical support
I4. Programme materials obtained
I5. Special facilities
I6. Other, specify:
One way mirror and bug in the ear receiver required for this intervention.
J. Feasibility
J1. Level of difficulty in training intervention agents
J1.1 High
J1.2 Moderate
63
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
J1.3 Low
J1.4 Unknown
J2. Cost to train intervention agents (specify if known):_________
J3. Rating cost to train intervention agents:
J3.1 High
J3.2 Moderate
J3.3 Low
J3.4 Unknown
Summary of Evidence for Group-Based Design Studies
Indicator
General Characteristics
General Design Characteristics
Statistical Treatment
Type of Programme
Stage of Programme
Concurrent/Historical Intervention
Exposure
Key Features
Measurement
Comparison Group
Primary Outcomes are Statistically
Significant
Educational/Clinical Significance
Identifiable Components
Implementation Fidelity
Descriptive or Supplemental
Criteria
External Validity Indicators
Length of Intervention
Intensity/Dosage
Dosage Response
Programme Implementer
Characteristics of the Intervener
Intervention Style/Orientation
Cost Analysis Data Provided
Overall
Evidence
Rating NNR
= No
numerical
rating or 03
Description of Evidence
(Strong, Promising, Weak,
No/limited evidence, or
Descriptive Ratings
NNR
NNR
NNR
NNR
One-group pretest-posttest
design
ANOVA
Intervention/Treatment
Established/Institutionalised
NNR
Unknown
1
NNR
Weak Evidence
No Control Group
3
2
0
3
NNR
NNR
NNR
NNR
NNR
NNR
NNR
NNR
Strong Evidence
Promising Evidence
No Evidence
Strong Evidence
Some Evidence
19 weeks
14 sessions
Unknown
Research Staff and Parents
Unknown
Behavioural/Psychodynamic
Unknown
64
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Training and Support Resources
NNR
Feasibility
NNR
Parents receive two training
sessions; One-way mirror
and bug in the ear receiver
Unknown
65
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol: Single-Participant Design
Domain:
School-and community-based intervention programs for social and behavioural
problems
Academic intervention programmes
Family and parent intervention programmes
School-wide and classroom-based programmes
Comprehensive and coordinated school health services
Name of Coder: ____TP_____
Date: January 16th, 2015
Full name of Study in APA format: Choate, M. L., Pincus, D. B., Eyberg, S. M., &
Barlow, D. H. (2001). Parent-Child Interaction Therapy for Treatment of Separation
Anxiety Disorder in Young Children : A Pilot Study, Cognitive and Behavioral
Practice,12(1),126–135.
Intervention Name (description from study):Parent-Child Interaction Therapy
(PCIT).
Study ID Number (Unique Identifier): 2
Type of Publication: (Check one)
Book/Monograph
Journal article
Book Chapter
Other (specify):
66
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
1. General Characteristics
A. General Design Characteriistics
A1. Type of Single-Participant Design
A1.1 Within-series design
A1.1.1 Simple phase change
A1.1.2 Complex phase change
A1.2 Between-series design
A1.2.1 Comparing two interventions
A1.2.2 Comparing interventions with no interventions
A1.3 Combined-series design
A1.3.1 Multiple baseline across participants
A1.3.2 Multiple baseline across behaviours
A1.3.3 Multiple baseline across settings
A1.3.4 Multiple probe design
A1.4 Mixed design
A1.4.1 Combined single-participant and group design
A1.4.2 Combined single-participant design
A1.4.2.1 Within-series design
i. Simple phase change
ii. Complex phase change
A1.4.2.2 Between-series design
i. Comparing two interventions
ii. Comparing interventions with no interventions
A1.4.2.3 Combined-series design
i. Multiple baseline across participants
ii. Multiple baseline across behaviours
iii. Multiple baselines across settings
iv. Multiple probe design
A1.5 Other, Specify
B. Other Design Characteristics (when randomisation is used)
B1. Unit of assignment to conditions/groups
B1.1 Individual
B1.2 Classroom
B1.3 School
B1.4 Other, Specify
B1.5 N/A (randomisation not used)
B2.Type of assignment to conditions/groups
B2.1 Random after matching, stratification, blocking
B2.2 Random simple (includes systematic sampling)
B2.3 Nonrandom, post hoc matching
B2.4 Nonrandom, other
B2.5 Other, Specify
B2.6 Unknown/insufficient information provided
67
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
B2.5 N/A (randomisation not used)
B3. Overall confidence of judgement on how participants were assigned to
conditions/groups
B3.1 Very low (little basis)
B3.2 Low (guess)
B3.3 Moderate (weak inference)
B3.4 High (strong inference)
B3.5 Very high (explicitly stated)
B3.6 N/A (randomisation not used)
B3.7 Unknown/unable to code
B4. Equivalence of conditions/groups tested at pretest
B4.1 Yes
B4.2 No
B4.3 Unknown/insufficient information
provided
B4.4 N/A (randomisation not used)
B5. Total size of sample (start of the study): 3
B6. Intervention group sample size: 3
B7. Control group sample size: 0
C. Type of Programme
C.1 Universal prevention programme
C.2 Selective prevention programme
C.3 Targeted prevention programme
C.4 Intervention/Treatment
C.5 Unknown
D. Stage of the Programme
D.1 Model/demonstration programmes
D.2 Early stage programmes
D.3 Established/institutionalised programmes
D.4 Unknown
E. Concurrent or Historical Intervention Exposure
E.1 Current exposure
E.2 Prior Exposure
E.3 Unknown
68
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
II. Key Features for Coding Studies and Rating Levels of Evidence/Support
(3 = Strong Evidence; 2 = Promising Evidence; 1 = Weak Evidence; 0 = No
Evidence)
A. Measurement
A1. Use of outcome measures that produce reliable sources
A1.1Yes
A1.2 No
A1.3 Unknown/Unable to code
A2. Multi-method
A2.1 Yes
A2.2 No
A2.3 N/A
A2.4 Unknown/Unable to code
A3. Multi-source
A3.1 Yes
A3.2 No
A3.3 N/A
A3.4 Unknown/Unable to code
The outcome measure (ADIS-C/P) is not standardized for children younger than 7 years of
age. It was therefore completed with the child and parent in only 2 out of the 3 families.
A4. Validity of measures reported
A4.1 Yes
A4.2 No
A4.3 Unknown/Unable to code
Rating for Measurement
3
2
1
0
B. Quality of Baseline
Participant 1
B1. Length: At least 3 data points during baseline
B1.1 Yes
B1.2 No
B1.3 Unknown/insufficient information
69
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
B2. Stability: Variability in scores does not eliminate the detection of treatment
effects
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
B3. Overlap: Extreme scores during baseline do not overlap with most scores
during intervention phase
B3.1 Yes
B3.2 No
B3.3 Unknown/insufficient information
B4. Level: Behaviour is serious enough during baseline to warrant an
intervention
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
B5. Trend: Behaviour is not systematically increasing or decreasing in the
desired direction of intervention effects during baseline.
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
Rating for quality of baseline for participant 1
3
2
1
0
Participant 2
B1. Length: At least 3 data points during baseline
B1.1 Yes
B1.2 No
B1.3 Unknown/insufficient information
B2. Stability: Variability in scores does not eliminate the detection of treatment
effects
B2.1 Yes
70
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
B2.2 No
B2.3 Unknown/insufficient information
B3. Overlap: Extreme scores during baseline do not overlap with most scores
during intervention phase
B3.1 Yes
B3.2 No
B3.3 Unknown/insufficient information
B4. Level: Behaviour is serious enough during baseline to warrant an
intervention
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
B5. Trend: Behaviour is not systematically increasing or decreasing in the
desired direction of intervention effects during baseline.
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
Rating for quality of baseline for participant 2
3
2
1
0
Participant 3
B1. Length: At least 3 data points during baseline
B1.1 Yes
B1.2 No
B1.3 Unknown/insufficient information
B2. Stability: Variability in scores does not eliminate the detection of treatment
effects
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
71
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
B3. Overlap: Extreme scores during baseline do not overlap with most scores
during intervention phase
B3.1 Yes
B3.2 No
B3.3 Unknown/insufficient information
B4. Level: Behaviour is serious enough during baseline to warrant an
intervention
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
B5. Trend: Behaviour is not systematically increasing or decreasing in the
desired direction of intervention effects during baseline.
B2.1 Yes
B2.2 No
B2.3 Unknown/insufficient information
Rating for quality of baseline for participant 3
3
2
1
0
Average Quality of Baseline Rating Across Participants
∑ of X/N = 0+0+0/3
Overall Rating for Quality of Baseline
3
2
1
0
72
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
C. Visual Analysis
Outcome #1:ADIS-C/P (Primary Outcome)
Participant
Participant 1
Participant 2
Visual Analysis
C1.1 Changes in levels
C1.1.1
large
C1.1.2
moderate-to-large
C1.1.3
moderate
C1.1.4
no change
C1.1.5
unknown/insufficient information
C1.2 No to minimal score overlap
C1.2.1
yes
C1.2.2
no
C1.2.3
unknown/insufficient information
C1.3 Change in trend
C1.3.1
yes
C1.3.2
no
C1.3.3
N/A (no trend present)
C1.3.4
unknown/insufficient information
C1.4 Adequate length
C1.4.1
yes
C1.4.2
no
C1.4.3
unknown/insufficient information
C1.5 Stable data
C1.5.1
yes
C1.5.2
no
C1.5.3
unknown/insufficient information
Rating for Visual Analysis for Participant 1:
C1.1 Changes in levels
C1.1.1
large
C1.1.2
moderate-to-large
C1.1.3
moderate
C1.1.4
no change
C1.1.5
unknown/insufficient information
C1.2 No to minimal score overlap
C1.2.1
yes
C1.2.2
no
C1.2.3
unknown/insufficient information
C1.3 Change in trend
C1.3.1
yes
C1.3.2
no
C1.3.3
N/A (no trend present)
C1.3.4
unknown/insufficient information
C1.4 Adequate length
C1.4.1
yes
C1.4.2
no
C1.4.3
unknown/insufficient information
C1.5 Stable data
C1.5.1
yes
C1.5.2
no
C1.5.3
unknown/insufficient information
0
1
2
3
73
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Participant 3
Rating for Visual Analysis for Participant 2:
C1.1 Changes in levels
C1.1.1
large
C1.1.2
moderate-to-large
C1.1.3
moderate
C1.1.4
no change
C1.1.5
unknown/insufficient information
C1.2 No to minimal score overlap
C1.2.1
yes
C1.2.2
no
C1.2.3
unknown/insufficient information
C1.3 Change in trend
C1.3.1
yes
C1.3.2
no
C1.3.3
N/A (no trend present)
C1.3.4
unknown/insufficient information
C1.4 Adequate length
C1.4.1
yes
C1.4.2
no
C1.4.3
unknown/insufficient information
C1.5 Stable data
C1.5.1
yes
C1.5.2
no
C1.5.3
unknown/insufficient information
0
1
2
3
Rating for Visual Analysis for Participant 3:
0
1
2
3
Measures Support Primary Outcomes
Average Rating for Measures Support Primary Outcomes Across Participants Using
Score from Visual Analysis
∑ of X/N = 0+0+0/3
Overall Rating for Measures Support Primary Outcomes:
3
2
1
0
74
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology Association, Kratochwill,
T.R. (2003)]
D. Educational/Clinical Significance
Outcome Variable
D1. Categorical Diagnosis
Data
ADIS-C/P; Dichotomous variable
verifying the existence of anxiety
(clinical severity score [CSR] of 4
or greater)
Pretest
Diagnostic information
regarding inclusion into the
study presented:
Yes
No
Unknown
D2. Outcomes Assessed via
continuous Variables
ADIS-C/P; Clinical Severity Score
Posttest
Positive change in diagnostic
criteria from pre to posttest:
Positive change in diagnostic
criteria from posttest to follow
up:
Yes No Unknown
Positive change in the
percentage of participants
showing clinical improvement
from pre to posttest:
Yes No Unknown
Positive change in the
percentage of participants
showing clinical improvement
from posttest to follow up:
Yes
D3. Subjective Evaluation:
The importance of behaviour
change is evaluated by
individuals in direct contact with
the participant.
Importance of behaviour
change is evaluated:
Yes No Unknown
D4. Social Comparison:
Behaviour of participant change Participant’s behaviour is
is evaluated by individuals in
compared to normative data:
direct contact with the
participant
Yes
No
Unknown
Follow-up
No
Unknown
Importance of behaviour
change from pre to posttest is
evaluate positively by
individuals in direct contact with
the participant:
Yes
No
Unknown
Participant’s behaviour has
improved from pre to posttest
when compared to normative
data:
Only on the PAPA MDD (not
PFC-S) measure
Yes
No
Unknown
Yes
No
Unknown
Importance of behaviour
change from posttest to follow
up is evaluate positively by
individuals in direct contact with
the participant:
Yes
No
Unknown
Participant’s behaviour has
improved from posttest to
follow up when compared to
normative data:
Yes
No
Unknown
75
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Rating for Educational/Clinical Significance:
3
2
1
E. Identifiable Components
E1. Evidence for primary outcomes:
3
2
1
0
Yes No
E2. Design allows for analysis of identifiable components
E3. Total number of components: 2
E4. Number of components linked to primary outcomes: 0
Additional criteria to code descriptively:
Yes
E5. Clear documentation of essential components
E6. Procedures for adapting the intervention are described
in detail
E7. Contextual features of the intervention are documented
Rating of Identifiable Components
3
2
1
0
F. Implementation Fidelity for Participants 1 to 3
F1. Evidence of Acceptable Adherence
F1.1 Ongoing supervision/consultation
F1.2 Coding intervention sessions/lessons or procedures
F1.3 Audio/video tape implementation
F1.3.1 Entire intervention
F1.3.2 Part of intervention
No
76
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
F2. Manualization
F2.1 Written material involving a detailed account of the exact procedures
and the sequence in which they are to be used
F2.2 Formal training session that includes a detailed account of the exact
procedures and the sequence in which they are to be used
F2.3 Written material involving an overview of broad principles and a
description of the intervention phases
F2.4 Formal or informal training session involving an overview of broad
principles and a description of the intervention phases
Yes
No
Unknown
F3. Adaptation procedures are specified
Rating for Implementation Fidelity
3
2
1
0
III. Other Descriptive or Supplemental Criteria to Consider
A. External Validity Indicators
Yes No
A1. Sampling procedures described in detail
Specify rationale for selection: Within age limits with a SAD diagnosis
Specify rationale for sample size: No rational provided
Yes
No
A1.1 Inclusion/exclusion criteria specified
A1.2 Inclusion/exclusion criteria similar to school practice
A1.3 Specified criteria related to concern
A2. Participant Characteristics Specified for Treatment and Control Group (see
Appendix 5).
A3. Details are provided regarding variables that:
Yes
A3.1 Have differential relevance for intended outcomes
Specify: comorbid diagnoses
A3.2 Have relevance to the inclusion criteria
Specify: Scores on screening measure (ADIS-C/P)
No
77
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
A4. Receptivity/acceptance by target participant population
No data reported for receptivity/acceptance
Participants from
Treatment Group
Results (What person
reported to have gained
from participation in
programme)
General Rating
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Child/Student
Parent/caregiver
Teacher
School
Other
Participants reported
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
Participants reported not
benefiting overall from the
intervention
A5. Generalisation of Effects:
A5.1 Generalisation over time
Yes No
A5.1.1 Evidence is provided regarding the sustainability of outcomes
after intervention is terminated. Specify: Follow-up 3-6 months after
treatment
A5.1.2 Procedures for maintaining outcomes are specified
A5.2 Generalisation across settings
Yes No
A5.2.1 Evidence is provided regarding the extent to which outcomes
are maintained in contexts that are different from the intervention
context. Specify:
A5.2.2 Documentation of the efforts to ensure application of
intervention to other settings. Specify: Daily homework
78
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
A5.2.3 Impact on implementers or context is sustained. Specify:
Unable to code A5.2.3 because there is insufficient information provided.
A5.3 Generalisation across persons
Yes
A5.3.1 Evidence is provided regarding the degree to which outcomes
are manifested with participants who are different than the original
group of participants for with the intervention was evaluated. Specify:
B. Length of Intervention
B1. Unknown/insufficient information provided
B2. Information provided (if information provided, specify one of the
following):
B2.1. Weeks: __
B2.2 Months: __
B2.3 Years: __
B2.4 Other: __
C. Intensity/dosage of Intervention
C1. Unknown/insufficient information provided
C2. Information provided (if information provided, specify one of the
following):
C2.1. Length of intervention session: unknown
C2.2 Frequency of intervention sessions: 6 to 7 sessions
D. Dosage Response
D1. Unknown/insufficient information provided
D2. Information provided (if information provided, answer D2.1):
D2.1 Describe positive outcomes associated with higher dosage:_______
E. Programme Implementer
E1. Research Staff
E2. School Speciality Staff
E3. Teachers
E4. Educational Assistants
E5. Parents
E6. College Students
E7. Peers
E8. Others
E9. Unknown/insufficient information provided
No
79
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
F. Characteristics of the Intervener
F1. Highly similar to target population on key variables (e.g., race,
gender, SES)
F2. Somewhat similar to target participants on key variables
F3. Different from target participants on key variables
Unknown/insufficient information provided
G. Intervention Style or Orientation
G1. Behavioural
G2. Cognitive-behavioural
G3. Experimental
G4. Humanistic/Interpersonal
G5. Psychodynamic/insight oriented
G6. Other, specify:
G7. Unknown/insufficient information provided
H. Cost Analysis Data
H1. Unknown/insufficient information provided
H2. Information provided (if information provided, answer H2.1):
H2.1 Estimated Cost of Implementation:______________________
I. Training and Support Resources
I1. Simple orientation given to change agents
I2. Training workshops conducted
Length of training for lead and co-therapist was not specified. Parents received one teaching
session during the PDI phase.
# of workshops provided: _1(parents)__
Average length of training: ____
Who conducted training:
I2.1 Project Director
I2.2 Graduate/project assistants
12.3 Other, specify:
12.4 Unknown
I3. Ongoing technical support
I4. Programme materials obtained
I5. Special facilities
I6. Other, specify:
One way mirror and bug in the ear receiver required for this intervention.
80
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
J. Feasibility
J1. Level of difficulty in training intervention agents
J1.1 High
J1.2 Moderate
J1.3 Low
J1.4 Unknown
J2. Cost to train intervention agents (specify if known):_________
J3. Rating cost to train intervention agents:
J3.1 High
J3.2 Moderate
J3.3 Low
J3.4 Unknown
Summary of Evidence for Group-Based Design Studies
Indicator
General Characteristics
General Design Characteristics
Other design characteristics (when
randomisation is used)
Type of Programme
Stage of Programme
Concurrent/Historical Intervention
Exposure
Key Features
Measurement
Comparison Group
Primary Outcomes are Statistically
Significant
Educational/Clinical Significance
Identifiable Components
Implementation Fidelity
Descriptive or Supplemental
Criteria
External Validity Indicators
Length of Intervention
Intensity/Dosage
Overall
Evidence
Rating NNR
= No
numerical
rating or 03
NNR
NNR
NNR
NNR
Description of Evidence
(Strong, Promising, Weak,
No/limited evidence, or
Descriptive Ratings
Multiple baseline across
participants
Not applicable
Intervention/Treatment
Established/Institutionalised
Unknown
NNR
1
0
Weak Evidence
No Evidence
0
2
0
2
No Evidence
Promising Evidence
No Evidence
Promising Evidence
NNR
NNR
NNR
Some Evidence
Unknown
6 to 7 sessions
81
[adapted from Task Force on Evidence-Based Interventions in School Psychology,
American Psychology Association, Kratochwill, T.R. (2003)]
Dosage Response
Programme Implementer
Characteristics of the Intervener
Intervention Style/Orientation
Cost Analysis Data Provided
Training and Support Resources
NNR
NNR
NNR
NNR
NNR
NNR
Feasibility
NNR
Unknown
Research Staff and Parents
Unknown
Behavioural/Psychodynamic
Unknown
Parents receive one training
session; One-way mirror
and bug in the ear receiver
Unknown