Doctorate in Educational and Child Psychology
Abigail Miranda
Case Study 1: An Evidence-Based Practice Review Report
Theme: Interventions for children with Special Educational Needs
How effective is Parent Child Interaction Therapy (PCIT) for improving externalising
behaviours and compliance in children with conduct disorder and associated
externalising behaviour disorders?
Summary
Parent Child Interaction Therapy (PCIT) is an intervention targeted at parents and
children displaying externalising behaviour disorders. It is implemented by trained
professionals, and is unrestricted by a set number of sessions but ends when parents
achieve mastery of learnt skills and the child scores below clinical cut-off points on
measures for conduct disorder. This literature review aimed to evaluate the efficacy of
PCIT on externalising behaviours in children with conduct disorder and associated
externalising behaviour disorders.
A systematic literature review was undertaken using three databases. After application of
inclusion and exclusion criteria, five studies were selected for review using Kratchowill’s
(2003) coding protocol, and the weight of evidence was subsequently calculated using
Gough’s (2007) Weight of Evidence Framework. The review found that PCIT can be
efficacious in improving disordered externalising behaviours in children. The implications
for minority group participants are discussed, as are methodological limitations. The
intervention would be preferable to no treatment in most cases; however, to control for
expectancy effects, further research using alternative interventions is recommended.
1
Doctorate in Educational and Child Psychology
Abigail Miranda
Introduction
What is PCIT?
PCIT is an intervention developed by Sheila M. Eyberg in the USA which uses a didactic
approach. A therapist or PCIT trained professional uses direct coaching with the parent,
as research suggests that modifications in behaviour of parents can have an effect on a
child’s behaviour. A meta-analytic review by Serketich and Dumas (1996) found that
parental behaviour modification programmes had positive effects in the short term on the
child’s behaviour. Conversely, a study by Strassberg, Dodge, Pettit, and Bates, (1994),
found that parental aggressive behaviour in the form of context-inappropriate spanking
increased the aggressive behaviour of children towards their peers. The latter suggests
that negative parental behaviours influence the behaviours of a child, and the former
indicates that positive behavioural changes in the parent can have beneficial effects on
the child’s behaviour.
There are two distinct phases in PCIT, which are Child Directed Interactions (CDI) and
Parent Directed Interactions (PDI). The former is a play-based relationship building
phase, whilst the latter consists of reinforcement of pro-social behaviours whilst ignoring
negative behaviours. The intervention consists of instructional sessions, followed by direct
coaching sessions using a one-way mirror and ear piece. This set-up enables the
therapist to instruct and coach the parent without imposing on the interaction between the
parent and child. Aside from sessions with the therapist, parents are given home
exercises to complete.
PCIT is unique in that the number of sessions given is based on parent performance and
reduction in problem behaviours in the child. As well as pre and post measures, the
2
Doctorate in Educational and Child Psychology
Abigail Miranda
intervention is evaluated after each session. This enables the therapist to cease the
intervention when the child’s behaviour falls within the normal range. Research by
Goldfine et al., (2008) indicates that the start-up cost is approximately $14,000, which
translates to approximately $1000 per parent and child dyad.
Psychological Basis
PCIT is an intervention designed to treat conduct disordered behaviour in children. The
programme aims to enable the carer to establish a nurturing and secure relationship with
their child. It is dyadic in the sense that it treats the parent in order to increase pro-social
behaviours in the child whilst decreasing negative behaviours. Due to the circular
relationship presented between parent and child behaviours, this review focusses on the
externalising behaviours presented by the child, as well as compliance to adult requests.
The programme draws knowledge from Baumrind’s research on differing parenting styles,
including her 1967 study on the positive outcomes associated with authoritative parenting.
This parenting style is inclusive of nurturing parenting and setting limits for the child. The
idea of the importance of a nurturing parenting style comes from Bowlby’s (1978)
Attachment Theory. The theory highlights the importance of an attentive and responsive
parent, who acts as a safe base for the child to explore their surroundings. The child
directed interaction (CDI) phase of PCIT serves as play-based therapy for the parent to
practice skills involved in creating a secure attachment relationship with the child.
Bandura’s (1977) Social Learning theory suggests that children learn through what they
observe, and are likelier to imitate behaviours of those perceived to be similar to them.
They also use responses from others when learning whether to maintain or cease
behaviours. This is supported by Skinner’s (1938) Operant Conditioning theory, in which
the response to behaviour can serve as either a deterrent or reinforcement. The former
3
Doctorate in Educational and Child Psychology
Abigail Miranda
and latter theories both suggests that if parental attention is the reward, then externalising
behaviour may function as a tool to illicit attention; positive or negative. Looking at both
Attachment theory and Social Learning theory, Patterson (1991) proposed an ‘Early
Starter Model,’ which suggests that a child’s maladaptive behaviours can be learnt and
reinforced by the behaviour of the parent. This theory suggests that change in the parent
is instrumental in altering disordered behaviour in the child. The parent directed
interaction phase (PDI) allows for anteceding behaviours to be altered, in an attempt to
improve negative behavioural outcomes for the child.
Rationale
Externalising behaviours are an area of focus within the field of both psychology and
politics, as children displaying conduct disordered behaviour were found by Scott, Knapp,
Henderson, and Maughan, (2001) to cost society up to ten times more than typically
developing children. Within the field of education, disruptive behaviour, physical assault,
verbal abuse and threatening behaviour account for four in ten permanent exclusions and
half of all fixed term exclusions (DfE, 2011). Aside from school exclusion, there are other
associated negative educational outcomes for children with externalising behaviour. A
longitudinal study by Fergusson and Horwood (1998) found that children displaying
externalizing behaviours at 8 years old were significantly likelier to leave school without
qualifications and be unemployed at the age of 18.
There are several factors thought to be involved in the maintenance of conduct disorders;
however singular causal links have not been established. Murray and Farrington (2010)
named several environmental as well as within child factors that had been identified via
longitudinal studies. Powell et al (2006) noted that research suggests that challenging
behaviour displayed by young children is usually interconnected with the quality and
consistency of the relationships and interactions the child has with their caregiver. Early
4
Doctorate in Educational and Child Psychology
Abigail Miranda
intervention for conduct disorder is preferential, as toddlers who display early signs of
conduct disorder and externalising behaviours are at risk of poor educational
experiences, negative peer relationships and impaired family functioning. This can lead to
a cyclical process that negatively impacts on their social and emotional development.
Critical periods in brain development have come to the fore in recent years, as Allen
(2013) suggests that between the ages of nought and three, a child’s brain is more
malleable and therefore early intervention is recommended in order to prevent problems
from escalating.
Parenting programmes focus on an area of risk where change is possible, and metaanalytic studies have demonstrated the efficacy of parenting programmes. Comer, Chow,
Chan, Cooper-Vince and Wilson, (2013) concluded in a meta-analysis that psycho-social
treatments for conduct disorder should be used as a first option, as supported by the
quantitative data analysed in the review. PCIT therefore has potential importance to the
field of Educational Psychology in enabling children displaying externalising behaviours to
reduce the frequency and amount of disordered behaviours, therefore reducing the risk of
school exclusion and failing to receive qualifications whilst in education.
Review Question
How effective is Parent Child Interaction Therapy (PCIT) for improving externalising
behaviours and compliance in children with conduct disorder and associated externalising
behaviour disorders?
Critical Review of the Evidence base
Literature Search
Searches were conducted between January and February 2014. PsychINFO, ERIC and
Medline databases were screened using a multi-field search. Search terms were input in
all fields, abstract and/ or title. For a detailed breakdown of search strategies, refer to
5
Doctorate in Educational and Child Psychology
Abigail Miranda
Table 1. All abstracts were screened according to the criteria in Table 2. After screening
for duplicates, this initial search produced 19 papers for review of the full text.
Table 1. Search terms used in PschINFO, ERIC and Medline database
*= wildcard term
Including and Excluding Studies
Table 2 displays the criterion that was applied to each paper found in the database
searches. In many cases, setting limits described in the table allowed papers that did not
meet the inclusion criteria to be screened out immediately.
Table 2. Inclusion and exclusion criteria
Inclusion Criteria
Exclusion Criteria
1. Type of
Publication
The study must be in a peer
reviewed journal. This ensures
methodological rigour, as peer
reviewed journals have been
assessed using stringent criteria.
The material is not in a peerreviewed journal (including books
and conference papers).
2. Language
The study is written in English,
due to unavailability of resources
for translation.
The study is not written in
English.
Multi- field search terms
Limits
PCIT (abstract) and conduct disorder
(all)
English language, peer reviewed journal,
2007- current
Parent Child Interaction Therapy
(abstract) and conduct disorder (all
fields)
English language, peer reviewed journal,
2007- current
PCIT (title) and conduct disorder (all
fields)
English language, peer reviewed journal,
2007- current
Parent Child Interaction Therapy (title)
and conduct disorder (all fields)
English language, peer reviewed journal,
2007- current
PCIT (abstract) and behavio* (abstract)
3. Type of
Study
English language, peer reviewed journal,
2007- current
The study must contain primary
The study does not contain
empirical data, to ensure
primary empirical data e.g.
6
Doctorate in Educational and Child Psychology
Abigail Miranda
originality of findings.
Literature reviews and metaanalyses
4.
Intervention
The study implements the full and
unabbreviated PCIT to individuals
in the experimental group as part
of the study.
The study implements an
abbreviated version of PCIT
alone, a modified version of
PCIT, or does not implement the
full programme.
5. Design
The study must be a group
design, to allow greater
generalisability of findings.
The study must utilise a clinical
control group of participants who
did not receive PCIT
The study is a single case design.
6. Dependent
variables
(measures)
Analysis
The study measures the efficacy
of PCIT in relation to child
behaviour.
The study reports means and
standard deviations of
behavioural measures.
The study does not measure the
efficacy of PCIT in relation to
child behaviour.
The study does not report means
and standard deviations of
behavioural measures.
7. Year of
Publication
The study is published post 2007,
and has not been included in the
previous 2007 systematic review.
The study is published pre 2007,
and/ or it has been included in the
previous 2007 systematic
literature review.
8.
Participants
Participants have been selected
on the basis that they have
externalising behaviour disorders
and/ or conduct disorder.
Participants must not be selected
on the basis that they have been
subjected to, or at risk of abuse
due to possible coercive nature of
intervention
Participants selected do not have
externalising behaviour disorders
and/ or conduct disorder.
The study did not have a clinical
control group of participants who
did not receive PCIT.
Participants have been selected
on the basis that they have been
subjected to, or at risk of abuse.
7
Doctorate in Educational and Child Psychology
Abigail Miranda
Flowchart 1 displays the details of the number of papers screened at each stage of
review.
Flowchart: Application of inclusion and exclusion criteria
PsycInfo
Eric
Medline
Total
n = 309
n = 61
n = 14
n = 384
Papers for
review of title
and abstract
384
Papers excluded:
Duplicates
n = 68
Inclusion criteria not met n =
297
Papers for review
of full text
19
Studies excluded:
Inclusion criteria 1
Inclusion criteria 2
Inclusion criteria 3
Inclusion criteria 4
Inclusion criteria 5
Inclusion criteria 6
Inclusion criteria 7
Inclusion criteria 8
Total excluded
14
n=0
n=0
n=1
n=3
n=6
n=3
n=0
n=1
n=
Studies Included
5
8
Doctorate in Educational and Child Psychology
Abigail Miranda
After the screening process was complete, five studies met the inclusion criteria and were
then subject to a critical review. Included studies are listed in Table 3. Full details of
excluded studies, as well as reasons for exclusion are listed in Appendix 1.
Table 3. Included studies
Included Studies
Bagner, D. M., & Eyberg, S. M. (2007). Parent–child interaction therapy for disruptive
behavior in children with mental retardation: A randomized controlled trial. Journal of
Clinical Child and Adolescent Psychology, 36(3), 418-429.
Leung, C., Tsang, S., Heung, K., & Yiu, I. (2009). Effectiveness of Parent—Child
interaction therapy (PCIT) among Chinese families. Research on Social Work Practice,
19(3), 304-313.
McCabe, K., & Yeh, M. (2009). Parent–child interaction therapy for Mexican Americans: A
randomized clinical trial. Journal of Clinical Child & Adolescent Psychology, 38(5), 753759.
Bagner, D. M., Graziano, P. A., Jaccard, J., Sheinkopf, S. J., Vohr, B. R., & Lester, B. M.
(2012). An initial investigation of baseline respiratory sinus arrhythmia as a moderator of
treatment outcome for young children born premature with externalizing behavior
problems. Behavior therapy, 43(3), 652-665.
Bagner, D. M., Sheinkopf, S. J., Vohr, B. R., & Lester, B. M. (2010). Parenting
intervention for externalizing behavior problems in children born premature: an initial
examination. Journal of developmental and behavioral pediatrics: JDBP, 31(3), 209.
Comparison of Selected Studies
Five studies were summarised (see Appendix 2) and then compared on various aspects
of design and outcomes. Out of the five studies that were critically appraised, four studies
were randomised control trials (RCT’s) and one study was a non-randomised between
participants block design. Three out of the five studies explicitly stated the randomisation
of participants to the point where it would be repeatable. All five of the studies were
critically examined using the UCL Educational Psychology Literature Review Coding
Protocol, which was adapted from the APA Task Force on Evidence Based Interventions
9
Doctorate in Educational and Child Psychology
Abigail Miranda
in School Psychology (Kratochwill, 2003). The use of the protocol facilitated a systematic
analysis of included studies, based on the same criterion. Areas of focus for analysis
were quality of measures, comparison group and analysis. All included studies contained
quantitative data, therefore were all analysed using this protocol.
The Weight of Evidence framework (Gough, 2007) was subsequently used as a
systematic tool to analyse the effectiveness of the studies in answering the proposed
research question. Table 4 outlines the aspects analysed using the framework. An overall
Weight of Evidence relevant to the research question was formulated scoring studies on
specific facets: Quality of Methodology, Relevance of Methodology and Relevance of
Evidence to the Review Question. Scores on the three dimensions were averaged in
order to obtain an Overall Weight of Evidence score. Details of weightings given to each
study can be found in Table 5, and Appendix 4 contains the raw scores from which the
Overall Weight of Evidence was calculated.
Table 4. Weight of Evidence Framework (Gough, 2007)
Weight of
Weight of
Weight of
Evidence
Evidence
Evidence
A
B
C
Generic judgement Review-specific
Review-specific
about the
judgement about
judgement about
coherence and
appropriateness of the relevance of
integrity of the
the evidence for
the focus of the
evidence
answering the
evidence for the
(Quality of
review question
review question
Methodology)
(Relevance of
(Relevance of
Methodology)
evidence to the
review question)
Weight of
Evidence
D
Overall
assessment of the
extent to which the
study contributes
evidence to answer
a review question
(Overall weight of
evidence)
10
Doctorate in Educational and Child Psychology
Abigail Miranda
Critical Review
Participants
As per the inclusion criteria, all participants were selected on the basis that they reached
clinical cut-off points for conduct disorder on parent report measures at the screening
stage. Only one study (Bagner and Eyberg, 2007) used DSM-IV criteria as a formal
diagnosis of oppositional defiant disorder (ODD). The study by Leung, Tsang, Heung, and
Yiu, (2009),used clinical cut-off points which were found to be relevant with the target
population of Chinese children and their families, which happened to be under the
standardised cut-off points on the ECBI scale. Two out of the four studies excluded
participants on the basis of co-morbid disorders which may have added confounding
variables to the intervention. All studies conducted statistical analyses to compare
homogeneity of groups.
Participants were recruited from a variety of sources such as health professionals, selfreferral and schools. Abuse cases were screened out during the inclusion process due to
the coerced nature of involvement in the study, which could act as a confounding
variable. Studies were carried out in USA and China. The demographics of participants
varied; however three out of the five studies appeared to have a cultural mixture of
participants. The remaining two studies were conducted on homogenous cultural groups,
although the participants in the McCabe and Yeh (2009) study would be considered as a
minority group, being Mexican-Americans in a USA study. The range of participants
between studies suggests that PCIT does have applications for use when working with
parents and families from diverse backgrounds; however caution should be taken when
interpreting co-morbid disorders, as these may not always be diagnosed or picked up by
the screening measures used in the study. Loeber and Keenan (1994) highlighted the
11
Doctorate in Educational and Child Psychology
Abigail Miranda
relationship between conduct disorder and co-morbid disorders that are commonly
associated with the condition.
Research Design
Control Group
All included studies utilised clinical control groups to form a comparison with the
intervention group. Three out of the five studies operated using a randomised control or
blocked design, which was explicitly stated and replicable. The Leung et al., (2009) study
used a quasi- experimental design, which was non-randomised as the control group
participants were recruited and allocated to their experimental condition from different
sources to the intervention group. All studies either established group equivalence by
random assignment, or by post-hoc tests.
The McCabe and Yeh (2009) study was the only one that used an alternative intervention
control group. In this study, control participants received treatment as usual from a
therapist. Ethically, this study is sounder in that participants were given immediate access
to treatment. Additionally, the alternative intervention acted as a control for expectancy
effects, as the effects of improvement from merely receiving additional support was
considered in designing this experiment.
Intervention
All interventions were carried out in clinical settings. This was necessary due to the
requirement for a one-way mirror. The absence of sound longitudinal data meant that it
was unclear whether the results generalised to long-term gains for participants. Studies
utilising PCIT in natural or group settings would be delivering a modified version of the
12
Doctorate in Educational and Child Psychology
Abigail Miranda
programme, however it may be beneficial to examine the effects of the naturalistic setting
on results.
Interventions were carried out by therapists and trained professionals. Two studies
explicitly stated the role and qualifications of the therapists delivering the intervention, and
the remaining two studies made reference to a ‘therapist,’ although it was unclear who
this individual was in terms of role and experience. PCIT is a manualised programme, and
all studies reviewed reported that interventions were carried out by professionals trained
in delivering PCIT.
Measures
All five studies used at least one of the following measures: Child Behavior Checklist for
11 2 to 5 Year Olds (CBCL; Achenbach and Rescorla, 2000), Eyberg Child Behavior
Inventory (ECBI; Eyberg and Pincus, 1999) and the Dyadic Parent–Child Interaction
Coding System (DPICS; Eyberg, Nelson, Duke and Boggs, 2004). These measures were
chosen for further inspection due to their relevance to the review question, as they
captured parent reported and observed externalising behaviours.
The ECBI was deemed suitable for the review as it measured the number of externalising
behaviours, as well as the number of times they occurred. This measure yielded reliability
coefficients ranging from .80 - .95 in the five studies that used it, suggesting that it is a
highly reliable measure. The CBCL consists of both an internalising and externalising
behaviour scale. For relevance to the review question, where applicable only scores for
the externalising component of the scale were analysed. The CBCL yielded reliability
coefficients ranging from .54 - .94 in the three studies that used it (Bagner and Eyberg,
2007; McCabe and Yeh, 2009 and Bagner, D. M., Sheinkopf, S. J., Vohr, B. R., and
13
Doctorate in Educational and Child Psychology
Abigail Miranda
Lester, B. M. (2010).. For studies that used either the ECBI or the CBCL as a standalone
measure, the subjectivity of a parental report can be questioned due to the vested interest
of the reporter. All studies reported the reliability coefficients of all measures used, bar the
Leung (2009) study which did not report reliability for the DPICS. This was an aspect
reflected in the low weighting given for measures. All other studies received a medium
rating for measures.
The DPICS was used as an observational measure in all studies bar one (Bagner,
Graziano, Jaccard, Sheinkopf., Vohr & Lester, 2012). This measure yielded reliability
coefficients of .50 - .89 in the three studies that reported this data. Although the manual is
very specific about how to code indirect and direct commands, with families who speak
English as an additional language (EAL) linguistic subtleties may be incorrectly coded,
and this is not discussed in terms of the validity of the observational tool; nor is prior
exposure of coders to diverse groups. Given the cultural diversity in many of the studies,
this would require some acknowledgement in future studies. Four out of the five studies
used two or more methods of report, from two or more sources. This triangulation formed
part of the medium to high ratings on Weight of Evidence C, for all but two studies (Leung
et al, 2009 and Bagner et al, 2012).
Analysis
Four out of the five studies used appropriate methods of analysis for results. The study
that did not, McCabe and Yeh (2009), calculated within group effect sizes for pre and post
data collection. A between group comparison would have been more relevant to the
14
Doctorate in Educational and Child Psychology
Abigail Miranda
hypotheses stated in the paper. Two studies (Bagner and Eyberg, 2007 and Leung et al.,
2009) used the analysis of covariance (ANCOVA), which was considered the most
suitable statistical test used in the selected studies as it is effective when data has been
collected multiple times for the same measures. High weight of evidence ratings were
given to these studies for the analysis component. The remaining two studies (Bagner et
al, 2010 and Bagner et al., 2012), used calculations of effect sizes and a multiple
regression to analyse results. Two studies (Leung et al, 2009 and McCabe and Yeh,
2009) used post hoc tests to control for experimenter error rate; Bonferroni correction and
Tukey’s post hoc tests respectively.
Findings
The majority of the studies found medium to large effects between intervention and
control groups, suggesting that PCIT had some efficacy in reducing externalising
behaviours. These results must be interpreted with caution, as three out of the five
studies (McCabe and Yeh, (2009), Bagner and Eyberg, (2007) and Bagner et al., (2010),
some effects were negative. This is a particularly pertinent finding in the McCabe and Yeh
(2009) study, which used an active comparison group.
The negative effect sizes, displayed in Table 5 for these three studies indicate that the
intervention had a negative impact on self-reported data, as measured by the CBCL and
ECBI. The Bagner and Eyberg (2007) results displayed a medium sized negative effect,
15
Doctorate in Educational and Child Psychology
Abigail Miranda
as described by Cohen (1988) on the number of externalising behaviours on the ECBI.
This suggests that parents reported fewer externalising behaviours before engaging with
the PCIT intervention. McCabe and Yeh (2009) reported results displaying medium to
large negative effect sizes on the CBCL Externalising Behaviours subscale, small to
medium negative effect sizes on the ECBI Frequency of externalising behaviours, and
small negative effect sizes in the number of externalising behaviours on the ECBI. This
suggests that in comparison with the alternative intervention of individualised sessions
with a therapist, parents who undertook PCIT reported a lesser range and frequency of
externalising behaviours before the intervention in comparison to after the intervention.
Similarly, the Bagner et al., (2012) study displayed a large negative effect within the ECBI
Intensity subscale, which indicated that parents reported more externalising behaviours
after the intervention in comparison to before.
Whilst this finding may be interpreted as negative, it could also be argued that PCIT increases
parental awareness of what externalising and negative behaviours are, which in turn allows them
to identify and report these behaviours more accurately. Studies that triangulated self-report data
with use of the DPICS would indicate that this was the case, as the DPICS effect sizes were
positive and large in the studies that used it as a measure (Bagner & Eyberg, 2007; Leung et al.,
2009; McCabe & Yeh, 2009; Bagner et al., 2010). These findings display improvements in
observed, coded behaviour of children whose parents undertook PCIT, comparing pre and post
intervention behaviours. This is also the case in the McCabe and Yeh (2009) study, where an
alternative intervention was used; however as improvements were also made with the alternative
intervention, this suggests that PCIT is more focussed on improving child behaviours than an
individualised therapeutic intervention.
The remaining three studies found positive small to large effects of the intervention when
compared to the control. The results and overall quality ratings can be found in Table 5.
16
Doctorate in Educational and Child Psychology
Abigail Miranda
Table 5. Summary of Effect sizes and overall quality of studies
Study
Bagner &
Eyberg,
2007
Measures
CBCL
(Achenbach and Rescorla,
2000).
ECBI
(Eyberg
and Pincus, 1999).
DPICS
(Eyberg, Nelson, Duke
andBoggs,
2004).
Leung et al., ECBI
2009
(Eyberg
and Pincus, 1999).
Effect sizes
Externalising behaviour,
Cohen’s d = .97 (large)
Overall
Quality
Rating
Medium
Frequency Cohen’s d =
1.5 (large)
Number Cohen’s d =
-.66 (medium)
Compliance Cohen’s d
= 1.53 (large)
Frequency
Cohen’s d = 1.59, CI
1.15 - 1.71
(large)
Low
Number
Cohen’s d = 1.52, CI
1.08 - 1.43
(large)
McCabe &
Yeh, 2009
DPICS
(Eyberg, Nelson, Duke and
Boggs,
2004).
Compliance
Partial eta-squared =
.49 (large)
CBCL
(Achenbach and Rescorla,
2000).
Externalising behaviour,
Cohen’s d =
-.79, (medium to large)
ECBI
(Eyberg
and Pincus, 1999).
Frequency
Cohen’s d =
-.49 (small to medium)
High
Number
Cohen’s d =
-.36 (small)
Bagner et
DPICS
(Eyberg, Nelson, Duke
andBoggs,
2004).
Cohen’s d =
.35 (small)
ECBI
Intensity
Low
17
Doctorate in Educational and Child Psychology
Abigail Miranda
al., 2012
(Eyberg
and Pincus, 1999).
Cohen’s d =
-1.10 (large)
Bagner et
al., 2010
CBCL
(Achenbach and Rescorla,
2000).
Externalising behaviours
Cohen’s d = 2.3 (large)
Medium
Frequency
Cohen’s d = 2.3 (large)
ECBI
(Eyberg
and Pincus, 1999).
Number
Cohen’s d = 1.4 (large)
DPICS
(Eyberg, Nelson, Duke and
Boggs,
2004).
Compliance
Cohen’s d =
.90 (large)
Note: CBCL=Child Behavior Checklist, ECBI=Eyberg Child Behavior Inventory, DPICS=Dyadic Parent Child
Interaction Coding System.
Table 6. Descriptors of small, medium and large effect sizes (Green and Salkind, 2008)
Type of effect size
Partial Eta Square
(Cohen, 1988)
Cohen's d (Cohen,
1988)
Small Medium
0.01
0.06
0.20
0.50
Large
0.14
0.80
Three out of the five studies showed that PCIT was effective when compared to a control
group. Out of these three studies, two studies were found to be of medium overall quality.
These two studies (Bagner and Eyberg, 2007 and Bagner et al., 2010) found large
positive effects of PCIT when compared to a waitlist control group. These findings
suggest that there is some evidence for the use of PCIT as a treatment for externalising
behaviours.
18
Doctorate in Educational and Child Psychology
Abigail Miranda
The Bagner and Eyberg, (2007) and Bagner et al., (2010) studies were given a medium
rating in regards to relevance to the research question. This was due to their use of a
waitlist control group, and triangulation of data. Both studies were judged to be of
medium overall quality. Positive large effect sizes of .97 and 2.3 were found on the CBCL.
The ECBI frequency scale yielded positive large effect sizes of 1.5 and 2.3. For the ECBI
intensity scale, a medium negative effect of -.66 was found in the Bagner and Eyberg
(2007) study, whereas the Bagner et al., (2010) study noted an effect size of 2.3 for this
subscale. The effect sizes on the DPICS were 1.53 and .90, and this measure was
considered to be the most objective used throughout the studies. Despite the promising
findings, it must be noted that the Bagner and Eyberg (2007) study was a first generation
study, as the PCIT author was also an author of the study.
The McCabe and Yeh (2009) study was the only study found to be of high relevance to
review, as it utilised an active control group. It also received a high overall rating, as it
established group equivalency and found 75% or more significant results. Considering
that the study was well designed, the negative effect sizes are all the more pertinent.
ECBI effect sizes were -.49 and -.36 on the frequency and intensity scales. The CBCL
effect size for externalising behaviour was -.79. A small positive effect of .35 was found
on the DPICS. These results indicate that although parents in the intervention condition
did not find significant positive improvements between groups, observers did find a small
positive between group effect.
The final two studies, Bagner et al., (2012) and Leung et al., (2009) were regarded as
being of low relevance to the review question due to a lack of triangulation of data and
implementation of the intervention by a person other than a therapist. The studies also
received low overall ratings, largely due to a lack of establishing group equivalency and
equivalent mortality.
19
Doctorate in Educational and Child Psychology
Abigail Miranda
ECBI intensity effect sizes were large but negative in the Bagner et al., (2012) study at 1.10, and large but positive in the Leung et al., (2009) study at 1.52. Bagner et al., (2012)
did not report any other relevant measures. Leung et al., (2009) found a large positive
effect of 1.59 on the ECBI frequency subscale, and a large partial eta-squared effect of
.49 for compliance on the DPICS, as described by Cohen (1988). Control group data was
not available to calculate the Cohen’s d effect size for standardisation purposes. Although
the results were promising, the low overall weighting and relevance to the review meant
that these findings were considered less robust.
Conclusion and Recommendations
The studies reviewed that received medium overall ratings (Bagner and Eyberg, 2007 and
Bagner et al., 2010) were deemed as efficacious on both parent report and observational
measures when compared to a non-active control group. The main factor that
compromised ratings was the absence of an active control group. It was also noted that
the author of the programme was also an author of the evaluative study, suggesting an
element of vested interest; however, the method outlined steps taken to reduce bias, such
as inter-rater coding of sessions to check fidelity, and only the first author being involved
in the delivery of PCIT.
Four out of the five studies used random assignment to treatment conditions, and only
one study used an active control group. This meant that expectancy effects were not
controlled for in the majority of studies analysed, and it was not apparent that these were
adjusted for via statistical analyses. In the McCabe and Yeh (2009) study that used an
20
Doctorate in Educational and Child Psychology
Abigail Miranda
active comparison group, although the parent report effects between groups were
negative, the observation effects between groups showed a small positive effect. This
may have been due to parents receiving non-directional one-to-one support from a
therapist in the control condition, therefore leading to an increased sense of control over
the treatment being compatible with their perceived needs.
Although highly prescriptive in the description of coding observed behaviours, the studies
that used the DPICS did not note any steps taken to ensure cultural sensitivity in the
application of this measurement instrument. It has been noted in a meta-analysis by Van
Ijzendoorn and Kroonenberg, (1988) that observational methods such as Ainsworth and
Bell’s (1970) Strange Situation procedure are subject to cross cultural differences in the
presentation of behaviours. Although some of the studies undertook inter-rater codings to
control for observer bias, if the coders all had the same perspectives and experience of
diverse ethnic groups, it could be argued that their ability to note subtle differences may
be similarly limited.
The studies were all clinical trials; therefore there is limited information as to how results
translate into real world settings. In cases where children were in educational settings, it
would have been feasible to collect pre and post teacher reported data in order to further
triangulate wider effects of the programme. This would heighten the case for validity,
whilst making a contribution to the field of Educational Psychology.
Recommendations
There are several recommendations for future research and implementing PCIT as a
therapeutic intervention. The use of the Dyadic Parent–Child Interaction Coding System
(DPICS) would be recommended in future studies, due to increased subjectivity when
21
Doctorate in Educational and Child Psychology
Abigail Miranda
using a trained coder blind to experimental conditions, and the ability to assess inter-rater
reliability. This would counterbalance the subjectivity of parental report methods used.
Further to this recommendation, the use of active control groups is recommended. A
comparison could then be made of the efficacy of PCIT with other treatments and would
subsequently inform best practice in terms of evidence based and cost effective
interventions.
As PCIT consists of two parts which have distinctive psychological underpinnings, PCIT
would be useful in future research to examine which aspects are successful in treating the
holistic profile of conduct disorder. This would then inform evidence based practice
targeting the components that make up an authoritative parent. In weighing up start-up
and delivery costs alongside the information presented in this review, PCIT would only be
recommended where no other alternative was available at this present time, and for
cases where parents required directive solutions, such as court or government required
parenting programmes in cases of physical abuse.
Future studies examining longitudinal or school based effects of PCIT in comparison to
alternative treatments are needed to find out the maintenance of the effects, although it is
granted that this research will be not without difficulty given the short-term attrition
demonstrated in the studies examined.
22
Doctorate in Educational and Child Psychology
Abigail Miranda
Reference and Appendices
References
Allen, G. (2011). Early intervention: the next steps, an independent report to Her
Majesty's government by Graham Allen MP. TSO Shop.
Bagner, D. M., & Eyberg, S. M. (2007). Parent–child interaction therapy for disruptive
behavior in children with mental retardation: A randomized controlled trial. Journal of
Clinical Child and Adolescent Psychology, 36(3), 418-429.
Bagner, D. M., Graziano, P. A., Jaccard, J., Sheinkopf, S. J., Vohr, B. R., & Lester, B. M.
(2012). An initial investigation of baseline respiratory sinus arrhythmia as a moderator of
treatment outcome for young children born premature with externalizing behavior
problems. Behavior therapy, 43(3), 652-665.
Bagner, D. M., Sheinkopf, S. J., Vohr, B. R., & Lester, B. M. (2010). Parenting
intervention for externalizing behavior problems in children born premature: an initial
examination. Journal of developmental and behavioral pediatrics: JDBP, 31(3), 209.
Barker, C., Pistrang, N., & Elliott, R. (2003). Research methods in clinical psychology: An
introduction for students and practitioners. John Wiley & Sons.
Baumrind, D. (1967). Child care practices anteceding three patterns of preschool
behavior. Genetic psychology monographs.
23
Doctorate in Educational and Child Psychology
Abigail Miranda
Bessmer, J. L. (1998). The Dyadic Parent-Child Interaction Coding System II (DPICS II):
Reliability and validity (Doctoral dissertation, ProQuest Information & Learning).
Boggs, S. R., Eyberg, S., & Reynolds, L. A. (1990). Concurrent validity of the Eyberg child
behavior inventory. Journal of Clinical Child Psychology, 19(1), 75-78.
Bowlby, J. (1978). Attachment theory and its therapeutic implications. Adolescent
psychiatry.
Comer, J. S., Chow, C., Chan, P. T., Cooper-Vince, C., & Wilson, L. A. (2013).
Psychosocial treatment efficacy for disruptive behavior problems in very young children: a
meta-analytic examination. Journal of the American Academy of Child & Adolescent
Psychiatry, 52(1), 26-36.
Department for Education. (2011). Statistical First Release PERMANENT AND FIXED
PERIOD EXCLUSIONS FROM SCHOOLS. DfE: London.
Fergusson, D. M., & Horwood, L. J. (1998). Early conduct problems and later life
opportunities. Journal of Child Psychology and Psychiatry, 39(8), 1097-1108.
Goldfine, M. E., Wagner, S. M., Branstetter, S. A., & Mcneil, C. B. (2008). Parent-Child
Interaction Therapy: An Examination of Cost-Effectiveness. Journal of Early & Intensive
Behavior Intervention, 5(1).
Gough, D. (2007). Weight of Evidence: A Framework for the Appraisal of the Quality and
Relevance of Evidence. Research Papers in Education, 22, 213-228.
24
Doctorate in Educational and Child Psychology
Abigail Miranda
Kazdin, A.E. (2005a). Evidence-based assessment for children and adolescents: issues
in measurement development and clinical application. Journal of Clinical Child
&Adolescent Psychology: The Official Journal for the Society of Clinical Child &
Adolescent Psychology, American Psychological Association, Division 53, 34(3),
548–558.
Kazdin, A. E. (2005). Parent management training: Treatment for oppositional,
aggressive, and antisocial behavior in children and adolescents. New York: Oxford
University Press.
Leung, C., Tsang, S., Heung, K., & Yiu, I. (2009). Effectiveness of Parent—Child
interaction therapy (PCIT) among Chinese families. Research on Social Work Practice,
19(3), 304-313.
Loeber, R., & Keenan, K. (1994). Interaction between conduct disorder and its comorbid
conditions: Effects of age and gender. Clinical Psychology Review, 14(6), 497-523.
McCabe, K., & Yeh, M. (2009). Parent–child interaction therapy for Mexican Americans: A
randomized clinical trial. Journal of Clinical Child & Adolescent Psychology, 38(5), 753759.
Murray, J., & Farrington, D. P. (2010). Risk factors for conduct disorder and delinquency:
key findings from longitudinal studies. Canadian journal of psychiatry. Revue canadienne
de psychiatrie, 55(10), 633-642.
Patterson, G. R., DeBaryshe, B. D., & Ramsey, E. (1989). A developmental perspective
on antisocial behavior (Vol. 44, No. 2, p. 329). American Psychological Association.
25
Doctorate in Educational and Child Psychology
Abigail Miranda
Patterson, G. R., Capaldi, D., & Bank, L. (1991). An early starter model for predicting
delinquency. In An earlier draft of this chapter was presented at the Earlscourt
Conference on Childhood Aggression, Toronto, Canada, Jun 1988.. Lawrence Erlbaum
Associates, Inc.
Pearl, E. S. (2009). Parent management training for reducing oppositional and aggressive
behavior in preschoolers. Aggression and Violent Behavior, 14(5), 295-305.
Powell, D., Dunlap, G., & Fox, L. (2006). Prevention and intervention for the challenging
behaviors of toddlers and preschoolers. Infants & Young Children, 19(1), 25-35.
Scott, S., Knapp, M., Henderson, J. and Maughan, B. (2001). Financial cost of social
exclusion: Follow‐up study of antisocial children into adulthood. British Medical Journal,
323(7306): 1–5. 191
Serketich, W. J., & Dumas, J. E. (1996). The effectiveness of behavioral parent training to
modify antisocial behavior in children: A meta-analysis. Behavior therapy, 27(2), 171-186.
Strassberg, Z., Dodge, K. A., Pettit, G. S., & Bates, J. E. (1994). Spanking in the home
and children's subsequent aggression toward kindergarten peers. Development and
psychopathology, 6(03), 445-461.
Thomas, R., & Zimmer-Gembeck, M. J. (2007). Behavioral outcomes of parent-child
interaction therapy and Triple P—Positive Parenting Program: A review and metaanalysis. Journal of abnormal child psychology, 35(3), 475-495.
Van Ijzendoorn, M. H., & Kroonenberg, P. M. (1988). Cross-cultural patterns of
attachment: A meta-analysis of the strange situation. Child Development, 147-156.
26
Doctorate in Educational and Child Psychology
Abigail Miranda
Appendix 1
Excluded articles from PsychINFO, ERIC and Medline
Studies
Abrahamse, M. E., Junger, M., Chavannes, E. L.,
Coelman, F. J., Boer, F., & Lindauer, R. J.
(2012). Parent–child interaction therapy for
preschool children with disruptive behaviour
problems in the Netherlands. Child and
adolescent psychiatry and mental health, 6(1), 19.
Rationale for exclusion
Criteria 5 – does not have a
clinical control group
2
Bagner, D. M., Rodriguez, G. M., Blake, C. A.,
Rosa-Olivares, J. (2013). Home-based
preventive parenting intervention for at-risk
infants and their families: An open trial. Cognitive
and Behavioral Practice. 20(3), 334-348.
Criteria 5 – does not have a
control group
3
Chase, R. M., & Eyberg, S. M. (2008). Clinical
presentation and treatment outcome for children
with comorbid externalizing and internalizing
symptoms. Journal of Anxiety Disorders, 22(2),
273-282.
Criteria 5 – does not have a
control group
4
Fernandez, M. A., & Eyberg, S. M. (2009).
Predicting treatment and follow-up attrition in
parent–child interaction therapy. Journal of
Abnormal Child Psychology, 37(3), 431-441.
Criteria 6 - The study does
not measure the efficacy of
PCIT in relation to child
behaviour.
5
Fernandez, M. A., Butler, A. M., & Eyberg, S. M.
(2011). Treatment outcome for low
socioeconomic status African American families
in parent-child interaction therapy: A pilot study.
Child & Family Behavior Therapy, 33(1), 32-48.
Criteria 4 - The study
implements a modified
version of PCIT
6
Graziano, P. A., Bagner, D. M., Sheinkopf, S. J.,
Vohr, B. R., Lester, B. M. (2012). Evidencebased intervention for young children born
premature: Preliminary evidence for associated
changes in physiological regulation. Infant
Behavior & Development, 35(3), 417-428.
Criteria 6 - The study does
not measure the efficacy of
PCIT in relation to child
behaviour.
7
Lanier, P., Kohl, P. L., Benz, J., Swinger, D.,
Moussette, P., Drake, B. (2011). Parent-child
interaction therapy in a community setting:
Examining outcomes, attrition, and treatment
setting. Research on Social Work Practice, 21(6),
689-698.
Criteria 5 - The study did not
have a clinical control group
of participants who did not
receive PCIT.
1
27
Doctorate in Educational and Child Psychology
Abigail Miranda
8
Lyon, A. R., & Budd, K. S. (2010). A community
mental health implementation of Parent–child
Interaction Therapy (PCIT). Journal of child and
family studies, 19(5), 654-668.
Criteria 5 - The study did not
have a clinical control group
of participants who did not
receive PCIT.
9
Matos, M., Bauermeister, J. J., & Bernal, G.
Criteria 4 - The study
implements a modified
(2009). Parent‐Child Interaction Therapy for
Puerto Rican Preschool Children with ADHD and version of PCIT
Behavior Problems: A Pilot Efficacy Study. Family
process, 48(2), 232-252.
10
Naik-Polan, A. T., & Budd, K. S. (2008). Stimulus
Generalization of Parenting Skills during ParentChild Interaction Therapy. Journal of Early &
Intensive Behavior Intervention, 5(3).
.
Criteria 8 - Participants have
been selected on the basis
that they have been
subjected to, or at risk of
abuse.
11
Phillips, J., Morgan, S., Cawthorne, K., & Barnett,
B. (2008). Pilot evaluation of Parent–Child
Interaction Therapy delivered in an Australian
community early childhood clinic setting.
Australian and New Zealand Journal of
Psychiatry, 42(8), 712-719.
Criteria 5 - The study is a
single case design.
12
Solomon, M., Ono, M., Timmer, S., & GoodlinCriteria 4 - The study
Jones, B. (2008). The effectiveness of parent–
implements a modified
child interaction therapy for families of children on version of PCIT
the autism spectrum. Journal of Autism and
Developmental Disorders, 38(9), 1767-1776.
13
Tiano, J. D., Grate, R. M., & McNeil, C. B. (2013).
Comparison of Mothers' and Fathers' Opinions of
Parent–Child Interaction Therapy. Child & Family
Behavior Therapy, 35(2), 110-131.
Criteria 5 - The study did not
have a clinical control group
of participants who did not
receive PCIT.
14
Timmer, S. G., Zebell, N. M., Culver, M. A., &
Urquiza, A. J. (2010). Efficacy of Adjunct InHome Coaching to Improve Outcomes in
Parent—Child Interaction Therapy. Research on
Social Work Practice, 20(1), 36-45.
Criteria 6 - The study does
not report means and
standard deviations of
behavioural measures.
28
Doctorate in Educational and Child Psychology
Abigail Miranda
Appendix 2
Summary of Studies
Author and Aim
(relevant to review)
Sample
Intervention
Bagner, D. M., and
Eyberg, S. M.
(2007).
USA
Treatment group
Quasireceived PCIT once experimental
a week, 1 hour per design
week.
QuasiEach family was
Experimental:
seen by two
Untreated control
therapists; Bagner
group design with
(2 cases) and 8
dependent pretest
graduate clinical
and posttest
and child
samples
psychology
students with prior
Randomly assigned
training and
to WL/ IT group
experience as a
using two computer
PCIT therapist.
generated random
numbers lists, one
Waitlist control
for boys and one
group received no
for girls. The list
input from the study was controlled by
at the time of pre
the second author
and post measures. who was not
To compare the
effectiveness of
PCIT on families
with children
displaying
significant
externalising
behaviour
problems, with
comorbid learning
needs (described
as mental
retardation in the
study), with a
waitlist control
group.
3 – 6 years
Treatment group
mean age: 52.40m
WL control mean
age: 55.87m
77% boys
30 participants
All diagnosed with
ODD.
60% Mild Mental
Retardation (IQ
score WISC)
Referral
80% Paediatrician
10% Teacher
10% self-referral
Design
Measures
Outcomes
Child Behavior
Checklist for 11
2 to 5 Year Olds
(CBCL; Achenbach
and Rescorla,
2000).
ANCOVA used to
analyse data.
Eyberg Child
Behavior Inventory
(ECBI; Eyberg
and Pincus, 1999).
Dyadic Parent–
Child Interaction
Coding System
(DPICS; Eyberg,
Nelson, Duke and
Boggs,
2004).
Compliance – pre
intervention means
63.88 (19.22)
differed significantly
from post
intervention means,
f (1, 18) = 9.68, p
<.05
ECBI Intensity –
pre and post means
differed
significantly, f (1,
19) = 13.00, p < .05
ECBI Frequency –
pre and post means
did not differ
significantly,
29
Doctorate in Educational and Child Psychology
involved in
recruitment or
assessment. Odd
numbers were
assigned to
treatment group,
even numbers were
assigned to the
waitlist control
group.
Excluded ASD,
major sensory
impairment,
suspected child
abuse cases
67% Caucasian,
17% African
American, 13%
biracial, and 3%
Hispanic,with a
mean Hollingshead
(1975) score of
41.30 (14.14)
Leung, C., Tsang,
S., Heung, K., and
Yiu, I. (2009).
53 (treatment) + 77
(control)
Ages 2- 8
To compare the
effectiveness of
PCIT on Chinese
families with
children displaying
significant
externalising
behaviour
problems, with a
waitlist control
group.
Referred by
hospitals,
social service
agencies,
preschools, or
primary schools,
or who were selfreferred because of
concerns about
their child’s
behaviour.
Participants had to
Abigail Miranda
Treatment sessions
were conducted
once per week and
lasted
approximately 1 hr.
.
Administered by
PCIT trained social
workers, in
Cantonese who
adhered to CAARE
manual guidelines.
Sessions were
implemented until
parents achieved
‘mastery’ over their
Quasiexperimental
design
QuasiExperimental:
Untreated control
group design with
dependent pretest
and posttest
samples
Post hoc tests to
control for
participant
differences.
F (1, 19) = 2.68, p <
.05.
These results
suggest that PCIT
was effective in
improving child
compliance and
intensity of problem
behaviours, but not
the number of
problem
behaviours.
Eyberg Child
Behavior Inventory
(ECBI; Eyberg
and Pincus, 1999).
ANCOVA used to
analyse data.
Bonferroni
correction applied.
Dyadic Parent–
Child Interaction
Coding System:
Abbreviated
Version
ECBI Intensity –
pre and post means
differed significantly
F(1, 106) = 60.90, p
< .001
ECBI Problem– pre
and post means
differed significantly
F(1, 106) = 70.55, p
<
.001
30
Doctorate in Educational and Child Psychology
reach the clinical
cut-off point on the
ECBI, which was
adjusted in this
case to match the
cultural norms of
the target
population.
Abigail Miranda
skills, and the
child’s behaviour
fell within the
normal range on
parent self-report
methods.
Compliance
F(2, 26) = 31.84, p
< .05
These results
suggest that PCIT
was effective in
improving child
compliance,
intensity and
number of problem
behaviours.
Significant between
group differencecs
in gender, χ2(1, N =
110) = 4.44, p <
.05, with more
females in the
intervention than
control group.
McCabe, K., and
Yeh, M. (2009).
USA: 58 3-7 year
olds.
To compare the
effectiveness of
PCIT, a culturally
adapted version of
PCIT and treatment
as usual, for a
sample of Mexican
American children
with clinically
significant
behavioural
problems.
PCIT – N = 19
(mean age = 48.9
months)
Control – N = 18
(mean age = 55.1
months)
Participants had to
reach the clinical
cut-off point on the
ECBI.
Groups did not
PCIT – Therapists
actively coached
parents using
detailed session
checklists outlined
in the PCIT
treatment manual.
Therapists were
allowed an
unlimited number of
sessions.
Treatment as usual
– parents were
assigned to
therapists who
Randomised
controlled design.
Pre and post data
collection
Alternative
intervention control
group compared to
intervention group
Child Behavior
Checklist for 11
2 to 5 Year Olds
(CBCL; Achenbach
and Rescorla,
2000).
Eyberg Child
Behavior Inventory
(ECBI; Eyberg
and Pincus, 1999).
Dyadic Parent–
Child Interaction
Coding System
(DPICS; Eyberg,
Effect sizes were
calculated for pre
and post measures
for each condition,
however this was a
within groups
comparison rather
than a between
groups comparison.
See coding protocol
for calculated effect
sizes between
groups.
Tukey post hoc
31
Doctorate in Educational and Child Psychology
differ significantly in
their demographic
characteristics.
PCIT M= 13.42,
(8.03); Control
M=10.94, 10.01),
F(2, 57)=62, p<.50
were not trained in
PCIT who used
approaches
described as
‘person centred
cognitive
behavioural,’
‘trauma focussed
cognitive
behavioural,’ and
‘family systems.’
Therapists were
allowed freedom in
the approach that
they used.
Therapists were
allowed an
unlimited number of
sessions.
Bagner, D. M.,
Graziano, P. A.,
Jaccard, J.,
Sheinkopf, S. J.,
Vohr, B. R., and
Lester, B. M (2012)
28 mothers and
their 20- to 60monthold
child who was born
< 37 weeks
gestation.
PCIT – one session
per week, one hour
in length.
To compare the
effectiveness of
PCIT on families
with children
displaying
significant
externalising
82% White, 10%
Biracial, 4% African
American, and 4%
Asian, and 21% of
children
were Hispanic.
Sessions were
videotaped, and
50% were randomly
selected and coded
by a separate
therapist for
integrity and fidelity.
Abigail Miranda
Randomised
controlled design.
Waitlist control
group as a base for
comparison.
Two computergenerated random
numbers lists,
one for boys and
one for girls, were
maintained by a
statistician
Nelson, Duke and
Boggs,
2004).
tests were applied.
Eyberg Child
Behavior Inventory
(ECBI; Eyberg
and Pincus, 1999).
Intensity scale
Multiple regression
analysis carried
out.
ECBI intensity
scores suggest a
significant
difference within
groups at pre and
post treatment
t =14 7.01, p <.01
32
Doctorate in Educational and Child Psychology
behaviour problems
who were born
premature, with a
waitlist control
group.
Hollingshead
(1975) score M=
43.39 (13.21),
middle range of
socio economic
status
Abigail Miranda
uninvolved in
recruitment,
intervention
delivery,
and data collection
PCIT – n = 14 Age
39.7 months (14.2)
WL – n = 14 Age
36.5 months (13.0)
Referred by
neonatal clinic
director, health
professionals or
self referred.
For study inclusion,
mothers had to rate
their child
above the clinically
significant range on
the
externalizing
problems scale of
the Child Behavior
Checklist
(CBCL), and be
able to speak and
understand English
Excluded from
study if a major
33
Doctorate in Educational and Child Psychology
Abigail Miranda
sensory impairment
was present,
significant motor
impairments,
oxygen
dependence for
chronic lung
disease, and
autism.
Bagner, D. M.,
Sheinkopf, S. J.,
Vohr, B. R., and
Lester, B. M (2010)
To compare the
effectiveness of
PCIT on families
with children
displaying
significant
externalising
behaviour problems
who were born
premature, with a
waitlist control
group.
28 mothers and
their 18- to 60monthold
child who was born
< 37 weeks
gestation.
PCIT – n = 14 Age
39.7 months (14.2)
WL – n = 14 Age
36.5 months (13.0)
Referred by
neonatal clinic
director, health
professionals or
self-referred.
For study inclusion,
mothers had to rate
their child
above the clinically
significant range (T
score _60) on the
PCIT – one session
per week, one hour
in length.
5 Child Directed
Intervention
sessions
Received treatment
until the child
achieved within ½
SD on the ECBI
Randomised
controlled design.
Waitlist control
group as a base for
comparison.
Two computergenerated random
numbers lists,
one for boys and
one for girls, were
maintained by a
statistician
uninvolved in
recruitment,
intervention
delivery,
and data collection
Child Behavior
Checklist for 11
2 to 5 Year Olds
(CBCL; Achenbach
and Rescorla,
2000).
Eyberg Child
Behavior Inventory
(ECBI; Eyberg
and Pincus, 1999).
Dyadic Parent–
Child Interaction
Coding System
(DPICS; Eyberg,
Nelson, Duke and
Boggs,
2004).
Cohen’s d effect
sizes were
calculated.
Results indicated
significant
improvements on
the ECBI intensity
and frequency
scales, as well as
Child compliance
(see coding
protocol for more
details).
34
Doctorate in Educational and Child Psychology
Abigail Miranda
externalizing
problems scale of
the Child Behavior
Checklist
(CBCL)
35
Doctorate in Educational and Child Psychology
Appendix 3
Coding Protocols 1- 5
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology
Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:___AM______________________
Date:__01.02.14__
Full Study Reference in proper format
Bagner, D. M., & Eyberg, S. M. (2007). Parent–
child interaction therapy for disruptive behavior in children with mental
retardation: A randomized controlled trial. Journal of Clinical Child and
Adolescent Psychology, 36(3), 418-429.
Intervention Name (description of study):_Parent
Child Interaction Therapy (PCIT)____
Study ID Number:__01______
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants) – blocked on sex
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
36
Doctorate in Educational and Child Psychology
Total size of sample (start of study): _30__
Intervention group sample size:__15__
Control group sample size:__15__
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see
following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes parent report and observation
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good,
consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
37
Doctorate in Educational and Child Psychology
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high – explicitly stated in text
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence – Levene’s test. This was not significant for all but one
subtest, indicating that differences in results between groups were due to the non-homogeneity of the
groups.
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings:_Attrition was 33% in the treatment group, 20% for WL group (not significant
difference between groups). Intent to treat analysis showed significant differences in the ECBI
scores of IT & WL groups, F(1, 29) = 5.79, p = .023, d = .67. This showed that disruptive
behaviour scores were lower in the IT group._
Overall Level of Evidence _2_
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
38
Doctorate in Educational and Child Psychology
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis- ANCOVA
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N – to get an effect size of <.97, a sample size n=14 per condition is needed.
Calculated using Gpower.
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key
construct.
at least 75%
50-74%
25%-49%
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for
each key construct.
at least 75%
50-74%
25%-49%
less than 25%
39
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Externalising
behaviours
Outcome 2
Frequency of
problem
behaviours
Outcome 3
Child
compliance
during
parent-child
social
interaction
Primary vs
Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who Changed
What
Changed
Source
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Treatment
Information
Intervention
vs. waitlist
control
Intervention
vs. waitlist
control
Intervention
vs. waitlist
control
Outcome
Measure Used
Reliability
Effect Size
Child
Behavior
Checklist
(CBCL)
Test- retest
.87- .90
Internal
consistency
.82
(externalising)
and .95 (total)
Test- retest .80
- .85 (12
weeks) & .75 .75 (10
months)
Cohen’s d = .97,
p < 0.05
Eyberg
Child
Behavior
Inventory
(ECBI)
Intensity
scale
Dyadic
Parent
Child
Interaction
Coding
System
(DPICS)
Internal
consistency
.90
Inter rater
reliability
kappa .55- .89.
Effect size
data taken
from the study.
Cohen’s d = 1.50,
p < 0.05
Effect size
data taken
from the study.
Cohen’s d = 1.53,
p >. 0.05
Effect size
data taken
from the study.
40
Doctorate in Educational and Child Psychology
Non-Significant
Outcomes
Outcome 1
Number of
problem
behaviours
Primary vs
Secondary
Primary
Secondary
Unknown
Who was targeted
for change
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
What was
targeted for
change
Behaviour
Attitude
Knowledge
Other
Unknown
Source
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Note null/ negative
outcomes
Outcome
Measure Used
Reliability
Effect Size
Cohen’s d = -.66,
p = 0.12
Eyberg
Child
Behavior
Inventory
(ECBI)
Problem
scale
Test- retest
.80 - .85
(12 weeks)
& .75 - .75
(10
months)
Cohen’s d
= -.66,
p > 0.05
There was no
significant difference
between the
Intervention & control
groups for the number
of problem behaviours,
post intervention
Internal
consistenc
y .91
Effect size
data taken
from the
study.
41
Doctorate in Educational and Child Psychology
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology
Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:___AM______________
Date:__01.02.14__
Full Study Reference in proper format:
Leung, C., Tsang, S., Heung, K., & Yiu, I. (2009).
Effectiveness of Parent—Child interaction therapy (PCIT) among Chinese
families. Research on Social Work Practice, 19(3), 304-313.
Intervention Name (description of study):__Parent Child Interaction Therapy (PCIT)____
Study ID Number:__02____
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants) – non randomised control trial as groups were
recruited from different sources
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): _130_
Intervention group sample size:_48_
Control group sample size:__62__
42
Doctorate in Educational and Child Psychology
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see
following table for a detailed breakdown on the outcomes)
Yes – however, no data on reliability for DPICS noted. Therefore, reliability = .94 + .93/ 3 = .62
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes parent report and observation
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good,
consider any cultural considerations)
Yes validated with specific target group – validated in a previous study by the authors on the target
population
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
43
Doctorate in Educational and Child Psychology
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high – explicitly stated in text
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post) – 15.38% attrition overall
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence _2_
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
44
Doctorate in Educational and Child Psychology
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable – Bonferroni correction
Sufficiently large N – required 40 participants overall to get an effect size of <.49. Calculated
using Gpower
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key
construct.
at least 75%
50-74%
25%-49%
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for
each key construct.
at least 75%
50-74%
25%-49%
less than 25%
45
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Frequency of
problem
behaviours
Outcome 2
Number of
problem
behaviours
Outcome 3
Child
compliance
during
parent-child
social
interaction
Primary vs
Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who Changed
What
Changed
Source
Treatment
Information
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Outcome
Measure Used
Eyberg Child
Behavior
Inventory
(ECBI)
Intensity scale
Reliability
Effect Size
Internal
consistency
.94
Cohen’s d =
1.59, CI 1.15 1.71
Eyberg Child
Behavior
Inventory
(ECBI)
Problem scale
Internal
consistency
.93
Dyadic Parent
Child
Interaction
Coding
System
(DPICS)
Inter-rater
reliability not
noted
Effect size
data taken
from the study.
Cohen’s d =
1.52, CI 1.08 1.43
Effect size
data taken
from the study.
Partial etasquared = .49,
p < 0.05
Effect size
data taken
from the study.
Data for
control group
was
unavailable to
calculate
Cohen’s d
effect size.
46
Doctorate in Educational and Child Psychology
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology
Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:____AM________
Date:__01.02.14____
Full Study Reference in proper format:
McCabe, K., & Yeh, M. (2009). Parent–child
interaction therapy for Mexican Americans: A randomized clinical trial. Journal of
Clinical Child & Adolescent Psychology, 38(5), 753-759.
Intervention Name (description of study):___
Parent Child Interaction Therapy (PCIT)__
Study ID Number:__03__
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis) – text says ‘randomly assigned’ but doesn’t have further information on how
this assignment was carried out
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): _58_
Intervention group sample size:_19__
47
Doctorate in Educational and Child Psychology
Control group sample size:_18__
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see
following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good,
consider any cultural considerations)
Yes validated with specific target group – (Eyberg & Pincus, 1999; Garcia-Tornel et al.,
1998)
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
48
Doctorate in Educational and Child Psychology
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high – explicitly stated in text
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_no differences between groups apart from PPQ scores, non-significant after
Familywise error rate control (Bonferroni correction)__
Overall Level of Evidence _2_
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
49
Doctorate in Educational and Child Psychology
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis – p values were not reported. Within group effect sizes were
calculated in the paper, rather than between groups which would have been more relevant to the
hypotheses.
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key
construct.
at least 75%
50-74%
25%-49% - however the effect is negative
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for
each key construct.
at least 75%
50-74%
25%-49%
less than 25%
50
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Externalising
behaviours
Primary vs
Secondary
Primary
Secondary
Unknown
Who Changed
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
What
Changed
Behaviour
Attitude
Knowledge
Other
Unknown
Source
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Treatment
Information
Intervention
vs. waitlist
control
Outcome
Measure Used
Child
Behavior
Checklist
(CBCL)
Reliability
Effect Size
Internal
consistency
.91 - .94
Cohen’s d =
-.79,
p > 0.05
Effect size
calculation=
between
groups mean
difference/poo
led standard
deviation
51
Doctorate in Educational and Child Psychology
Non-Significant
Outcomes
Outcome 2
Frequency of
problem
behaviours
Outcome 3
Number of
problem
behaviours
Outcome 4
Child
compliance
during cleanup
Primary vs
Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who Changed
What
Changed
Source
Treatment
Information
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Outcome
Measure Used
Eyberg Child
Behavior
Inventory
(ECBI)
Intensity scale
Reliability
Effect Size
Internal
consistency
.86
Cohen’s d =
-.49, p < 0.05
Eyberg Child
Behavior
Inventory
(ECBI)
Problem scale
Internal
consistency
.80
Dyadic Parent
Child
Interaction
Coding
System
(DPICS)
Internal
consistency
Kappa
coefficients
(pretreatment) .59
- .85
Effect size
calculation=
between
groups mean
difference/poo
led standard
deviation
Cohen’s d =
-.36, p < 0.05
Effect size
calculation=
between
groups mean
difference/poo
led standard
deviation
Cohen’s d =
.35, p < 0.05
Effect size
calculation=
between
groups mean
difference/poo
led standard
deviation
52
Doctorate in Educational and Child Psychology
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology
Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:__AM__
Date:__01.02.14__
Full Study Reference in proper format:
Bagner, D. M., Graziano, P. A., Jaccard, J.,
Sheinkopf, S. J., Vohr, B. R., & Lester, B. M. (2012). An initial investigation of
baseline respiratory sinus arrhythmia as a moderator of treatment outcome for
young children born premature with externalizing behavior problems. Behavior
therapy, 43(3), 652-665.
Intervention Name (description of study):_
Parent Child Interaction Therapy (PCIT)____
Study ID Number:__04___
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): _28__
Intervention group sample size:_14_
53
Doctorate in Educational and Child Psychology
Control group sample size:_14_
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see
following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good,
consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
54
Doctorate in Educational and Child Psychology
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high – explicitly stated in text
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings__ No significant difference between Time 1 ECBI Intensity score,
t(26)=0.43, p=.668.__
Overall Level of Evidence _2_
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
55
Doctorate in Educational and Child Psychology
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N - minimum sample of 9 per group was sufficient to detect expected
differences.
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key
construct.
at least 75%
50-74%
25%-49%
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for
each key construct.
at least 75%
50-74%
25%-49%
less than 25%
56
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Frequency of
problem
behaviours
Primary vs
Secondary
Primary
Secondary
Unknown
Who Changed
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
What
Changed
Behaviour
Attitude
Knowledge
Other
Unknown
Source
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Treatment
Information
Intervention
vs. waitlist
control
Outcome
Measure Used
Reliability
Effect Size
Eyberg
Child
Behavior
Inventory
(ECBI)
Intensity
scale
Test-retest
reliability
coefficient
of .80 (12
weeks) and
.75
(10 months)
Cohen’s d =
-1.03, CI -1.87
- -0.19
Internal
consistency
coefficient of
.95
Effect size
calculation=
between
groups mean
difference/poo
led standard
deviation
57
Doctorate in Educational and Child Psychology
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology
Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:__AM____
Date:__01.02.14__
Bagner, D. M., Sheinkopf, S. J., Vohr, B. R., &
Lester, B. M. (2010). Parenting intervention for externalizing behavior problems
in children born premature: an initial examination. Journal of developmental and
behavioral pediatrics: JDBP, 31(3), 209.
Full Study Reference in proper format:__
Intervention Name (description of study):_
Parent Child Interaction Therapy (PCIT)___
Study ID Number:__05__
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): _28_
Intervention group sample size:__11_
Control group sample size:__14__
58
Doctorate in Educational and Child Psychology
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see
following table for a detailed breakdown on the outcomes)
Yes - .78 when reliability scores were averaged
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good,
consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
59
Doctorate in Educational and Child Psychology
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high – explicitly stated
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_ No significant differences at time 2, F(1,25) = 24.2, p =.000, d =1.8._
Overall Level of Evidence _2_
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
60
Doctorate in Educational and Child Psychology
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N – power calculation by the authors suggest that a minimum sample of 9
per group was sufficient to detect expected differences.
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key
construct.
at least 75%
50-74%
25%-49%
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for
each key construct.
at least 75%
50-74%
25%-49%
less than 25%
61
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Externalising
behaviours
Outcome 2
Frequency of
problem
behaviours
Outcome 3
Number of
problem
behaviours
Outcome 4
Child
compliance
during
parent-child
social
interaction
Primary vs
Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who Changed
What
Changed
Source
Treatment
Information
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Intervention
vs. waitlist
control
Child
Teacher
Parent/Sig.A
Ecology
Other
Unknown
Behaviour
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Outcome
Measure Used
Child
Behavior
Checklist
(CBCL)
Reliability
Effect Size
internal
consistency
.54 - .81.
Cohen’s d =
2.3, p < 0.05
Eyberg Child
Behavior
Inventory
(ECBI)
Intensity scale
internal
consistency
.90
Eyberg Child
Behavior
Inventory
(ECBI)
Problem scale
internal
consistency
.85.
Dyadic Parent
Child
Interaction
Coding
System
(DPICS)
Kappa .50
(noncomply)
to 0.78
(praise)
Effect size
data taken
from the study
Cohen’s d =
2.3, p < 0.05
Effect size
data taken
from the study
Cohen’s d =
1.4, p < 0.05
Effect size
data taken
from the study
Cohen’s d =
.90, p < 0.05
Effect size
data taken
from the study
62
Doctorate in Educational and Child Psychology
Appendix 4
Weighting of Studies
A: Methodological Quality
The Kratochwill (2003) coding protocol was used to weight experimental and
quasi-experimental designs; Studies were weighted on ‘Measures’, ‘Comparison
Group’ and ‘Analysis’.
Measures
Weighting
High
Medium
Low
Description
All measures used produce reliable scores of 0.85 for the
population under study.
Data collected using multiple methods AND from multiple sources.
A case for validity must be presented.
Study uses at least 75% of measures that produce reliable scores
of 0.70 for the population under study.
Data collected using either multiple methods AND/OR from multiple
sources.
A case for validity does not need to be presented.
Study uses at least 50% of measures that produce reliable scores
of 0.50 for the population under study.
Data collected using either multiple methods AND/OR from multiple
sources, however this is not a requirement.
A case for validity does not need to be presented.
Comparison Group
Weighting
Description
Uses at least one type of ‘active’ comparison group.
Establishes initial group equivalency.
High
Evidence of counterbalancing of change-agents and demonstrates
equivalent mortality and low attrition at the post test.
Medium
Low
Uses at least a ‘no intervention group’ comparison group.
Evidence for at least two of: 1) Counterbalancing of change-agents,
2) Group equivalency, 3) Equivalent mortality with low attrition.
Conducted an intent-to-intervene analysis if equivalent mortality is
not demonstrated, with non-significant results
Uses a comparison group.
Evidence for at least one of: 1) Counterbalancing of changeagents, 2) Group equivalency, 3) Equivalent mortality with low
63
Doctorate in Educational and Child Psychology
attrition.
Conducted an intent-to-intervene analysis if equivalent mortality is
not demonstrated, with non-significant results
Analysis
Weighting
High
Medium
Low
Description
Appropriate statistical analysis conducted including: appropriate
unit of analysis, familywise/ experimentwise error rate controlled (if
applicable) and a sufficiently large N.
Significant outcomes for at least 75% of the total primary outcome
measures
Three of the following:
Appropriate statistical analysis conducted including: 1) appropriate
unit of analysis, 2) familywise/ experimentwise error rate controlled
(if applicable) and 3) a sufficiently large N.
Significant outcomes for at least 50% of the total primary outcome
measures
Two of the following:
Appropriate statistical analysis conducted including: 1) appropriate
unit of analysis, 2) familywise/ experimentwise error rate controlled
(if applicable) and 3) a sufficiently large N.
Significant outcomes for at least 25% of the total primary outcome
measures
Overall Methodological Quality
Overall methodological quality of studies was calculated by assigning scores of:
- ‘3’ for ‘High’ weightings
- ‘2’ for ‘Medium’ weightings
- ‘1’ for ‘Low’ weightings
Scores were then averaged.
Overall Methodological
Quality
High
Medium
Low
Average Scores
At least 2.5
Between 1.5 and 2.4
Less than 1.4
64
Doctorate in Educational and Child Psychology
Studies
Measures
Comparison
Group
Analysis
Bagner
and
Eyberg,
2007
Weighting
score
Medium
2
Medium
2
High
3
Overall
Quality of
Methodology
Medium
2.3
Leung et
al., 2009
Weighting
score
Low
1
Medium
2
High
3
Medium
2
McCabe
and Yeh,
2009
Weighting
score
Medium
2
Medium
2
Low
1
Medium
1.7
Bagner et
al., 2012
Weighting
score
Low
1
Medium
2
Low
1
Low
1.3
Bagner et
al., 2010
Weighting
score
Medium
2
Medium
2
Low
1
Medium
1.7
B: Relevance of Methodology
This weighting reviews the appropriateness of the evidence for answering the
review question.
Weighting
High
Description
Uses random assignment of participants in an attempt to
establish group equivalence pre intervention.
Uses an active comparison group to measure the efficacy
of the intervention in comparison with an alternate
intervention.
Medium
Uses random assignment of participants in an attempt to
establish group equivalence pre intervention.
Uses a waitlist control comparison group to compare the
effects of the intervention with treatment as usual.
Low
Does not have to use random assignment of participants.
Uses a comparison group.
Demonstrates group equivalence.
C: Relevance of Evidence to Review Question
65
Doctorate in Educational and Child Psychology
This weighting is a review-specific judgement about the relevance of the focus
of the evidence for the review question.
Weighting
High
Medium
Low
Description
Uses the PCIT intervention delivered by a trained
therapist..
Uses children with externalising behaviour problems
which are significant at clinical cut-offs on CBCL/ ECBI.
Triangulates data using self- report and observation in
order to assess observable and perceived behavioural
changes.
Uses the PCIT intervention delivered by a trained
therapist...
Uses children with externalising behaviour problems
which are significant at clinical cut-offs on CBCL/ ECBI.
Triangulates data using self- report and observation in
order to assess observable and perceived behavioural
changes.
Uses the PCIT intervention delivered by a trained
professional...
Uses children with externalising behaviour problems
which are not significant at clinical cut-offs on CBCL/
ECBI.
Does not have to triangulate data using self- report and
observation
D: Overall Weight of Evidence
This is an overall assessment of the extent to which the study contributes
evidence to answer the review question. It is calculated by giving studies scores:
- ‘3’ for ‘High’ weightings
- ‘2’ for ‘Medium’ weightings
- ‘1’ for ‘Low’ weightings
These scores were then averaged to find the overall weight of evidence score.
Overall Methodological
Quality
High
Medium
Low
Average Scores
At least 2.5
Between 1.5 and 2.4
Less than 1.4
66
Doctorate in Educational and Child Psychology
Studies
(C)
Relevance
(A)
(B)
of
Quality of
Relevance of evidence
Methodology Methodology
to the
review
question
Medium
Medium
Medium
2.3
2
2
(D)
Overall
Weight
of
Evidence
Bagner
and
Eyberg,
2007
Weighting
score
Medium
2.1
Leung et
al., 2009
Weighting
score
Medium
2
Low
1
Low
1
Low
1.3
McCabe
and Yeh,
2009
Weighting
score
Medium
1.7
High
3
High
3
High
2.6
Bagner et
al., 2012
Weighting
score
Low
1.3
Medium
2
Low
1
Low
1.4
Bagner et
al., 2010
Weighting
score
Medium
1.7
Medium
2
Medium
2
Medium
1.9
67