Doctorate in Educational and Child Psychology
Rheanne Jalali
Case Study 1: An Evidence-Based Practice Review Report
Theme: Interventions for children with Special Educational Needs
How effective is the Incredible Year’s parenting programme at improving
parenting behaviour towards children with Conduct Problems/Disorder and/or
Oppositional Defiant Disorder?
Summary
The Incredible Years is a series of parenting programmes designed to encourage
positive parenting skills and reduce negative conduct behaviours in children.
Depending on intensity and child age, parent training can last up to twelve weeks;
using a structured sequence of topics and activities to develop children’s social,
emotional competence.
This literature review aimed to evaluate the effectiveness of the Incredible Years
parenting programme at improving parent behaviours towards children with Conduct
Problems, Conduct Disorder and/or Oppositional Defiant Disorder. A systematic
literature search was conducted using three databases. Five selected studies were
analysed using a coding protocol adapted from Kratochwill (2003), with quality of
evidence determined by Harden & Gough (2012) criteria. This review found that
there was moderate evidence to suggest the effectiveness of the Incredible Years in
improving positive parenting, reducing negative parenting and improving social and
emotional well-being. Specific methodological limitations are explored and further
research has been suggested to strengthen findings within this area.
Doctorate in Educational & Child Psychology
Introduction
Description of the Incredible Years Parenting Programme
The Incredible Years (IY) is a series of preventative and targeted programmes aimed
at reducing challenging behaviour and building social and emotional competence in
children who display signs of Conduct problems (CP) and/or Conduct disorder (CD)
(Brotman et al., 2003; Brotman et al., 2005; Scott et al., 2010; The Incredible Years,
2013). Originally designed from a videotape modelling intervention, the IY
programme has expanded to include multiple levels of intervention (e.g. basic and
advanced), age groups (e.g. infancy, pre-school, school age) and training schemes
(e.g. parent, teacher and child). Additionally, these programmes target a wide range
of behavioural difficulties including Oppositional Defiant Disorder (ODD) and
Attention Deficit Hyperactivity Disorder (ADHD) (Linares, Montalto, Li & Oza, 2006;
The Incredible Years, 2013).
The goal of training is to improve parenting practice by encouraging appropriate
discipline, positive reinforcement, social skills training and play interaction, promoting
the child’s social and emotional competence (defined as cognitive understanding of
emotions and social reactions) (Cline & Frederickson, 2009; Goleman, 1996; Kim,
Cain & Webster-Stratton, 2008). This aims to prevent the formation of later conduct
problems
(Posthumus, Raaijmakers, Maassen, van Engeland & Matthys, 2012;
Webster-stratton, Reid, & Hammond, 2001).
Parent training lasts for up to twelve weeks, consisting of a structured sequence of
topics (e.g. positive behaviour and aggression strategies) using a collaborative
approach between facilitator and parent (Beauchaine, Webster-Stratton, & Reid,
2005). Facilitators use vignettes and video interaction to encourage modelling, group
1
Doctorate in Educational & Child Psychology
discussion, problem solving and self-management (Gardner, Hutchings, Bywater, &
Whitaker, 2010; Webster-stratton et al., 2001; The Incredible Years, 2013). Initially
targeted at families of low socio-economic status (increased risk factors), the
parenting programmes are culturally and economically sensitive, available
internationally and distributed by non-profit voluntary organisations (Gross et al.,
2003).
Psychological Basis
Developmental models postulate an interaction of multiple influences on the
development of child conduct problems. One of these influences is negative
parenting (Brotman et al., 2009). The IY programme is consistent with social
interactional learning models, depicting that behaviour patterns occur within the
home from primary socialisers (Brotman et al., 2003; Patterson, 1986). The IY
intervention is based on the premise that children’s negative behaviours are learnt
from parents through reinforcement. This assumption is based on social learning
theory and behaviourist psychology principles. Such theories suggest that all
behaviours are produced by experiences of response consequences, resulting in
learned behaviours over time (Akers, Krohn, Lanza-Kaduce, & Radosevich, 1979;
Cline & Frederickson, 2009; Miller, 1994). Conduct problem behaviours establish
during early childhood from negative interchanges between parent and child
(Brotman et al., 2005; Cunningham & Boyle, 2002). These behavioural difficulties
escalate with developmental challenges (e.g. transition to schooling) where the child
is unable to cope with social and academic pressures (Campbell, Shaw & Gilliom,
2000; Patterson, 1986). Within school and home environments, children learn to
escape pressures by escalating their negative behaviour. As part of the social
2
Doctorate in Educational & Child Psychology
interactional learning model, Patterson labels this a ‘coercive process’ (Patterson,
Reid & Dishion, 1992).
Intervention primarily involves changing social discourses in the family environment
(Webster-Stratton, 1984; Webster-Stratton, Kolpacoff & Hollinsworth, 1988). As
primary socialisers, parent training aims to change children’s behavioural difficulties
within the family context (Miller, 1994). Parents learn social-learning principles
whereby appropriate behaviours are positively reinforced and inappropriate
behaviours discouraged. Many studies have confirmed the effectiveness of the
behavioural parent training method (McCart, Priester, Davies, & Azen, 2006).
The specific methods used in the IY are equally supported by psychological theory.
Within the IY programme, parents watch video clips of child-adult interactions
facilitating group discussion, problem solving and the sharing of ideas. The origins of
videotape modelling relate to Attachment Theory principles in which the goal is to
illuminate the affective nature of parent-child interactions. Video tape modelling can
support changes in parenting behaviour by providing opportunities to observe
affective, nurturing interactions and identify the responses that produce positive
outcomes. In the IY programme, parents experiment initiations with their own child
through a scaffolding approach, fostering the development of warm and positive
relationships (Bowlby, 2008;
The Incredible Years, 2013; Webster-Stratton &
Hammond, 1997; Wood, Bruner & Ross, 1976). Research has indicated the
effectiveness of video tape modelling methods (Webster-Stratton, 1984; WebsterStratton et al, 1988; Webster-stratton, 1992; Webster-Stratton & Hammond, 1997;
Webster-Stratton, 1998).
3
Doctorate in Educational & Child Psychology
Rationale
The incidence of child conduct problems is an issue shown to be increasing
internationally (Posthumus et al., 2012; Webster-Stratton et al., 2001; Websterstratton, Reid & Hammond, 2004). As aggressive behaviours become increasingly
resistant to change, longitudinal studies suggest that intervention should occur at the
earliest point possible. Furthermore, without early intervention, behavioural
difficulties such as ODD and CP/CD result in entrenched patterns by eight years of
age, leading to later difficulties (e.g. peer rejection, academic failure and criminality)
(Eron, 1990; Fossum, Mørch, Handegård, Drugli, & Larsson, 2009; Miller, 1994;
Snyder, 2001).
In the UK, such outcomes highlight the importance of preventive and targeted
intervention to children displaying conduct difficulties. Government strategies have
focused on early intervention for children since the introduction of the ‘Every Child
Matters’ agenda (DfES, 2003).
Typically, Educational Psychologist (EP) involvement will occur when the child is
presenting behavioural difficulties within school. When considering the psychological
development of conduct difficulties, one could suggest that such involvement is too
late to eradicate all negative behaviours. However, research suggests that parenting
interactions are one of the most important protective/ risk factors for early onset
conduct behavioural problems, with parent training a key mechanism for change
(Miller, 1994; Webster-stratton et al., 2004). This gives insights into the ways that
EPs can support the child and reverse the impact of some behavioural difficulties,
promoting long lasting change. A meta-analysis of the IY by Menting, Orobio de
Castro & Matthys (2013) demonstrated the effectiveness of the parent training in
4
Doctorate in Educational & Child Psychology
improving child CP/CD and/or ODD behaviours. However, as noted by the authors a
premise of the IY is that the child’s behaviour is ultimately influenced by the parent’s.
In considering the potential benefits of parent training, analysing specific
improvements in parental behaviours and well-being is an area yet to be explored.
By targeting and modifying parental practices at the source, it is expected that there
will be longer lasting improvements in the child, particularly for later school life
(Menting et al., 2013). With this in mind, this systematic literature review provides an
appraisal of the evidence base for the effectiveness of the IY parenting programme
in improving parental behaviours towards children with CP/CD and/or ODD.
Review Question
How effective is the Incredible Year’s Parenting Program at improving parenting
behaviour towards children with conduct problems/disorder and/or oppositional
defiant disorder?
Critical Review of the Evidence Base
A comprehensive literature search was carried out on the 30th December 2013, to
locate all studies relevant to the research question. A search was undertaken in
three databases (ERIC, Medline and PsycINFO) using the search terms outlined in
Table 1.
Table 1
Search Terms Applied
Database
Search Term
5
Doctorate in Educational & Child Psychology
ERIC
Contains ‘intervention’ (all fields) AND
‘conduct’ (all fields) AND ‘parent*’ (all
fields)
Medline
Contains ‘intervention’ (all fields) OR
‘incredible years’ (all fields) AND
‘conduct’ (all fields) AND ‘parent*’ (all
fields)
PsycINFO
Contains ‘intervention’ (title) OR
‘incredible years’ (abstract) AND
‘parent*’ (abstract) AND ‘conduct’ (title)
OR ‘conduct problem*’(title)
This review aimed to develop a research area suggested by Menting et al. (2013).
The searches were limited so that only studies in peer-reviewed journals (to control
for research quality) and written in English (translation resources were not available)
were included in search results. Studies containing group-based Incredible Years
parenting interventions, pre-post data and the use of an intervention and control
group were necessary for their appropriateness to the research question. This is in
line with evidence hierarchies (Guyatt et al., 1995). However in considering the
availability of studies, it was decided that randomisation need not be necessary.
Removing duplicates, the relevant search pages were screened for appropriate
studies, initially via title and then by abstract. A flow diagram of the study selection
process is illustrated below. Reasons for exclusion are outlined in Appendix 1. Of
the 5 studies chosen, 3 were identified by Medline, 1 by PsycINFO and 1 by ERIC
although there was overlap in presence across databases. The 5 studies that met
the inclusion criteria for critical analysis are presented in Table 3 below.
Diagram 1
Flow Diagram of Full Literature Search
6
Doctorate in Educational & Child Psychology
Papers identified from search terms
(PsycINFO, ERIC & Medline)
n = 620
Excluded based on Title
n - 600
Abstracts Assessed
n = 20
Excluded based on Abstract
(Using Inclusion Criteria)
n-2
Full Text Assessed
n = 18
Excluded based on Inclusion Criteria
n - 13
Studies included in synthesis
n=5
7
Doctorate in Educational & Child Psychology
Table 2
Inclusion & Exclusion Criteria
Inclusion Criteria
Exclusion Criteria
Type of
Article
The article is in a peer
reviewed journal. As these
studies are assessed using a
stringent criteria,
methodological rigour is
ensured.
The article is not in a peer-reviewed
journal.
Language
The study is written in English.
This is due to a lack of
resources for translation.
The study is not written in English.
Type of data
The study must contain
primary empirical data. This
ensures originality of findings.
The study does not contain primary
empirical data e.g. Meta-analyses.
Intervention
The study contains a condition
whereby the parent IY training
is solely implemented as part
of the study. Children within
this condition receive no
additional intervention
alongside their parents. This
ensures accuracy of findings.
The study does not contain a
condition whereby the parent IY
training is solely implemented as
part of the study. Children within this
condition receive additional
interventions alongside their
parents.
Measures
The study measures
(internal/external)
improvements in parenting
behaviour.
The study does not measure
improvements in parenting
behaviour.
Design
The study is a controlled
design with pre and post
(and/or follow up) intervention
measures of parenting
behaviour. This can ensure
accurate detection of
improvements in parenting
behaviour from the IY training.
The study is not a controlled design
with pre and post (and/or follow up)
measures of parenting behaviour.
8
Doctorate in Educational & Child Psychology
Control
Group
Participants
Children
The study uses a wait list
control group. In comparing
the findings of the effects of
the IY training to the
comparison group, this type of
control ensures a greater
accuracy, as participants will
have similar characteristics,
but receiving no additional
intervention (as of yet).
The study does not use a wait list
control group.
The study uses an alternative
intervention as its only control
measure.
The study’s participants are
children aged 2 – 9. This
targets a wide range of
negative behaviours before
they become potentially
entrenched. The study uses
participants who have been
labelled as having CP/CD
and/or ODD.
The study uses children older than 9
years and/or without a label of
CP/CD and/or ODD.
Parents
The study includes parent(s)
The study does not include parents.
that are mother, father or both;
biological, foster and/or step
parents.
Analysis
The study reports quantitative
analysis. This allows
confounding variables to be
separated out and ensures a
focus on improvements in
parenting behaviour.
The study does not report
quantitative analysis in reference to
improvements in parenting
behaviour.
Year of
Publication
The study uses the
intervention called the
Incredible Years.
The study uses an older version of
IY known as the Videotape
Modelling Programme (VMP).
9
Doctorate in Educational & Child Psychology
Table 3
List of Included Studies
Included Studies
1.
Gardner, F., Burton, J., & Klimes, I. (2006). Randomised controlled trial of a
parenting intervention in the voluntary sector for reducing child conduct
problems: outcomes and mechanisms of change. Journal of Child Psychology
and Psychiatry, 47(11), 1123-1132.
2.
Hutchings, J., Bywater, T., Daley, D., Gardner, F., Whitaker, C., Jones, K.,
Eames, C., & Edwards, R. T. (2007). Parenting intervention in Sure Start
services for children at risk of developing conduct disorder: pragmatic
randomised controlled trial. BMJ: British Medical Journal, 334(7595), 678-682.
3.
Larsson, B., Fossum, S., Clifford, G., Drugli, M. B., Handegård, B. H., &
Mørch, W. T. (2009). Treatment of oppositional defiant and conduct problems
in young Norwegian children. European Child & Adolescent Psychiatry, 18(1),
42-52.
4.
Letarte, M. J., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent
training program “Incredible Years” in a child protection service. Child Abuse &
Neglect, 34(4), 253-261.
5.
Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating children
with early-onset conduct problems: Intervention outcomes for parent, child,
and teacher training. Journal of Clinical Child and Adolescent Psychology,
33(1), 105-124.
Comparison of Selected Studies
The five studies were summarised (Appendix 2) and compared on methodological
features. Of the selected studies, 4 were randomised controlled trials (Gardner,
Burton & Klimes, 2006; Hutchings et al., 2007; Larsson et al., 2009; WebsterStratton et al., 2004) and one used a controlled design without random assignment
to conditions (Letarte, Normandeau & Allard, 2010).
10
Doctorate in Educational & Child Psychology
The critical appraisal for quality and relevance framework, (Weight of Evidence,
WoE) developed by Harden & Gough (2012) is designed to guide evaluation by
weighting three aspects of each chosen study; Methodological Quality (WoE A),
Methodological Relevance (WoE B) and Topic Evidence (WoE C). Within this review,
weightings were numerically rated and averaged to create an Overall Weight of
Evidence (WoE D) (Gough, 2004). This framework permits an objective judgement in
determining each studies value and appropriateness to the review question. The
WoE framework is shown in Table 4. The WoE for each study is shown in Table 5.
To assess the quality of methodology (WoE A), the UCL Educational Psychology
Literature Review Coding Protocol (adapted from APA Task Force on Evidence
Based Interventions in School Psychology was used) (Kratochwill, 2003). These
coding guidelines contain clear and extensive criteria, designed specifically for the
type of studies included in this review. Within these guidelines, the quality of
measures, comparison group and statistical analyses were examined. Completed
coding protocols can be found in Appendix 3. Further information about the WoE
framework is in Appendix 4.
11
Doctorate in Educational & Child Psychology
Table 4
Framework for Weight of Evidence (Harden & Gough, 2012)
Weight of
Evidence
A
Quality of
execution of the
study in relation to
quality standards
for studies of that
type
(Methodological
Quality)
Weight of
Evidence
B
Weight of
Evidence
C
Weight of
Evidence
D
Appropriateness of
research design for
addressing Review
Question
(Methodological
Relevance)
Appropriateness of
focus of study to
Review Question
(Topic Relevance)
Considering A, B &
C to rate the overall
degree to which the
study contributes in
answering the
Review Question
(Overall weight of
evidence)
Table 5
Weight of Evidence for each Study
Weighting
Score
Studies
Topic
Methodological Methodological
Relevance
Quality A
Relevance B
C
Overall
Weight of
Evidence D
Gardner, Burton &
Klimes
(2006)
Medium
High
High
High
Hutchings et al
(2007)
Medium
High
Medium
Medium
Larsson et al
(2009)
Medium
Medium
Medium
Medium
Low
Low
Medium
Medium
Medium
Medium
Letarte, Normandeau
& Allard
(2010)
Webster-Stratton, Reid
& Hammond (2004)
Low
Medium
12
Doctorate in Educational & Child Psychology
Critical Review
Outcome measures.
Improvements in parenting behaviour were defined as increased positive parenting,
reduced negative parenting and increased parent emotional wellbeing. Reported
effect sizes used Cohen’s d and Eta-squared (n2) (calculated to 2dp), judged
according to relevant criterion (Cohen, 1988; Ferguson, 2009). See appendix 2 for
full outcome measures.
Improvements in positive parenting behaviour.
Positive parenting can be defined as nurturing and supportive behaviours that
encourage children’s social and emotional development. Such behaviours include
praise, boundary setting, appropriate discipline and positive affect (Webster-Stratton,
1989; Webster-Stratton, 1998). Although measured differently, all studies reported
statistically significant differences in positive parenting at post intervention (see table
6). Of the five studies, three achieved a medium weighting and one a high weighting
for overall quality (Gardner et al., 2006; Hutchings et al., 2007; Larsson et al., 2009 &
Webster-Stratton et al., 2014). . Effect sizes varied from small to high with one
medium weighted study reporting sustained effectiveness at six months follow up
(Hutchings et al., 2007). In the higher weighted study, smaller effect sizes were
reported at post intervention (Gardner et al., 2006). This was similarly reported by
Letarte et al. (2010). However due to this study’s low rating for methodological
quality/relevance, less weighting should be given to findings.
Studies by Webster-Stratton et al. (2004), Hutchings et al. (2007) and Gardner et al.
(2006)
scored
medium
methodological
quality
and
medium
to
high
13
Doctorate in Educational & Child Psychology
methodological/topic relevance. Given their overall quality (WoE D), greater
consideration should be given to their findings. From this, evidence suggests a small
to moderate effect of the IY programme in improving positive parenting behaviour.
Table 6
Effect sizes for Positive Parenting
Study
Outcome*
Effect sizes
Pre-Post
Gardner,
Burton &
Klimes
(2006)
Observed Positive
Parenting
Larsson et
al
(2009)
Pre-Follow Up
Descriptor
small
Cohen’s d
= .38
No effect sizes
moderate
Self-reported
Parenting Skill
Hutchings
et al
(2007)
Descriptor
Overall
Quality
High
reported
Cohen’s d
= .65
moderate
Observed & SelfReported Positive
Parenting
Cohen’s d
= .57
Self-Reported
Positive Parenting
Mother:
Cohen’s d
= 1.44
moderate
Cohen’s d = .62
Medium
Parental
behaviour not
assessed
Medium
No follow up
Low
high
high
Father:
Cohen’s d
= 2.24
Letarte,
Normande
au &
Allard
(2010)
n2 = .03
small
Self-Reported
Praise &
Incentives.
Self-Reported
Appropriate
n2 = .08
small
14
Doctorate in Educational & Child Psychology
Study
Outcome*
Effect sizes
Pre-Post
Descriptor
Pre-Follow Up
Overall
Quality
Descriptor
Discipline
small
n2 = .07
Self-Reported
Behaviour
Monitoring
Self-Reported
Positive Verbal
Discipline
WebsterStratton,
Reid &
Hammond
(2004)
smallmoderate
n2 = .18
moderate
Observed & SelfReported Positive
Parenting
Mother:
Cohen’s d
= .51
No effect sizes
reported
Medium
small
Father:
Cohen’s d
= .35
*see appendices for references
Reductions in negative parenting behaviour.
Negative parenting can be described as the parenting behaviours that may
contribute to a child’s behavioural difficulties. Negative parenting behaviours may
include the
use of harsh or inconsistent discipline, physical punishment, verbal
criticism and laxness (Arnold, O'Leary, Wolff & Acker, 1993; Gardner et al., 2006;
Webster-stratton et al., 2004). Four out of five studies reported statistically significant
differences in parent negative behaviours after the IY programme (see table 7). In
comparing results between negative and positive parenting, it appears that there are
larger effects of the IY intervention in reducing negative parent behaviours. Studies
by Gardner et al. (2006) and Webster-Stratton et al. (2004) reported moderately
15
Doctorate in Educational & Child Psychology
larger effect sizes than in positive parenting. Interestingly, the study by Hutchings et
al. (2007) reported a non-significant effect at post intervention but a significant,
moderate effect size at follow up. Significance differences may be due to the
different type of statistical analyses used at post intervention and follow up. In
placing more weighting on the studies that scored higher in overall quality (WoE D);
there is moderate evidence to suggest that the IY is effective in reducing parent’s
negative behaviours.
There are a number of possible reasons why reported effect sizes for negative
parenting behaviour appear larger than positive parenting. Research indicates that
child conduct behaviours arise from negative parental interchanges (Brotman et al.,
2005; Cunningham & Boyle, 2002), implying that during intervention, parents may
have an increased awareness of their negative behaviours rather than positive.
Subsequently, a reduction in negative behaviours would more likely be reported.
Table 7
Effect Sizes for Negative Parenting
Study
Outcome*
Effect sizes
Pre-Post
Descriptor
Pre-Follow Up
Gardner,
Burton &
Klimes
(2006)
Observed & SelfReported Reduced
Negative Parenting
Cohen’s d =
.74
moderatelarge
No effect sizes
reported
Hutchings
et al
(2007)
Observed & SelfReported Reduced
Negative Parenting
Not significant
Cohen’s d =
.32
small
Significant
Cohen’s d =
.58
Overall
Quality
Descriptor
High
moderate
Medium
16
Doctorate in Educational & Child Psychology
Larsson et
al
(2009)
Self-Reported
Reduced Harsh
Discipline
Self-Reported
Reduced
Inconsistent
Discipline
Mother:
Cohen’s d =
.61
moderate
Father:
Cohen’s d =
.82
high
Mother:
Cohen’s d =
1.05
high
Father:
Cohen’s d =
.67
high
Letarte,
Normande
au &
Allard
(2010)
Self-Reported
Reduced Harsh
Discipline.
n2 = .04
small
Self-Reported
Reduced Physical
Punishment
n2 = .01
small
WebsterStratton,
Reid &
Hammond
(2004)
Observed & SelfReported Reduced
Negative Parenting
Mother:
Cohen’s d =
.81
high
Father:
Cohen’s d =
.51
moderate
Parental
behaviour not
assessed
Medium
No follow up
Low
No effect
sizes reported
Medium
*see appendices for references
Increased emotional well-being.
Studies also detected changes in parents’ emotional wellbeing (table 8), defined as
stress levels, feelings of depression and perceived competence (Gardner et al.,
2006; Hutchings et al., 2007).
Stress.
17
Doctorate in Educational & Child Psychology
Using the Parenting Stress Index (Abidin, 1990) two studies measured stress levels.
Both studies reported significantly moderate to high effect sizes in the reduction of
parent’s stress with the study by Hutchings et al. (2007) reporting moderate to large
effects at follow up. The study by Larsson et al. (2009) reports sex differences
between parents in perceived stress. Study findings by Hutchings et al. (2007)
should be given greater consideration due to the diverse nature of measures used.
In considering this, it appears that only one study gives a reliable indication of the
effectiveness of the IY in improving parent stress levels. In terms of generalisation,
findings should be treated with caution.
Depression.
Using the Beck Depression Inventory (Beck, Ward, Mendelson, Mock & Erbaugh,
1961) two studies measured parent’s feelings of depression. One study reported a
non-significant effect of the IY (Gardner et al., 2006) the other study reporting small
to moderate effects at post intervention and follow up (Hutchings et al., 2007).
Although both studies obtained medium to high WoE D, the mix of significant and
non-significant findings (within the same measure) suggest limited evidence in the
effectiveness of the IY in improving depression.
Competence.
One study used the term self-efficacy (relating to parental confidence) and reported
non-significant findings (Letarte et al., 2010). Across studies, competence was
measured using different instruments, raising concerns regarding construct validity
levels. Two studies measured competence, reporting small to large effect sizes
(Gardner et al., 2006; Hutchings et al., 2007). Despite a medium/high overall quality
18
Doctorate in Educational & Child Psychology
(WoE D), differences in effect sizes suggest insufficient evidence to report the
effectiveness of the IY in improving sense of competence.
Table 8.
Effect Sizes for Emotional Well-being
Study
Relevant
Measures*
Effect sizes
Post
Gardner,
Burton &
Klimes
(2006)
Hutchings
et al
(2007)
Beck Depression
Inventory
Parenting Scale
Parent Sense of
Competence
Competence
Cohen’s d = .4
(small-moderate)
Parenting Scale
Parenting Stress
Index
Beck Depression
Inventory
Stress
Cohen’s d = .66
(moderate)
Overall
Quality
Follow Up
No effect sizes
reported
High
Depression
Not Significant
Cohen’s d = .35
(small)
Depression
Cohen’s d = .48
(small-moderate)
Competence
Cohen’s d = .95
(high)
Stress
Cohen’s d = .79
(moderatelarge)
Medium
Depression
Cohen’s d = .51
(moderate)
Competence
Cohen’s d= 1.18
(high)
Larsson et
al
(2009)
Parenting Practices
Interview (PPI)
Parent Stress Index
Stress
Mother: Cohen’s d =
.67 (moderate)
Father Cohen’s d =
.86 (high)
Parental behaviour
not assessed
Medium
Letarte,
Normandea
u & Allard
(2010)
Parenting Practices
Interview (PPI)
Parenting Selfefficacy Measure;
Parenting Self-
Self-efficacy
Not significant
n2 = .02
(small)
No follow up
Low
19
Doctorate in Educational & Child Psychology
Study
Relevant
Measures*
Effect sizes
Post
Overall
Quality
Follow Up
Agency Measure
& Maternal
Confidence in
Toddlerhood
WebsterStratton,
Reid &
Hammond
(2004)
Composite scores:
Parenting Practices
Interview (PPI)
Parent Discipline
Inventory (DDI)
Calculated via
composite scores
No follow up
Medium
*see appendices for references
Participants.
The studies were carried out in different countries; United Kingdom (UK), Canada,
United States of America and Norway. Studies that were conducted within the UK
were given medium/ high weightings for topic relevance. Additionally, these studies
had sufficient sample sizes, strengthening the reliability of findings. In judging effect
sizes, there is evidence to suggest that the IY is effective in improving UK parental
behaviour (Gardner et al., 2006; Hutchings et al., 2007). Three non UK studies had
insufficient samples sizes (Larsson et al., 2009; Letarte et al., 2010; WebsterStratton et al., 2004), scoring lower on statistical analyses (WoE A) and
methodological quality (WoE B). This may limit the reliability of effect sizes reported.
All participants were referred as having CP/CD and/or ODD. One study included
children with learning difficulties and ADHD as well participants with CP (Letarte et
al., 2010). Due to differences in sample, this study obtained a low weighting for topic
relevance, posing difficulties in generalising the effectiveness of the IY program to all
children with CP/CD and/or ODD.
20
Doctorate in Educational & Child Psychology
As stated in inclusion criteria, participants ranged from 2-9 years of age with only
one study using a concentrated age range of 3-4 years (Hutchings et al., 2007). This
indicates that the intervention was carried out for children in different developmental
stages. Research has highlighted that conduct behaviours can become entrenched
through age, introducing additional difficulties such as peer rejection (Eron, 1990;
Miller, 1994; Snyder, 2001). These may influence the level of intervention and parent
behaviours, meaning that improvements may not be consistent across all ages. The
study by Hutchings et al. (2007) focused on pre-school children and reported
relatively moderate effect sizes. As the study obtained a medium quality weighting,
one could suggest the IY parenting programme is more effective for improving parent
behaviour with younger age groups. With studies that recruited wider age ranges, it
is uncertain whether the effectiveness of the IY can be generalised as improving
behaviours in all parents.
The referral process for the IY differed across studies. Recruitment of children and
parents were based upon referral by professionals (Gardner et al., 2006; Larsson et
al., 2009; Letarte et al., 2010), parent requests (Webster-Stratton et al., 2004), and
clinical scores (Hutchings et al., 2007). It could be considered that referral by
professional judgement is the most objective way of selecting children with CP/CD
and/or ODD for treatment. This was one of many aspects in the study by Gardner et
al. (2006) that obtained a high weighting for WoE C. Studies that relied on parental
request could introduce confounding variables meaning that reported improvements
could be exaggerated. Such differences affected studies’ weighting of topic
relevance (WoE C) (Webster-Stratton et al., 2004).
In four out of five studies, the IY parenting intervention was aimed at ‘treating’
children. In comparing effect sizes of the medium/higher weighted studies it appears
21
Doctorate in Educational & Child Psychology
that the IY intervention is effective as a targeted treatment particularly in reducing
negative parenting. However, is this finding due to the nature of targeted treatments?
One could suggest that targeted interventions are remedial in correcting existing
behaviours, whilst preventative interventions aim to enhance positive environments.
Only one study used the IY intervention as a preventative measure (Hutchings et al.,
2007). The effect sizes given within this study support this assumption, reporting a
non-significant reduction in negative parenting but a moderate improvement in
positive parenting. Further research would need to be conducted to verify this. The
use of the IY as a preventative measure affected the study’s WoE C, scoring at a
medium level.
Research Design.
Four out of the five studies randomly assigned participants to conditions. Evidence
derived from randomised controlled trials has been identified as having greater
reliability (Guyatt et al., 1995). This strengthens the accuracy of findings, particularly
with medium to higher weighted studies (Gardner et al., 2006; Hutchings et al., 2007;
Larsson et al., 2009; Webster-stratton et al., 2004). In one study (Letarte et al., 2010)
random assignment was not used as participants had to register interest. This may
have impacted on participant motivation, contributing to the lower weighting given for
methodological relevance, (WoE B).
Control Group.
Examining effectiveness by a wait list control meant that all participants would be
equally eligible for intervention, sharing similar behavioural characteristics. This
enabled a fairer starting point at pre intervention, ensuring greater accuracy in
comparing scores. However the nature of a wait list control may have also influenced
22
Doctorate in Educational & Child Psychology
effect sizes. The use of a wait list control may introduce additional moderator
variables e.g. expectancy effects (participants may behave worse to receive the
intervention quicker) (Baker-Henningham, 2011; Barker, Pistrang & Elliott, 2003).
This could question the reliability of effects sizes. Furthermore, expectancy effects
may have contributed to the lack of strong evidence in increased positive parenting.
Measures.
The studies all used a range of measurements to ascertain improvements in
parenting behaviour. Although some measures overlapped between studies, there
appeared to be no relationship between measures and effect sizes.
As stated in the Kratochwill’s coding protocol (WoE A) (see appendix 3) studies were
given greater weighting if they used a range of measurements from a range of
sources. Similarly stated in WoE B and C criteria studies were allocated higher
weightings if they used multiple, diverse measures of parent behaviour. The use of
observation was significant in meeting high weighting criteria as it enables a “direct
objective measure that reduces bias” (p.6 Hutchings et al., 2007). Some studies met
this criterion, scoring high for methodological relevance (Gardner et al., 2006;
Hutchings et al., 2007.) One study used observation but obtained medium weighting
due to the number of intervention groups (Webster-Stratton et al., 2004). As well
using observation, studies given medium/high weightings also used a range of selfreport measures. Two studies used pure self-report methods to measure
improvements, with one study relying upon self-report measures from one source
(Larsson et al., 2009; Letarte et al., 2010). These were therefore allocated low
weightings (Letarte et al., 2010). Although self-report reliability statistics were
reported as medium to strong, the accuracy and credibility of self-report methods can
be criticised. This is due to the subjective nature of participant response as there can
23
Doctorate in Educational & Child Psychology
be a wish to please the researcher as well as self-enhancement purposes (Barker et
al., 2003; Delroy, Vazire & Vazire, 2007). Whilst all studies used some form of selfreport method (particularly for measuring emotional well-being) studies that
triangulated these measures gained increased accuracy in detecting behavioural
improvements. Studies that relied on self-report lacked credibility checks. Although
parental behaviours were the only interest in this review, it should be considered that
a lack of multi-method and multi-source data collection leads to a single dimension
conceptualisation of outcomes. Thus, caution should be placed in generalising these
study results.
Conclusion & Recommendations
Conclusion
Four out of five studies reviewed suggest that the IY parenting programme is
moderately effective in improving positive parent behaviour, reducing negative
behaviour and improving emotional well-being. Specifically, effect sizes indicate that
the IY is more effective in reducing negative parenting behaviour than other
behavioural domains. Three studies reviewed support these findings through the
use of objective measures (Gardner et al., 2006; Hutchings et al., 2007; Websterstratton et al., 2004). Overall three studies obtained medium for overall quality
(Hutchings et al., 2007; Larsson et al., 2009; Webster-stratton et al., 2004) with one
obtaining a high weighting (Gardner et al., 2006). Although the study by Hutchings et
al. (2007) was given high weighting for methodological relevance, the overall
medium weighting (WoE D) derived from limitations in WoE A and C.
A number of methodological issues have been discussed within this critical review,
influencing the reliability of findings.
Firstly, due to the diverse age range of
24
Doctorate in Educational & Child Psychology
participants, it is uncertain whether the IY’s effectiveness can be generalised to
children of all ages. Although wait list control groups are well grounded in ethical
responsibility, the difficulties in controlling confounding variables limit accuracy
levels. These difficulties also exist in relying upon self-report measures. Such
confounding variables may have contributed to the apparent stronger effects of the
IY on negative parenting.
Additionally, only one study made the statistical relation that improvements in
parenting behaviour produce improvements in child behaviour. Achieving a higher
quality weighting, this study supports the hypotheses made by Menting et al. (2013),
reinforcing the effectiveness of parent training methods (Hutchings, Lane & Gardner,
2004).
Recommendations
Based on a review of the findings there is evidence to suggest that the IY parenting
programme is effective in reducing negative parenting behaviours as well as
improving some aspects of parent emotional well-being. As most of the studies
reviewed used the IY programme to ‘treat’ conduct difficulties (rather than prevent),
recommendations for this programme are as a targeted intervention for younger age
groups, as according to research, negative behaviours are not yet entrenched at this
stage (Snyder, 2001).
Although there was some evidence to suggest positive preventative effects, further
research comparing targeted versus preventative intervention would need to be
conducted (particularly in measuring negative parenting behaviours). Additionally,
future reviews would benefit focusing on particular age categories, discovering if
there is a ‘critical time’ for child and parent behaviour improvement. Such findings
may impact upon the setting the intervention is delivered in. In considering the
25
Doctorate in Educational & Child Psychology
relationship between parent and child interchanges, studies need to make more
statistical links between parent and child improvements. This would be highly useful
for strengthening the effectiveness of the IY in promoting whole family improvement.
26
Doctorate in Educational & Child Psychology
References
Abidin, R. R. (1990). Parenting stress index. Psychological Assessment Resources
Inc. Odessa: FL.
Abidin, R. R. (1995). Parenting stress index, 3rd Edition. Psychological Assessment
Resources Inc. Odessa: FL.
Akers, R. L., Krohn, M. D., Lanza-Kaduce, L., & Radosevich, M. (1979). Social
Learning and Deviant Behavior: A specific test of a general theory. American
Sociological Review, 636–655.
American Psychiatric Association. (1994). Diagnostic and Statistical Manual of
Mental Disorders (4th Edition). Washington, DC.
Arnold, D., O’Leary, S., Wolff, L., & Acker, M. (1993). The Parenting Scale: A
measure of dysfunctional parenting and discipline situations. Psychological
Assessment, 5, 137–144.
Baker-Henningham, H. (2011). Transporting Evidence-Based Interventions Across
Cultures: Using focus groups with teachers and parents of pre-school children to
inform the implementation of the Incredible Years Teacher Training Programme in
Jamaica. Child: Care, Health and Development, 37(5), 649–61.
27
Doctorate in Educational & Child Psychology
Barker, C., Pistrang, N., & Elliott, R. (2003). Research Methods in Clinical
Psychology: An Introduction for Students and Practitioners. Oxford: John Wiley &
Sons.
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An
Inventory For Measuring Depression. Arch Gen Psychiatry; 4, 561-571.
Beck, A.T. (1972). Measuring depression: The depression Inventory. Washington:
Government Printing Office.
Beauchaine, T. P., Webster-Stratton, C., & Reid, M. J. (2005). Mediators,
Moderators and Predictors of 1-Year Outcomes Among Children Treated for Earlyonset Conduct Problems: a latent growth curve analysis. Journal of consulting and
clinical psychology, 73(3), 371–88.
Bowlby, J. (2008). Attachment: Second Edition. New York: Basic books.
Brestan, E. V., Jacobs, J. R., Rayfield, A. D., & Eyberg, S. M. (2000). A Consumer
Satisfaction Measure for Parent-Child Treatments and its Relation to Measures of
Child Behavior Change. Behavior Therapy, 30(1), 17-30.
Brotman, L. M., Gouley, K. K., Chesir-Teran, D., Dennis, T., Klein, G., & Shrout, P.
(2005). Prevention for Preschoolers at High Risk for Conduct Problems : Immediate
Outcomes on Parenting Practices and Child Social Competence. Journal of Clinical
Child & Adolescent Psychology, 34(4), 724–734.
28
Doctorate in Educational & Child Psychology
Brotman, L. M., Klein, R. G., Kamboukos, D., Brown, E. J., Coard, I., & Sosinsky, L.
S. (2003). Preventive Intervention for Urban , Low-Income Preschoolers at Familial
Risk for Conduct Problems : A Randomized Pilot Study. Journal of Clinical Child and
Adolescent Psychology, 32(2), 246–257.
Brotman, L. M., O’Neal, C. R., Huang, K.-Y., Gouley, K. K., Rosenfelt, A., & Shrout,
P. E. (2009). An Experimental Test of Parenting Practices as a Mediator of Early
Childhood Physical Aggression. Journal of Child Psychology and Psychiatry, and
Allied Disciplines, 50(3), 235–45.
Campbell, S. B., Shaw, D. S., & Gilliom, M. (2000). Early Externalizing
Behaviour Problems: Toddlers and Pre-schoolers at Risk for Later Maladjustment.
Development & Psychopathology, 12, 467–488.
Cline, T., & Frederickson, N. (2009). Special Educational Needs, Inclusion and
Diversity. Buckingham: Open University Press.
Cohen, J. (1988). Statistical Power Analysis for the Behavioural Sciences. Hillsdale
New Jersey: Lawrence Erlbaum.
Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112(1), 155-159.
Cunningham, C. E., & Boyle, M. H. (2002). Pre-schoolers at Risk for Attention-Deficit
Hyperactivity Disorder and Oppositional Defiant Disorder: Family, parenting, and
behavioural correlates. Journal of Abnormal Child Psychology, 30, 555–569.
29
Doctorate in Educational & Child Psychology
Delroy, L., Vazire, P., & Vazire, S. (2007). The Self-Report Method. In R. W. Robins,
R. C. Fraley, & R. F. Krueger (Eds.), Handbook of Research Methods in Personality
Psychology (pp. 224–239). New York: Guilford Press.
Department for Education and Skills. (2003). Every Child Matters. London: The
Stationery Office.
Dumka, L. E., Stoerzinger, H. D., Jackson, K. M., & Roosa, M. W. (1996).
Examination of the Cross-Cultural and Cross-Language Equivalence of the
Parenting Self-agency Measure. Family Relations, 45, 216–222.
Eron, L. D. (1990). Understanding Aggression. Bulletin of the International Society
for Research on Aggression, 12, 5–9.
Eyberg, S., & Ross, A. W. (1978). Assessment of Child Behaviour Problems: the
validation of a new inventory. Journal of Clinical Child Psychology, 7, 113-116.
Eyberg, S. M., & Robinson, E. A. (1981). Dyadic Parent-Child Interaction Coding
System. Parenting Clinic, Seattle: University of Washington.
Ferguson, C. J. (2009). An Effect Size Primer: A Guide for Clinicians and
Researchers. Professional Psychology: Research and Practice, 40(5), 532–538.
Fossum, S., Mørch, W.-T., Handegård, B. H., Drugli, M. B., & Larsson, B. (2009).
Parent Training for Young Norwegian Children with ODD and CD Problems:
30
Doctorate in Educational & Child Psychology
Predictors and mediators of treatment outcome. Scandinavian Journal of
Psychology, 50(2), 173–81.
Gardner, F., Sonuga-Barke, E., & Sayal, K. (1999). Parents Anticipating
Misbehaviour: An observational study of strategies parents use to prevent conflict
with behaviour problem children. Journal of Child Psychology and Psychiatry, 40,
1185–1196.
Gardner, F., Burton, J., & Klimes, I. (2006). Randomised Controlled trial of a
Parenting Intervention in the Voluntary Sector for Reducing Child Conduct Problems:
Outcomes and Mechanisms of Change. Journal of Child Psychology and Psychiatry,
and Allied Disciplines, 47(11), 1123–32.
Gardner, F., Hutchings, J., Bywater, T., & Whitaker, C. (2010). Who Benefits and
How Does it Work? Moderators and mediators of outcome in an effectiveness trial of
a parenting intervention. Journal of Clinical Child and Adolescent Psychology,
American Psychological Association, Division 53, 39(4), 568–80.
Goleman, D. (1996). Emotional Intelligence: Why It Can Matter More Than IQ.
London: Bloomsbury Publishing.
Gough, D. A. (2004). Systematic Research Synthesis to Inform the Development of
Policy and Practice. In G. Thomas & R. Pring (Eds.), Evidence-based Practice (pp.
44–62). Buckingham: Open University Press.
31
Doctorate in Educational & Child Psychology
Gross, D., & Rocissano, L. (1988). Maternal Confidence in Toddlerhood: It's
measurement for clinical practice and research. Nurse Practioner, 13, 19–29.
Gross, D., Fogg, L., Webster-Stratton, C., Garvey, C., Julion, W., & Grady, J. (2003).
Parent Training of Toddlers in Day Care in Low-income Urban Communities. Journal
of Consulting and Clinical Psychology, 71(2), 261–278.
Guyatt, G., Sackett, D., Sinclair, J., Hayward, R., Cook, D., & Cook, R. (1995).
Users’ Guide to the Medical Literature.IX. A Method for Grading Health Care
Recommendations. Journal of the American Medical Association, 274, 1800–1804.
Harden, A., & Gough, D. (2012). Quality and Relevance Appraisal. In D. Gough, S.
Oliver, & J. Thomas (Eds.), An Introduction to Systematic Reviews (pp. 153–178).
London: Sage.
Hutchings J. (1996).The Personal and Parental Characteristics of Preschool Children
Referred to a Child and Family Mental Health Service and their Relation to
Treatment Outcome [PhD thesis]. Bangor: University of Wales.
Hutchings, J, Lane, E., & Gardner, F. (2004). Making Evidence-based Interventions
Work. In D. Farrington, C. Sutton, & D. Utting (Eds.), Support from the Start: Working
with Young Children and Families to Reduced Risks of Crime and Anti-social
Behaviour. London: DFES.
Hutchings, Judy, Gardner, F., Bywater, T., Daley, D., Whitaker, C., Jones, K.,
Eames, C., & Edwards, R. T. (2007). Parenting Intervention in Sure Start Services
for Children at Risk of Developing Conduct Disorder: Pragmatic randomised
controlled trial. British Medical Journal, 334(7595), 678 – 682.
32
Doctorate in Educational & Child Psychology
Johnston, C., & Mash, E. (1989). A Measure of Parenting Satisfaction and Efficacy.
Journal of Clinical Child Psychology, 18, 167–175.
Jones, L. M. (2000). Maltreating Caregivers: An Application of the Trans-theoretical
Model of Change. Doctoral Thesis. Kingston: University of Rhode Island.
Kennedy, H. (2011). What is Video Interactive Guidance. In H. Kennedy, M. Landor,
& L. Todd (Eds.), Video Interaction Guidance. A Relationship-Based Intervention to
Promote Attunement, Empathy and Wellbeing. (pp. 20–42). London: Jessica
Kingsley.
Kim, E., Cain, K. C., & Webster-Stratton, C. (2008). The Preliminary Effect of a
Parenting Program for Korean American Mothers: A randomized controlled
experimental study. International Journal of Nursing studies, 45(9), 1261–73.
Kratochwill, T. R. (2003). Task Force on Evidence-based Interventions in School
Psychology (Adapted). American Psychology Association. Retrieved 3/01/2014,
from: https://moodle.ucl.ac.uk/mod/folder/view.php?id=790365.
Larsson, B., Fossum, S., Clifford, G., Drugli, M. B., Handegård, B. H., & Mørch, W.T. (2009). Treatment of Oppositional Defiant and Conduct Problems in Young
Norwegian Children : Results of a randomized controlled trial. European Child &
Adolescent Psychiatry, 18(1), 42–52.
Letarte, M.-J., Normandeau, S., & Allard, J. (2010). Effectiveness of a Parent
Training Program “Incredible Years” in a Child Protection Service. Child Abuse &
Neglect, 34(4), 253–61.
33
Doctorate in Educational & Child Psychology
Linares, L. O., Montalto, D., Li, M., & Oza, V. S. (2006). A Promising Parenting
Intervention in Foster Care. Journal of Consulting and Clinical Psychology, 74(1),
32–41.
McCart, M. R., Priester, P. E., Davies, W. H., & Azen, R. (2006). Differential
Effectiveness of Behavioral Parent-training and Cognitive-Behavioral Therapy for
Antisocial Youth: A meta-analysis. Journal of Abnormal Child Psychology, 34(4),
527–43.
Menting, A. T. A., Orobio de Castro, B., & Matthys, W. (2013). Effectiveness of the
Incredible Years Parent Training to Modify Disruptive and Prosocial Child Behaviour:
A meta-analytic review. Clinical Psychology Review, 33(8), 901–913.
Miller, L. S. (1994). Primary Prevention of Conduct Disorder. Psychiatric Quarterly,
65(4), 273-285.
Patterson, G. R. (1986). Performance Models for Antisocial Boys. American
Psychologist, 41(4), 432.
Patterson, G., Reid, J., & Dishion, T. (1992). Antisocial Boys: A social interactional
approach (Vol. 4). Eugene, OR: Castalia.
Posthumus, J. Raaijmakers, M. J., Maassen, G. H., van Engeland, H., & Matthys,
W. (2012). Sustained Effects of Incredible Years as a Preventive Intervention in
Preschool Children with Conduct Problems. Journal of Abnormal Child Psychology,
40(4), 487–500.
34
Doctorate in Educational & Child Psychology
Reid, M. J., Webster-stratton, C., & Baydar, N. (2004). Halting the Development of
Conduct Problems in Head Start Children : The Effects of Parent Training. Journal of
Clinical Child and Adolescent Psychology, American Psychological Association,
Division 53, 33(2), 279–291.
Robinson, E., Eyberg, S., & Ross, A. (1980). The Standardization of an Inventory of
Child Conduct Problem Behaviours. Journal of Clinical Child Psychology, 9, 22–28.
Robinson, E. A., & Eyberg, S. M. (1981).The Dyadic Parent–child Interaction Coding
System: Standardization and validation. Journal of Consulting and Clinical
Psychology, 49, 245–250.
Scott, S., Sylva, K., Doolan, M., Price, J., Jacobs, B., Crook, C., & Landau, S.
(2010). Randomised Controlled Trial of Parent Groups for Child Antisocial Behaviour
Targeting Multiple Risk Factors: The SPOKES project. Journal of Child Psychology
and Psychiatry and Allied Disciplines, 51(1), 48–57.
Snyder, H. (2001). Epidemiology of Offical Offending. In R. Loeber & D. P.
Farrington (Eds.), Child Delinquents: Devlopment, Intervention & Service Needs. (pp.
25–46). California: Sage Publishing.
The Incredible Years. (2013). Incredible Years Series. The Incredible Years Inc.
Retrieved January 25, 2014, from http://incredibleyears.com/about/incredible-yearsseries/
The Incredible Years. (2013). Incredible Years Series. The Incredible Years Inc.
Retrieved April 09, 2014 from http://incredibleyears.com/for-researchers/
35
Doctorate in Educational & Child Psychology
Webster-Stratton, C. (1984). Randomized Trial of Two Parent-training Programs for
Families with Conduct-disordered Children. Journal of Consulting and Clinical
Psychology, 52(4), 666–78.
Webster-Stratton, C., Kolpacoff, M., & Hollinsworth, T. (1988). Self-Administered
Videotape Therapy for Families with Conduct-problem Children: Comparison with
two cost-effective treatments and a control group. Journal of Consulting and Clinical
Psychology, 56(4), 558–66.
Webster-Stratton, C. (1989). Dyadic Parent–Child Interaction Coding System–
Revised. Seattle: University of Washington.
Webster-stratton, C. (1992). Individually Administered Videotape Parent Training:
"Who Benefits? Cognitive Therapy & Research, 16(1), 31–35.
Webster-Stratton, C., & Hammond, M. (1997). Treating Children with Early-onset
Conduct Problems: A comparison of child and parent training interventions. Journal
of Consulting and Clinical Psychology, 65(1), 93–109.
Webster-Stratton, C. (1998). Preventing Conduct Problems in Head Start Children:
Strengthening parenting competencies. Journal of Consulting and Clinical
Psychology, 66(5), 715–30.
Webster-Stratton, C., & Hancock, L. (1998a). Training for Parents of Young Children
with Conduct Problems: Content, Methods & Therapeutic Processes. In J. M.
Briesmeister & C. E. Schaefer (Eds.), Handbook for Parent Training (pp. 98–152).
New York: John Wiley & Sons Inc.
36
Doctorate in Educational & Child Psychology
Webster-stratton, C., Reid, M. J., & Hammond, M. (2001). Preventing Conduct
Problems , Promoting Social Competence: A Parent and Teacher Training
Partnership in Head Start. Journal of Clinical Child and Adolescent Psychology,
American Psychological Association, Division 53, 30(3), 283–302.
Webster-stratton, C., Reid, M. J., & Hammond, M. (2004). Treating Children With
Early-Onset Conduct Problems: Intervention Outcomes for Parent , Child and
Teacher Training. Journal of Clinical Child and Adolescent Psychology, American
Psychological Association, Division 53, 33(1), 105–124.
Wood, D. J., Bruner, J. S., & Ross, G. (1976). The Role of Tutoring in Problem
Solving. Journal of Child Psychiatry and Psychology, 17(2), 89–100.
37
Doctorate in Educational & Child Psychology
Appendices
Appendix 1
Articles from PsycINFO, ERIC & Medline that did not meet inclusion criteria.
Studies
Rationale for Exclusion
1
Brotman, L. M., Klein, R. G., Kamboukos, D., Brown, E.
J., Coard, S. I., & Sosinsky, L. S. (2003). Preventive
intervention for urban, low-income pre-schoolers at
familial risk for conduct problems: A randomized pilot
study. Journal of Clinical Child and Adolescent
Psychology, 32(2), 246-257.
Intervention: The study
does not implement the
parent IYPT solely as part
of the study e.g. Uses
“additional component
strategies to reduce some
of the… risk factors”
(p.248) as part of the
study.
2
Brotman, L. M., Gouley, K. K., Chesir-Teran, D.,
Dennis, T., Klein, R. G., & Shrout, P. (2005).
Prevention for pre-schoolers at high risk for conduct
problems: Immediate outcomes on parenting practices
and child social competence. Journal of Clinical Child
and Adolescent Psychology, 34(4), 724-734.
Intervention (children):
The study contained a
condition whereby the
child participants receive
additional interventions
alongside parents e.g.
Incredible Years Dinosaur
Social Skill groups.
3
Kim, E., Cain, K. C., & Webster-Stratton, C. (2008).
The preliminary effect of a parenting program for
Korean American mothers: A randomized controlled
experimental study. International Journal of Nursing
Studies, 45(9), 1261-1273.
Control Group: Study
does not use wait list
control group.
4
Linares, L. O., Montalto, D., Li, M., & Oza, V. S. (2006).
A promising parenting intervention in foster care.
Journal of Consulting and Clinical Psychology, 74(1),
32-41.
Participants: Do not
have the label CP/CD
and/or ODD
Posthumus, J. A., Raaijmakers, M. A., Maassen, G. H.,
Van Engeland, H., & Matthys, W. (2012). Sustained
effects of incredible years as a preventive intervention
in preschool children with conduct problems. Journal of
Abnormal Child Psychology, 40(4), 487-500.
Control Group: Study
does not use wait list
control group.
Reid, M. J., Webster-Stratton, C., & Baydar, N. (2004).
Halting the development of conduct problems in Head
Start children: The effects of parent training. Journal of
Clinical Child and Adolescent Psychology, 33(2), 279291.
Type of data: Study uses
secondary data from
previous studies.
5
6
38
Doctorate in Educational & Child Psychology
7
Scott, S., Sylva, K., Doolan, M., Price, J., Jacobs, B.,
Crook, C., & Landau, S. (2010). Randomised controlled
trial of parent groups for child antisocial behaviour
targeting multiple risk factors: the SPOKES project.
Journal of Child Psychology and Psychiatry, 51(1), 4857.
Control Group: The
study uses an alternative
intervention as its only
control measure.
8
Webster-Stratton, C. (1984). Randomized trial of two
parent-training programs for families with conductdisordered children. Journal of Consulting and Clinical
Psychology, 52(4), 666-678.
Year of Publication: The
study uses an older
version of the Incredible
Years (VMG).
9
Webster-Stratton, C., Kolpacoff, M., & Hollinsworth, T.
(1988). Self-administered videotape therapy for families
with conduct-problem children: Comparison with two
cost-effective treatments and a control group. Journal
of Consulting and Clinical Psychology, 56(4), 558-566.
Year of Publication: The
study uses an older
version of the Incredible
Years.
10
Webster-Stratton, C. (1992). Individually administered
videotape parent training: “Who benefits?” Cognitive
Therapy and Research, 16(1), 31-52.
Year of Publication: The
study uses an older
version of the Incredible
Years.
11
Webster-Stratton, C., & Hammond, M. (1997). Treating
children with early-onset conduct problems: A
comparison of child and parent training interventions.
Journal of Consulting and Clinical Psychology, 65(1),
93-109.
Year of Publication: The
study uses an older
version of the Incredible
Years.
Webster-Stratton, C. (1998). Preventing conduct
problems in Head Start children: strengthening
parenting competencies. Journal of consulting and
clinical psychology, 66(5), 715-730.
Year of Publication: The
study uses an older
version of the Incredible
Years.
Webster-Stratton, C., Reid, M. J., & Hammond, M.
(2001). Preventing conduct problems, promoting social
competence: A parent and teacher training partnership
in Head Start. Journal of clinical child psychology,
30(3), 283-302.
Control Group: The
study does not use a wait
list control group.
12
13
39
Doctorate in Educational & Child Psychology
Appendix 2
Summary of Studies
Study
Gardner,
Burton &
Klimes (2006)
Sample
Design
Relevant Measures
Findings
UK Children (& relative
parent) aged 2-9
Years, referred for help
with conduct problems
& scored above the
clinical cut off on
Eyberg Problem scale
(Robinson, Eyberg &
Ross, 1980).
Randomised
controlled trial:
Wait list control, pretest, post-test and
follow up measures
(at 6 and 18 months).
Observed positive
parenting (Gardner et al,
1999), Observed
negative parenting,
Parenting Scale (Arnold,
O’Leary, Wolff & Acker,
1993), Parent Sense of
Competence (Johnston &
Mash, 1989), Beck
Depression Inventory
(BDI; Beck, 1972) and
Consumer satisfaction
Therapy Attitude
Inventory (Brestan,
Jacobs, Rayfield &
Eyberg, 1999).
Significant intervention effects on parent
observed use of negative strategies (large
effect size) with an increase in the use of
positive strategies (small effect size).
Parents reported an increased sense of
competence (small effect size) and efficacy
but not on feelings of satisfaction as a
parent.
Self-reported changes in parenting skills
via the Parenting Scale total score and
reactivity scale (which measures harsh
styles of parenting) (medium effect size).
No intervention effects on parent
depression.
75% of parents felt behaviour problems
had improved and 88% felt they had learnt
useful skills.
Changes were maintained at follow up (as
evident by non-significant change within
groups between post and follow up).
Improvements in positive parenting
correlated with improvements in child’s
negative behaviour (r= .40, p = .001)
Randomised
controlled trial (using
block design): Wait
list control, pre-test,
post-test and follow
up measures (at 6
months).
Parenting Stress Index
(Abidin, 1990), Parenting
Scale (Arnold, O’Leary,
Wolff & Acker, 1993),
Beck Depression
Inventory (Beck, Ward,
Mendelson, Mock &
Observation: There was a mean
difference of 9.6 (3.7 to 15.5, p< .002)
between groups at follow up for positive
parenting behaviours (medium effect size).
Levels of parental criticism reduced (not
significant at post-test, p = .076 but at follow
up, p<.002).
Intervention Group: 44
Wait list Control: 32
Hutchings et
al (2007)
153 UK parents with
children aged 3 to 4.11
years at risk of conduct
disorder defined by
scoring above the
clinical cut off on the
Eyberg Behaviour
40
Doctorate in Educational & Child Psychology
Study
Sample
Design
Inventory (Eyberg &
Ross, 1978).
Intervention Group:104
Wait list Control: 49
Relevant Measures
Findings
Erbaugh, 1961),
Personal Data & Health
Questionnaire
(Hutchings, 1996),
Dyadic Parent Child
Interactive Coding
System (Eyberg &
Robinson, 1981).
Significant effects of reported reductions
in stress (p<.001) and depression levels (p<
.008) and improvements in parenting
competencies (p<.001).
Larsson et al
(2009)
127 Norwegian children
(and parent) aged 4-8
years referred due to
oppositional or conduct
problems as indicated
by scoring above the
clinical cut off on the
Eyberg Child Behaviour
Inventory, ECBI
(Robinson & Eyberg,
1981).
Intervention Group: 47
Wait list Control: 28
PT + CT: 52
Randomised
controlled study: Wait
list control (&
additional intervention
group), pre-test, posttest and follow up
measures (at one
year).
Parenting Practices
Interview (PPI) (WebsterStratton, Reid &
Hammond, 2004), Parent
Stress Index (Abidin,
1995), & Consumer
Satisfaction
Questionnaire.
Mothers reported to employ significantly
less harsh child disciplinary practices
(p<.001), significant differences in mother
reports of decreased inconsistent discipline
(p<.001)
Mother and fathers reported to use
significantly more positive parenting after
intervention (p<.001).
Maternal and paternal stress was reported
to be significantly lower following
intervention (p<.05, p=.008).
Moderate to large effect sizes were
reported in these domains.
Letarte,
Normandeau
& Allard
(2010)
45 Canadian parents
(and child) who were
monitored for their child
neglecting behaviours
were identified by their
case workers to
participate.
Intervention Group: 26
Wait list Control: 9
(10 dropped out)
A repeated measures
design: Wait list
control, pre-test and
post-test measures.
Parenting Practices
Interview (PPI) (WebsterStratton, 1998),
Parenting Self-efficacy
Measure; Parenting
Self-Agency Measure,
PSAM (Dumka,
Stoerzinger, Jackson &
Roosa, 1996) and the
Maternal Confidence in
Toddlerhood (Gross &
Significant effects found from PPI in
harsh discipline (p<.05), physical
punishment (p<.05), praise and incentives
(p<.05), appropriate discipline (p<.001),
monitoring (p<.05) and positive verbal
discipline (p<.001). Effect sizes are small to
moderate.
Follow ups reported parents using less
harsh discipline (p<.05), more praise and
incentives (p<.05), more appropriate
discipline (p<.001) and more positive verbal
41
Doctorate in Educational & Child Psychology
Study
WebsterStratton, Reid
& Hammond
(2004)
Sample
Families were recruited
from requesting
treatment at a
parenting clinic. 159
American Children
aged 4-8 years were
selected if parents
reported behaviours
that existed above the
cut off score of the
Eyberg Child Behaviour
Inventory (Robinson et
al, 1980) and if they
met the Diagnostic and
Statistical Manual of
th
Mental Disorders 4
Edition (DSM-IV) (APA,
1994).
Intervention Group: 31
Wait list Control: 26
Other Intervention
Groups:
Parent & Teacher
Training: 24
Child Training: 30
Child & Teacher
Design
Randomised
Controlled Trial: Wait
list control (&
additional intervention
groups), pre-test,
post-test and follow
up measures (at one
year).
Relevant Measures
Findings
Rocissano, 1988) (Jones,
2000).
discipline (p<.05). Effect sizes small to
moderate.
Results showed no differences in parent’s
reported self-efficacy (p = .48)
96.1% of parents showed positive reaction
to the program, 38.5% / 34.6% reported
greater confidence in their ability to solve
problems with their child.
Parenting Positive &
Negative Composite
Scores (calculated from
variables in observational
and self-report measures
given below. Included
self-reported harsh
discipline, ratio of critical
to positive discipline and
observed use of critical
statements). Taken from:
Parenting Practices
Interview (PPI), Dyadic
Parent Child Interactive
Coding System
(Revised) (WebsterStratton, 1989) (Original)
(Robinson & Eyberg,
1981), Coder
Impressions Inventory
for parents (CIIParenting Style) &
Parent Discipline
Inventory (DDI)
(Webster-Stratton &
Spitzer, 1991).
Analysis on composite scores:
Comparisons between treatment and control
suggest that intervention produced
significant improvements on parent
behaviour.
Significant effects of intervention on
mother negative parenting composite score
(p<.001), father negative parenting
composite score (p<.001) and both mother
and father positive parenting composite
scores (p<.01, p<.05 respectively).
Significant effect of intervention on mother
and father’s negative parenting.
(Effect sizes range from medium to large).
Significant effect of intervention on mother
and father’s positive parenting (small effect
size).
Non-significant differences in composite
scores at follow up suggesting no change
from post-test scores.
42
Doctorate in Educational & Child Psychology
Study
Sample
Design
Relevant Measures
Findings
Training: 23
Parent/Child/Teacher
Training: 25
43
Doctorate in Educational & Child Psychology
Appendix 3
[Adapted from Task Force
on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder: RJ
Date: 03/01/2014
Full Study Reference in proper format:
Gardner, F., Burton, J., & Klimes, I. (2006). Randomised controlled trial of a parenting
intervention in the voluntary sector for reducing child conduct problems: outcomes and
mechanisms of change. Journal of Child Psychology and Psychiatry, 47(11), 1123-1132.
Intervention Name (description of study): (14 Week) Webster-Stratton Incredible Years
Parenting Program.
Study ID Number: 02
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
44
Doctorate in Educational & Child Psychology
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 87 children and respective parent
(11 were assigned to another programme)
Intervention group sample size: 44
Control group sample size: 32
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Other – Participants have not been exposed to previous interventions
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes – But only for one measure; observation intra-class correlation all above .9
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
45
Doctorate in Educational & Child Psychology
Yes –Parenting scale, Parent Sense of Competence, Beck Depression Inventory (BDI) &
the Consumer Satisfaction, Therapy Attitude Inventory.
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes – Self-report & observation
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
46
Doctorate in Educational & Child Psychology
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence 2
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
47
Doctorate in Educational & Child Psychology
C Appropriate Statistical Analysis
Analysis 1: Between Groups ANCOVA
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Analysis 2: Mann Whitney (For skewed variables, to compare the two groups on change
scores from baseline to post intervention)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Analysis 3: Follow up Analyses: Paired t-tests & non parametric equivalence (for within
groups comparison of intervention group post intervention and at 18 month follow up)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Overall Rating of Analysis:
3
2
1
0
48
Doctorate in Educational & Child Psychology
[Adapted from Task Force
on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder: RJ
Date: 03/01/2014
Full Study Reference in proper format:
Hutchings, J., Bywater, T., Daley, D., Gardner, F., Whitaker, C., Jones, K., Eames, C., &
Edwards, R. T. (2007). Parenting intervention in Sure Start services for children at risk of
developing conduct disorder: pragmatic randomised controlled trial. BMJ: British Medical
Journal, 334(7595), 678-682.
Intervention Name (description of study): (12 Week) Incredible Years Basic Parenting
Program.
Study ID Number: 03
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
49
Doctorate in Educational & Child Psychology
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 153 children and respective mother
Intervention group sample size: 104
Control group sample size: 49
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Other
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes –Parenting Stress Index (PSI), the Beck Depression Inventory (BDI) & the Dyadic
Parent-Child Interaction Coding system (DPCICS).
50
Doctorate in Educational & Child Psychology
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes – Self report & observation.
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
51
Doctorate in Educational & Child Psychology
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings – “assumed no change from baseline assessment for those lost to
follow up: no significant differences between intervention families who remained in
the study and those lost to follow up with X2 and a two sample t test. Not tested in
control families because of small numbers”.
Overall Level of Evidence 2
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
52
Doctorate in Educational & Child Psychology
C Appropriate Statistical Analysis
Analysis 1: Between Groups ANCOVA (Intervention and control group)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Overall Rating of Analysis:
3
2
1
0
53
Doctorate in Educational & Child Psychology
[Adapted from Task Force
on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder: RJ
Date: 03/01/2014
Full Study Reference in proper format:
Larsson, B., Fossum, S., Clifford, G., Drugli, M. B., Handegård, B. H., & Mørch, W. T.
(2009). Treatment of oppositional defiant and conduct problems in young Norwegian
children. European Child & Adolescent Psychiatry, 18(1), 42-52.
Intervention Name (description of study): (12-14 Weeks) Incredible Years Basic Parenting
Program.
Study ID Number: 03
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
54
Doctorate in Educational & Child Psychology
Unknown/unable to code
B Participants
Total size of sample (start of study): 136 children and respective parent
Intervention group sample size: 51 (parent training) 55 (parent and child training)
Control group sample size: 30
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure – some child participants with diagnosis of ADHD were receiving
medication (only if they had been receiving it 6 months prior to the study)
Prior exposure
Unknown
Other
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes – Above .65.
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes –Parent Practices Interview (PPI), Parenting Stress Index (PSI) & Consumer
Satisfaction Scale.
No
55
Doctorate in Educational & Child Psychology
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None – not needed
56
Doctorate in Educational & Child Psychology
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings - “Because intent to treat analyses showed identical results to per
protocol analyses for primary measure (ECBI) outcomes are reported for completers
only”
Overall Level of Evidence 3
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
57
Doctorate in Educational & Child Psychology
C Appropriate Statistical Analysis
Analysis 1: Between Groups ANCOVA (between post treatment and follow up for
intervention group and control)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Overall Rating of Analysis:
3
2
1
0
58
Doctorate in Educational & Child Psychology
[Adapted from Task Force
on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder: RJ
Date: 03/01/2014
Full Study Reference in proper format:
Letarte, M. J., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent training
program “Incredible Years” in a child protection service. Child Abuse & Neglect, 34(4), 253261.
Intervention Name (description of study): (16 Weeks) Incredible Years Parenting Training
Program.
Study ID Number: 04
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
59
Doctorate in Educational & Child Psychology
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 45 parents and respective child
Intervention group sample size: 36
Control group sample size: 9
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure – “Plus regular services”
Unknown
Other
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes – All except one above .7
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes – Parent Practices Interview (PPI) & the Parenting Self Efficacy Measure
N/A
Unknown/unable to code
60
Doctorate in Educational & Child Psychology
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
61
Doctorate in Educational & Child Psychology
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Overall Level of Evidence 1
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
62
Doctorate in Educational & Child Psychology
C Appropriate Statistical Analysis
Analysis 1: ANOVA (intervention and control group, pre and post intervention)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Overall Rating of Analysis:
3
2
1
63
Doctorate in Educational & Child Psychology
[Adapted from Task Force
on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder: RJ
Date: 03/01/2014
Full Study Reference in proper format:
Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating children with early-onset
conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of
Clinical Child and Adolescent Psychology, 33(1), 105-124.
Intervention Name (description of study): (22-24 Weeks) Webster Stratton Incredible Years
Parenting Program.
Study ID Number: 06
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
64
Doctorate in Educational & Child Psychology
Unknown/unable to code
B Participants
Total size of sample (start of study): 159 parents (mother and/or father) and respective child
Intervention group sample size: 31 (parent training)
Control group sample size: 26
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Other
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes – Almost all above .5 to .98
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes – Parenting composite scores, the Parenting Practices Interview (PPI), the Dyadic
Parent-Child Interaction Coding system (DPICS-R), Coder Impressions Inventory (CII) &
the Parent Daily Discipline Inventory (DDI)
No
65
Doctorate in Educational & Child Psychology
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes – Self report & observation.
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
66
Doctorate in Educational & Child Psychology
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings
Overall Level of Evidence 2
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
67
Doctorate in Educational & Child Psychology
C Appropriate Statistical Analysis
Analysis 1: Between Groups ANCOVA (Intervention groups and control group
pre/post-test)
Appropriate unit of analysis
Family wise/experimenter wise error rate controlled when applicable
Sufficiently large N
Overall Rating of Analysis:
3
2
1
0
68
Doctorate in Educational & Child Psychology
Appendix 4
Weighting of Studies
Weight of Evidence A – Methodological Quality
The UCL Literature Review Coding Protocol adapted from Kratochwill (2003) for
group intervention studies was used to weight all the studies on the quality of their
methodology. Studies were assessed on their measures, comparison group and
statistical analysis used.
Measures
Weighting
Description
High
Strong evidence: measures must produce reliable scores for the
majority of primary outcomes and for population of study, with a
reliability coefficient of .85 and above.
Uses multiple methods AND multiple sources.
Medium
Promising evidence: measures must produce reliable scores for the
primary outcomes and for population of study, with a reliability
coefficient of .07 and above.
Uses multiple methods AND/OR sources.
Low
Weak evidence: measures that produce somewhat reliable scores for
primary outcomes and for population of study, with a reliability
coefficient of at least .50. Uses measures that produce reliable scores
of 0.50 for the population under study.
Uses multiple methods AND/OR sources, however this is not required.
Zero
Measures did not produce reliable scores with a reliability coefficient of
< .50, AND/OR did not collect using multiple methods, AND/OR did not
collect using multiple sources.
Comparison Group
Weighting
Description
High
Strong evidence: Uses at least one type of ‘active comparison group’.
Initial group equivalency must be established preferably through
random assignment. Must be evidence that change agents were
counterbalance and met equivalent mortality with low attrition at posttest.
Medium
Promising evidence: Uses at least a ‘no intervention group’ type of
comparison. There must be evidence of at least two of the following:
1. Counterbalancing of change agents
2. Group equivalence established
3. Equivalent mortality with low attrition
If equivalent mortality was not demonstrated then an intent-to-intervene
69
Doctorate in Educational & Child Psychology
Weighting
Description
analysis must have been conducted.
Low
Weak evidence: Uses a comparison group. There must be evidence of
at least one of the following:
1. Counterbalancing of change agents
2. Group equivalence established
3. Equivalent mortality with low attrition.
If equivalent mortality was not demonstrated then an intent-to-intervene
analysis must have been conducted.
Zero
No efforts were made to ensure group equivalence.
Statistical Analysis
Weighting
Description
High
Appropriate statistical analysis includes:
1. Appropriate unit of analysis
2. Familywise error rate controlled
3. Sufficiently Large N*
Medium
Appropriate statistical analysis includes two of the following:
1. Appropriate unit of analysis
2. Familywise error rate controlled
3. Sufficiently Large N*
Low
Appropriate statistical analysis includes one of the following:
1. Appropriate unit of analysis
2. Familywise error rate controlled
3. Sufficiently Large N*
Zero
Appropriate statistical analysis did NOT include ANY of the
following:
1. Appropriate unit of analysis
2. Familywise error rate controlled
3. Sufficiently Large N*
*Sufficiently large N was judged according to criteria for a 2 group Analysis of Variance (ANOVA)
(statistical analysis used in all studies) as given by Cohen (1992). Based on a Cohen’s d effect size of
0.5 indicated that with an alpha level of .05 a sample size of 64 participants would result in a power
level of 80% (Cohen, 1988)..
Overall Methodological Quality
In determining overall methodological quality, each weight was assigned a score of:




3 for ‘High’ scores
2 for ‘Medium’ scores
1 for ‘Low’ scores
0 for ‘Zero’ scores
70
Doctorate in Educational & Child Psychology
These scores were then averaged:
Overall Methodological
Quality
High
=
Medium
=
Low
=
Average Scores
2.5
Between 2.4 & 1.5
1.4 & Less
The table below indicates the overall weight of evidence for methodological quality
(WoE A) in the 5 studies:
Weighting
Score
Comparison
Group
Statistical
Analysis
Overall Quality
of Methodology
3
2
2
2.3
Medium
Hutchings et al
(2007)
2
2
2
2
Medium
Larsson et al
(2009)
1
3
3
2.3
Medium
1
1
2
1.3
Low
2
2
2
2
Medium
Studies
Measures
Gardner, Burton &
Klimes
(2006)
Letarte, Normandeau
& Allard
(2010)
Webster-Stratton,
Reid & Hammond
(2004)
Weight of Evidence B – Methodological Relevance
This refers to the appropriateness of the research design for addressing the Review
Question.
71
Doctorate in Educational & Child Psychology
Weighting
High
Description
Uses random assignment of participants
Uses a wait list control group as only comparison*
Wait list control group has adequate sample size
Takes multiple post measurements
Uses 3 or more measures of parenting behaviour
Uses at least 3 different types of measurement (at least one
being observation)
Conducts statistical analyses on all measures of parenting
behaviour
Demonstrates group equivalence.
Medium
Uses random assignment of participants
Has additional intervention groups (e.g. child and/or teacher
training as well as parent training)**
Takes two post measurements
Uses at least 2 measures of parenting behaviour
Uses at least 2 different types of measurement
Conducts statistical analyses on most measures of parenting
behaviour
Demonstrates group equivalence.
Low
Does not randomly assign participants to conditions
Has additional intervention groups (e.g. child and/or teacher
training as well as parent training)**Sample size is too small
Takes one post measurement
Uses self-report measures of parenting behaviours only
Conducts statistical analyses on one measure of parenting
behaviour
Does not have to demonstrate group equivalence.
*Wait list control group, who are yet to receive intervention, act as only comparison to intervention
group.
**Additional intervention groups score as medium/low weighting due to their lack of relevance to
review question
N.B Weightings are allocated if the study meets the majority of criteria.
Weight of Evidence C – Topic Relevance
This refers to the appropriateness of the focus of the study to the Review Question.
Weighting
Description
High
Uses families within the UK
Uses children who have referral/diagnosis for conduct
problems &/OR ODD
Uses IY as targeted treatment
Measures parenting behaviours as primary measure
72
Doctorate in Educational & Child Psychology
Weighting
Medium
Low
Description
Measures specific improvements in particular parenting
behavioural domains (primary outcome)
Measures improvements in parent’s mental states
Parenting behaviour is reported to improve child
behaviour
Reports the effectiveness of the IYPT in improving parent
behaviour.
Uses Non UK families
Focuses on child behaviour.
Preventative intervention rather than treatment
Measures self-reported improvements in parenting
behaviour
Measures improvements in parent’s mental states
Parenting behaviour isn’t reported to improve child
behaviour
Children have attended previous programmes for
behaviour.
School receives intervention as well as parents
Uses children from specialised backgrounds e.g. looked
after
Reports self-reported improvements in parent’s mental
states but not behaviour.
Does not measure improvements.
N.B Weightings are allocated if the study meets the majority of criteria.
Overall Weight of Evidence D
This considers weight of evidence A, B & C to rate the overall degree to which the
study contributes in answering the Review Question. This was calculated by
assigning studies a score of:



3 for ‘High’ weightings
2 for ‘Medium’ weightings
1 for ‘Low’ weightings
These scores were then averaged:
Overall Weight of
Evidence
High
Medium
Low
Average Scores
=
=
=
2.5
Between 2.4 & 1.5
1.4 & less
73
Doctorate in Educational & Child Psychology
The table below indicates the overall weight of evidence (WoE D) of the 5 studies:
Weighting
Score
Studies
Gardner, Burton &
Klimes
(2006)
Topic
Methodological Methodological
Relevance
Quality A
Relevance B
C
2.3
3
3
Medium
Overall Weight
of Evidence D
2.8
High
Hutchings et al
(2007)
2
Medium
3
2
2.3
Medium
Larsson et al
(2009)
2.3
Medium
1
2
1.8
Medium
1.3
Low
1
2
1.4
Low
2.3
Medium
2
2
2.1
Medium
Letarte,
Normandeau &
Allard
(2010)
Webster-Stratton,
Reid & Hammond
(2004)
74