Doctorate in Educational and Child Psychology
Rachel Standen
Case Study 1: An Evidence-Based Practice Review Report
Theme: Interventions for children with Special Educational Needs
Is Multidimensional Treatment Foster Care (MTFC) an effective intervention for
reducing externalising behaviours in children with Social, Emotional and Behavioural
Difficulties, whose challenging behaviours are resulting in their current placement
being at risk?
Summary
The Multidimensional Treatment Foster Care (MTFC) program involves placing young people, whose
challenging behaviours are resulting in current placements potentially becoming unsustainable, in
well-trained and supervised foster homes. The six to nine month intervention aims to enable young
people to live successfully in their community through the use of intensive supervision, support, and
skill development and to simultaneously prepare their original placement to provide effective
parenting to enable a positive reintroduction to the family.
The purpose of this systematic review was to establish whether MTFC is an effective intervention for
reducing externalising behaviours in children with social, emotional and behavioural difficulties,
whose challenging behaviours are resulting in their current placement being at risk. A systematic
literature search identified five studies which met the inclusion criteria outlined in Table.1. These
articles were coded using an adapted version of Kratochwill’s Literature Coding Protocol
(Kratochwill, 2003), and evaluated using the ‘Weight of Evidence Framework’ (Gough, 2007).
MTFC was reported to be an effective intervention for reducing children’s externalising behaviours
in three of the five studies. The critical analysis identified limitations in all of the studies meaning the
findings must be interpreted with caution. Therefore this review does not enable confident
1
Doctorate in Educational and Child Psychology
Rachel Standen
recommendation of MTFC to families of children with social, emotional and behavioural difficulties
displaying externalising behaviours.
Introduction
The Department for Children, Schools and Families (DCSF) describes social, emotional and
behavioural difficulties as behaviours being demonstrated by children such as being withdrawn or
isolated, disruptive and disturbing; being hyperactive and lacking concentration; having immature
social skills; or presenting challenging behaviours arising from other complex special needs (DCSF,
2007). Whether a child is perceived to have emotional and behavioural difficulties will depend on
the nature, frequency, persistence, severity and/or cumulative effect of the behaviour, compared to
normal expectations for the age of the child (Department for Education, 1994). Such behaviours are
often categorised as externalising or internalising behaviours. Childhood behaviours marked by
defiance, impulsivity, disruptiveness, aggression, antisocial features and over-activity are classified
as externalising behaviours, where behaviours characterised by withdrawal, anxiety and dysphoria
are known as internalising behaviours. Externalising behaviours are more stable, more resistant to
intervention and are associated with negative outcomes in later life (Hinshaw, 1992). However,
estimates of prevalence vary. This is partly due to many research investigations focussing on specific
topographies, (such as aggression or self-injurious behaviour) where others have focused on more
general categorisation. This use of differing definitions has contributed to wide variation in the
assessment of prevalence (Lowe, Allen, Jones, Brophy, Moore, & James, 2007).
Externalising behaviours, academic underachievement, and poor development of positive social
skills among students are a key concern for educators and parents of children and young
people(Luiselli, Putnam, Handler & Feinberg, 2005) . Aggression and violence in young people has
been found to have implications on the progression of psychiatric impairment, school difficulties,
2
Doctorate in Educational and Child Psychology
Rachel Standen
and legal involvement (Rappaport & Thomas, 2004) where children who engage in challenging
behaviours at a young age are more likely than their nonaggressive peers to respond similarly when
older and as adults (Luiselli et al., 2005).
The needs of looked after children are of increasing importance to both academic researchers and
policy-makers (Goddard, 2000). Looked after children are disproportionately represented in statistics
relating to low levels of school attainment, exclusion from school, high levels of truancy and social
exclusion (Dent & Cameron, 2003). Recognition of the emotional and behavioural difficulties
experienced by looked after children (and the impact of these) has also been increasing (Sargent &
O’Brien, 2004) alongside evidence that placement disruption and externalising behaviour problems
are associated. For example findings by Newton, Litrownik and Landsverk (2000) have suggested
that multiple placements in out-of-home care are associated with both immediate and long-term
negative outcomes for the child. Children who are disruptive, aggressive and/or dangerous to others
are at higher risk of experiencing placement change (Newton et al., 2000).
Educational psychologists can play a central role in the process of helping teachers, parents/foster
carers and peers to support individual children and are well placed to give a view on the impact of
different types intervention on the child’s care or behaviour (Dent & Cameron, 2003). Educational
Psychologists are able to provide informed perspectives that are drawn from psychological research
and theory, and can help other professionals to develop a deeper understanding of pupils’ personal
and social development, their behaviour and their learning (Dent & Cameron, 2003).
Despite increasing official and academic interest, there remains significant gaps in the available
literature in relation to looked after children (Dent & Cameron, 2003). The recruitment of children
for research projects through schools involves co-operation from local authorities, school staff and
parents. In the process of recruiting looked after children, the number of gatekeepers is increased to
3
Doctorate in Educational and Child Psychology
Rachel Standen
include social services and foster parents among others. Looked after children are often felt to be
particularly vulnerable, and therefore there is a tendancy to protect them from the perceived
potential negative effects of research participation (Heptinstall, 2000). This may mean children are
prevented from taking part in research despite the recognised need to increase the evidence base in
this area and the young people having expressed a wish to do so.
It is recognised in the emerging literature that many looked after children require comprehensive
behavioural supports that involve family, school, and community participation (Sugai,Homer, Dulap,
Hieneman, Lewis, Nelson, Scott, Liaupsin, Sailor, Turnbull, Rutherford-Turnbull, Wickham, Wilcox &
Ruef, 2000). The Multidimensional Treatment Foster Care (MTFC) program began as a communitybased alternative to placement in group or residential care for children and adolescents with severe
emotional and behavioural problems (Chamberlain, 2003).
MTFC is based on social learning theory (Bandura, 1977) and aims to benefit from the positive
socialising influence of the family. In the social learning system new patterns of behaviour can be
acquired through direct experience or by observing the behaviour of others. By observing a model of
the desired behaviour an individual forms an idea of how response components must be combined
and sequenced in order to produce favourable outcomes. This representation serves as a guide for
subsequent behavioural reproduction. During the course of learning, people not only perform a
particular behaviour, but also observe the differential consequences that accompany the behaviour.
Some of the responses will prove negative or unsuccessful while others produce more favourable
effects. Through this process of differential reinforcement successful modes of behaviour are
selected, while ineffectual ones are discarded (Bandura, 1977).
MTFC involves placing young people, whose challenging behaviours are resulting in current
4
Doctorate in Educational and Child Psychology
Rachel Standen
placements potentially becoming unsustainable, in well-trained and supervised foster homes
(Rhoades, Chamberlain, Roberts, & Leve, 2013). The period of intervention typically lasts six to nine
months and is focused on implementing an intensive programme across multiple settings (e.g.,
home, school, peer group, and community). The ultimate goals of MTFC are for young people to live
successfully in their community through the use of intensive supervision, support, and skill
development and to simultaneously prepare their original placement to provide effective parenting
to enable a positive reunion with the family (Smith, Chamberlain, & Eddy, 2010).
Four key elements are targeted during and after treatment. The MTFC model provides a consistent
reinforcing environment where the youths are mentored and encouraged to perform specific tasks
or behaviours designed to increase their skill base. Clear expectations and limits and well-specified
consequences are delivered in a teaching-oriented manner. The young people are closely supervised
and they are supported in avoiding social situations where peers engage in similar behaviours, and
given assistance in establishing positive peer networks (Smith et al., 2010).
The components of MTFC include:
1. Daily telephone contact with MTFC parents using the Parent Daily Report checklist
(Chamberlain & Reid, 1987),
2. Weekly foster parent group meetings led by the program supervisor that are focused on
supervision, training in parenting practices, and support,
3. An individualized behaviour management program implemented daily in the home by the
foster parent,
4. Individual therapy for the youth,
5. Individual skills training/coaching for the youth,
6. Family therapy (for biological/original family unit of the youth) focused on parent
management strategies. As part of family therapy, home visits are used throughout the
5
Doctorate in Educational and Child Psychology
Rachel Standen
program for parents and youth to practice their skills in the context of their family
environment,
7. Close monitoring of school attendance, performance, and homework completion,
8. Case management to coordinate the MTFC, family, peer, and school settings,
9. 24-hour on-call staff availability to MTFC and biological parents,
10. Psychiatric consultation as needed.
(Rhoades et al., 2013).
Parents receive 20 hours of training conducted by program supervisors and current MTFC foster
parents prior to providing a MTFC placement for a young person. Training is focused on the use of
behaviour management methods to establish and maintain a structured, supervised, and consistent
daily living environment. Parents are supported by a team of professionals including the program
supervisor, who acts as the clinical lead and has a small caseload of a maximum of 10 families. The
rest of team includes family and individual therapists, a skills trainer and a foster parent
recruiter/trainer. Program supervisors maintain daily contact with MTFC parents to collect data on
youth adjustment and to provide ongoing consultation, support, and crisis intervention (Rhoades, et
al.,, 2013; Eddy & Chamberlain, 2000; Van Ryzin & Leve, 2012).
This review offers a critical appraisal of the evidence base and explores whether MTFC is an effective
intervention for reducing externalising behaviours in children with social, emotional and behavioural
difficulties, whose challenging behaviours are resulting in their current placement being at risk. This
analysis is focused on children who are of school-age (4 – 19) due to alternate forms of the MTFC
intervention being used with children who are of preschool age (Multidimensional Treatment Foster
Care - Preschool). Differences in many areas of implementation including training for foster carers
and the use of playgroup weekly sessions for development of pro-social skills resulted in an inability
to review the two simultaneously.
6
Doctorate in Educational and Child Psychology
Rachel Standen
Critical Review of the Evidence Base
Literature Search
Studies of Multidimensional Treatment Foster Care were identified through a search of PubMed,
PsychINFO, ERIC and Science Direct using the key words Multidimensional Treatment Foster Care OR
MTFC AND behavi* within the title and abstract of publications. No restriction regarding the year of
publication was placed on the searches.
All retrieved studies were reviewed to determine if they met criteria for inclusion in the present
evaluation. Once the duplicates were removed, the titles and abstracts of seventy-two studies were
screened for eligibility, using the inclusion criteria, outlined in table 1, resulting in nine potential
studies for consideration. A full article review was subsequently undertaken of these remaining nine
studies in order to confidently make a decision regarding the admissibility, which left five remaining
studies that were deemed suitable for answering the review question. In the process of reviewing
the retrieved studies, an alternative term, used in the United Kingdom for Multidimensional
Treatment Foster Care, was highlighted. Therefore the search was repeated using this term Intensive
Fostering within all four bibliographical databases. This search did not return any additional studies
to the original systematic search.
Figure 1 depicts the process followed, identifying studies from the initial searches to screening and
selecting studies for the review. The excluded studies plus rationale for exclusion are listed in
Appendix B.
7
Doctorate in Educational and Child Psychology
Rachel Standen
Table. 1. Inclusion and Exclusion Criteria.
Criteria
Inclusion
Intervention
Intervention studied was Studies
that
Multidimensional Treatment interventions.
Foster Care (MTFC).
Dependent
variable
Standardised measures of Measures were not of participant externalising
externalising or challenging behaviour
behaviours.
Dependent
variable
Behaviours recorded are not Measures were limited to behaviours of an illegal
limited to those of an illegal nature only.
nature.
(e.g. Elliot measure of delinquency).
Participants are of school age Studies where participants were described to be
attending preschool/nursery settings.
Sample
Exclusion
investigated
alternative
Participants aged 3years old or younger
Type of study
Type of Study
Language
Contains primary empirical
data.
Published in a peer-reviewed
journal.
Written in the English
Language
Participants aged 19 years or above.
Does not contain primary empirical data
(e.g., review paper, theoretical paper).
Not published in a peer-reviewed journal.
(e.g. book, dissertation paper, thesis).
Written in a language other than English
Table. 2. Studies identified to be suitable in answering the research question;
1
Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. a, & Reid, J. B. (2008).
Prevention of behavior problems for children in foster care: outcomes and mediation
effects. Prevention Science : The Official Journal of the Society for Prevention Research, 9(1),
17–27.
2
Green, J. M., Biehal, N., Roberts, C., Dixon, J., Kay, C., Parry, E., … Sinclair, I. (2013).
Multidimensional Treatment Foster Care for Adolescents in English care: randomised trial
and observational cohort evaluation. The British Journal of Psychiatry : The Journal of
Mental Science.
3
Hansson, K., & Olsson, M. (2012). Effects of multidimensional treatment foster care (MTFC):
Results from a RCT study in Sweden. Children and Youth Services Review, 34(9), 1929–1936
4
Leve, L. D., Chamberlain, P., & Reid, J. B. (2005). Intervention outcomes for girls referred
from juvenile justice: effects on delinquency. Journal of Consulting and Clinical Psychology,
73(6), 1181–5.
5
Westermark, P. K., Hansson, K., & Olsson, M. (2011). Multidimensional treatment foster
care (MTFC): results from an independent replication. Journal of Family Therapy, 33(1), 20–
41.
A full summary of the included studies including participants and outcome measures are detailed in
Appendix A.
8
Doctorate in Educational and Child Psychology
Rachel Standen
FIGURE.1. Diagram of Study Screening Process.
9
Doctorate in Educational and Child Psychology
Rachel Standen
Quality and relevance of studies
The Gough (2007) framework and an adapted version of the Kratochwill (2003) Task Force on
Evidence Based Interventions in School Psychology coding protocol was used to analyse the research
studies selected. The weightings of the studies were formed based on judgements of 3 aspects of
the studies:
A) Methodological quality (the quality of execution of the study in relation to quality standards
for studies of that type)
B) Methodological relevance (the appropriateness of the evidence/research design for
answering the review question)
C)
Topic relevance (the appropriateness of the focus of the evidence for the review question)
The Overall Weight of Evidence was calculated by determining the mean score across each criteria,
with each one holding equal importance. The weightings for each study are given in Table 3 and
Appendix C. The full coding protocol is outlined in Appendix D, with details of how judgements were
made regarding the weightings being detailed below.
Measures and Design
The systematic literature review highlighted seven studies that investigated the effect of MTFC on
behavioural outcomes using measures that solely targeted illegal measures (e.g. Elliot Scale of
Delinquency). These studies were excluded from this review (see table 1) as it was felt that limiting
behavioural outcomes to those that are illegal in nature, and often with a focus on related alcohol
and drug use, will narrow the implications this review may have on Educational Psychology practise.
Therefore it was felt that these studies were too specific in focus to answer the research question
when aiming to reduce externalising behaviours of all description with children with social,
emotional and behavioural difficulties.
10
Doctorate in Educational and Child Psychology
Rachel Standen
The designs of the included studies were similar with each study collecting baseline data of
externalising behaviours such as aggression and disruptive behaviour. However, the time that postintervention measures were taken by the researchers varied from five to twenty-four months and
this will have an effect on the outcomes. This was highlighted in a meta-analysis of parent training
programmes where researchers found that follow-up measures taken one year following the
completion of an intervention, the effects remained meaningful, but were small in magnitude
(Lundahl, Risser, & Lovejoy, 2006). Three studies used more than one source of data collection, and
studies conducted by Leve et al., (2005) and (Chamberlain et al., 2008) only used data collected from
the perspective of the foster parent. This affected the overall quality of methodology in the study
and was reflected in the weight of evidence (table 3, Appendix C).
The majority of the studies used similar measures of challenging behaviours. Four studies (Green et
al., 2013; Hansson & Olsson, 2012; Leve et al., 2005; Westermark, Hansson, & Olsson, 2011) used
the Child Behaviour Checklist (Achenbach, 1991). Although many of the studies did not report
reliability or validity for the Child Behaviour Checklist (CBCL), Westermark et al. (2011) reported a
Cronbach α of 0.78 – 0.97 and test re‐test reliability score of 0.94. However it is important to note
that all but one study did not used the full CBCL, with Hansson & Olsson, (2012) and Westermark et
al. (2011) using the problem scale and Leve et al. (2005) only using the delinquency subscale. This
reduction in content will have implications for the reliability of the measure.
A second measure that was used alongside the CBCL in two studies (Hansson & Olsson, 2012;
Westermark et al., 2011) was the Youth Self Report (Achenbach & Rescorla, 2001) with a reported
Cronbach α of 0.71 – 0.95 and test re‐test reliability score of 0.87 (Westermark et al., 2011). Similar
to the CBCL the problem scale of the Youth Self Report was used without the competence scale,
which will have implications for the measure’s reliability. The final study (Chamberlain et al., 2008)
11
Doctorate in Educational and Child Psychology
Rachel Standen
used the Parent Daily Report Checklist (Chamberlain & Reid, 1987). The reliability and validity for this
measure was not reported however, this questionnaire is well referenced in the literature.
Participants
The total number of participants ranged from ten years to eighteen years of age. Three studies
reported the mean ages of the participants (Hansson & Olsson, 2012; Leve et al., 2005; Westermark
et al., 2011) with similar mean ages in years of 14.8, 15.3 and 15.4 respectively. Only one of the five
studies met the Kratochwill (2003) criteria for sufficient sample size and this is reflected in
weightings of evidence awarded for the ‘Analysis’ in table 3 (see Appendix C). All studies used
random allocation of participants to intervention and control trials, resulting in the intervention
group consisting of 50.83% of all participants. All studies were awarded a ‘promising’ weighting of
evidence for the comparison group within ‘Quality of Methodology’ because, although participants
were randomly assigned, the control of change agents was not used in any study.
Due to the limited research on MTFC, there is no current meta-analysis to identify an expected effect
size for this intervention. Therefore analysis of the current literature was extended to the effects of
parent training programs, with the aim of reducing challenging behaviours in children. This search
resulted in a meta-analysis written by Lundahl, Risser and Lovejoy (2006) who concluded that the
mean effect size for child behaviour was 0.42. This indicates a minimal (0.08) deficit from what
Cohen (1988) describes as a medium effect size. Subsequently a medium effect size was expected
for the five studies selected for this analysis, with a sufficiently large N of 64 or above. Only one
study (Chamberlain et al., 2008) met this criteria.
12
Doctorate in Educational and Child Psychology
Rachel Standen
Intervention
The integrity by which the intervention itself was carried out was explicitly discussed in the studies,
where the Parent Daily Report Checklist was used in each research project to ensure fidelity to the
intervention design, and reference to the MTFC manual mentioned in the majority of studies.
However the training and monitoring received by the foster carers was variable within the studies.
The Manual for the MTFC explicitly states the initial training that should be received by foster
parents (which varies depending on the age of the young people being placed in their care) and the
ongoing supervision and support that should be received in order to maintain fidelity of
implementation. The fidelity of the MTFC implementation will affect the ability of the publication to
answer the research question due to the impact it has on the overall quality of the study. Therefore
the fidelity of the MTFC implementation, as reported in the research, was reflected in the weight of
evidence (table 3, Appendix C).
Table 3. Overall Weight of Evidence.
Is Multidimensional Treatment Foster Care an effective intervention for reducing externalising
behaviours in children with social, emotional and behavioural difficulties, whose challenging
behaviours are resulting in their current placement being at risk?
(A)
(B)
(C)
(D)
Quality of
Methodology
Relevance of
Methodology
Relevance of evidence to
the review question
Overall Weight of
Evidence
Westermark, Hansson, &
Olsson. (2011)
Medium
High
High
High
2
3
3
2.6
Hansson & Olsson. (2012)
Medium
High
High
High
2
3
3
2.6
Medium
High
High
High
2
3
3
2.6
Medium
High
Medium
Medium
2
3
2
2.3
Low
High
Medium
Medium
1
3
2
2
Green et al. (2013)
Chamberlain et al. (2008)
Leve, Chamberlain &
Reid. (2005)
13
Doctorate in Educational and Child Psychology
Rachel Standen
Overall Weight of Evidence
Mean Score
High
At least 2.5
Medium
Between 1.5 and 2.4
Low
Less than 1.4
Effectiveness of MTFC in reducing externalising behaviour
All studies reported that MTFC had a positive impact on externalising behaviours demonstrated by
the young people. Table 4 summarises the results of each of the five individual studies, and a brief
discussion of the results is detailed below;
Three of the studies identified a significant reduction in externalising behaviours. Westermark et al.
(2011) concluded that only the young people included within the MTFC intervention showed a
consistent statistical significance in externalising behaviour reduction. She stated that the clinically
significant changes illustrated that MTFC, in comparison with Treatment As Usual (TAU), had more
young people reducing their externalising behaviours at a minimum of one standard deviation of a
normal population. This was supported by the results of Leve et al. (2005) who found that, only
young people who were allocated into the MTFC intervention showed statistical significance in
externalising behaviour reduction, as rated by their Foster Caregivers. Similarly Chamberlain et al.
(2008) found that foster parents in the MTFC condition reported significantly lower externalising
behaviour rates in post-intervention measures than those in the TAU group. Chamberlain et al.
(2008) concluded that the results showed that participation in MTFC increased parental
effectiveness, which related to decreased child behaviour problems. This was particularly evident in
families who reported higher levels of externalising behaviours at the baseline assessment.
However, Hansson and Olsson, (2012) found that both groups of young people showed a significant
but similar reduction in externalising behaviours over time. Comparably Green et al. (2013) found
that the young people in the MTFC group had a marginally better outcome for the measure of CGAS,
14
Doctorate in Educational and Child Psychology
Rachel Standen
which was a composite score developed from the CBCL and YSR measures, than the usual care group
but this was not statistically significant. Therefore Green et al. (2013) and Hansson and Olsson (2012)
both suggest that there is no evidence that MTFC results in in a greater reduction in externalising
behaviours when compared with usual treatment.
Table 4. Outcomes and effect sizes of included studies.
Study
Westermark,
Hansson, & Olsson.
(2011)
Hansson & Olsson,
(2012)
Green et al., (2013)
Chamberlain et al.,
(2008)
Leve, Chamberlain &
Reid (2005).
Outcome
Effect Size*
Reduction
in
behaviours
considered as challenging as
rated
by
participants
themselves (YSR) and by Foster
Parents (CBCL)
(cohen’s d)
Reduction
in
behaviours
considered as challenging as
rated
by
participants
themselves (YSR) and by Foster
Parents (CBCL)
(cohen’s d)
Reduction
in
behaviours
considered as challenging as
rated by participants by Foster
Parents (CGAS)
(cohen’s d)
Reduction
in
behaviours
considered as challenging as
rated by participants by Foster
Parents (PDR)
(cohen’s d)
Reduction
in
behaviours
considered as challenging as
rated by participants by Foster
Parents (CBCL)
(Partial Eta
Square)
Effect Size
Interpretation
Small
YSR 0.33
CBCL 0.19
Overall Weight of
Evidence
High
2.6
Small
YSR 0.20
CBCL 0.25
High
2.6
Medium
High
CGAS: 0.50
2.8
Small
Medium
PDR 0.26
2.2
Medium
CBCL 0.07
Medium
2
*All effect sizes were calculated and reported within the original studies. Indicators of a small,
medium or large effect size are dependent on the method used for calculation (see Table 5).
Table 5. Indicators of a small, medium or large effect size depending on the method used for
calculation
Type of effect size
Partial Eta Square
(Cohen, 1988)
Small Medium
Cohen's d (Cohen, 1988
0.01
0.06
0.20
0.50
Large
0.14
0.80
(Green & Salkind, 2008)
15
Doctorate in Educational and Child Psychology
Rachel Standen
Conclusions
The systematic review analysed five studies to explore whether Multidimensional Treatment Foster
Care (MTFC) an effective intervention for reducing externalising behaviours in children with social,
emotional and behavioural Difficulties, whose placements are at risk due to challenging behaviour.
The Three studies awarded ‘high’ for ‘Overall Weight of Evidence’ resulted in contradictory findings.
Westermark et al. (2011) provided evidence that MTFC may be an effective intervention for reducing
externalising behaviour. Small effect sizes were found for the difference between the two groups
with results of statistical significance in externalising behaviour reduction in two measures of
externalising behaviours from multiple sources (d= 0.19 and d=0.33). Conversely However, Green et
al. (2013) did find a medium effect size (d=0.50) and Hansson and Olsson, (2012) reported small
effect sizes (d=0.20 and d=0.25). These effect sizes need to be interpreted with caution. Despite
achieving a ‘high’ weight of evidence, none of these studies used a large enough participant number
to state an effect size at 80% power which may contribute to the contradictory evidence.
Two studies awarded ‘medium’ for ‘Overall Weight of Evidence’ (Chamberlain et al., 2008; Leve et
al., 2005) provided support for MTFC being an effective intervention for reducing externalising
behaviour. A medium effect size was found for the difference between the two groups in the CBCL
delinquency subscale (Leve et al., 2005) and a small effect size was found for the difference in the
PDR checklist measure (Chamberlain et al., 2008). Change between baseline and end of intervention
scores was also significant for children in the MTFC groups in both studies. However, these results
must be considered in light of the methodological limitations of the studies resulting in them being
weighted as ‘medium’ rather than ‘high’. Neither study used multiple sources of data to enable
triangulation of information. Therefore both studies are limited by a lack of inter-rater reliability and
the possibility of bias from the MTFC Foster Carers who, following a substantial training and
intervention timeframe, may have been invested in a positive outcome for the MTFC condition.
16
Doctorate in Educational and Child Psychology
Rachel Standen
Secondly, neither measure was norm referenced to a population of children in out-of-home
placement and thirdly both studies only used a single measure of externalising behaviour. This has
an impact on the reliability. In particular the CBCL delinquency subscale is comprised of thirteen
items and this use of a single subscale will reduce the applicability of the Cronbach α of 0.78 – 0.97
and test re-test reliability score of 0.94 reported in the manual for the entire CBCL.
Due to the limitations of the reviewed studies, and the contradictory results reported within them,
the review suggests that it is not appropriate for Educational Psychologists to recommend
Multidimensional Treatment Foster Care to the families of children with social, emotional and
behavioural difficulties who are displaying externalising behaviours on the evidence that has been
produced so far. This has been concluded in light of the significant resources required to ensure a
high level of fidelity and the potential implications of removing a child from their family home and
placing them in a treatment placement that has not been conclusively established as effective. As
stated in the introduction, Newton et al. (2000) have suggested that multiple placements in out-ofhome care are associated with both immediate and long-term negative outcomes for the child. If a
reduction in externalising behaviours is not noted in response to MTFC, there is a potential for the
young person being unable to return to their previous placement (often family home) resulting in a
second out-of-home placement and the associated negative outcomes.
MTFC needs to be further evaluated using designs that incorporate measures which have good
evidence for reliability and validity, using multiple sources of data that can be triangulated in order
to reduce bias. Importantly this needs to be done with a larger number of participants in order to
gain 80% power. In addition the design of studies could incorporate a wider range of measure
to increase the relevance of research to educational Psychology Practice.
17
Doctorate in Educational and Child Psychology
Rachel Standen
Despite being a recommended parent training intervention on the Department of Education
website, few educational outcomes have been researched as secondary outcomes alongside
the primary outcome of a reduction in challenging behaviours. One such study has been
completed by (Rhoades et al., 2013) who found a medium effect size (d=0.48) of MTFC on
School Activities which included attendance and time spent engaged with homework.
Similarly (Leve & Chamberlain, 2007) found medium effect sizes in attendance (݊ଶ= 0.07)
and homework completion (݊ଶ= 0.09). Both of these factors have positive implications on
outcomes for academic success and social development in young people (Coutts, 2004;
Sheldon, 2007). However, both of these studies focussed on adolescent girls and further
research would be required to see if these results generalise to other populations.
18
Doctorate in Educational and Child Psychology
Rachel Standen
References
Achenbach. (1991). Manual for the Child Behavior Checklist and 1991 profile. Burlington, VT:
University of Vermont Department of Psychiatry.
Achenbach & Rescorla. (2001) Manual for the ASEBA School-age Form and Profiles. Burlington, VT:
Research Centre for Children, Youth and Families, University of Vermont.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological
Review, 84(2), 191-215.
Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. a, & Reid, J. B. (2008). Prevention of
behavior problems for children in foster care: outcomes and mediation effects. Prevention Science :
The Official Journal of the Society for Prevention Research, 9(1), 17–27.
Coutts, P. M. (2004). Meanings of Homework and Implications for Practice. Theory Into Practice,
43(3), 182–188.
Dent, R. J., & Cameron, R. J. S. (2003). Developing Resilience in Children Who are in Public Care: The
educational psychology perspective. Educational Psychology in Practice, 19(1), 3–19.
Eddy, J. M., & Chamberlain, P. (2000). Family managment and deviant peer association as mediators
of the impact of treatment condition on youth antisocial behavior. Journal of Consulting and Clinical
Psychology, 68(5), 857–863.
Goddard, J. (2000). The education of looked after children. Child and Family Social Work, 5(1), 79 –
86.
Gough. (2007). Weight of Evidence: a framework for the appraisal of the quality and relevance of
evidence. Applied and Practice Based Research, 22(2), 213–228.
Green, J. M., Biehal, N., Roberts, C., Dixon, J., Kay, C., Parry, E., … Sinclair, I. (2013). Multidimensional
Treatment Foster Care for Adolescents in English care: randomised trial and observational cohort
evaluation. The British Journal of Psychiatry : The Journal of Mental Science.
Hansson, K., & Olsson, M. (2012). Effects of multidimensional treatment foster care (MTFC): Results
from a RCT study in Sweden. Children and Youth Services Review, 34(9), 1929–1936.
Heptinstall, E. (2000). Research note. Gaining access to looked after children for research purposes:
lessons learned. British Journal of Social Work, 30(6), 867–872.
Kratochwill, T. R., & Steele Shernoff, E. (2004). Evidence-Based Practice: Promoting Evidence-Based
Interventions in School Psychology. School Psychology Review, 33, 34-48
Leve, L. D., & Chamberlain, P. (2007). A Randomized Evaluation of Multidimensional Treatment
Foster Care: Effects on School Attendance and Homework Completion in Juvenile Justice Girls.
Research on Social Work Practice, 17(6), 657–663.
Leve, L. D., Chamberlain, P., & Reid, J. B. (2005). Intervention outcomes for girls referred from
juvenile justice: effects on delinquency. Journal of Consulting and Clinical Psychology, 73(6), 1181–5.
19
Doctorate in Educational and Child Psychology
Rachel Standen
Lowe, K., Allen, D., Jones, E., Brophy, S., Moore, K., & James, W. (2007). Challenging behaviours:
prevalence and topographies. Journal of Intellectual Disability Research : JIDR, 51(Pt 8), 625–36.
Luiselli, J. K., Putnam, R. F., Handler, M. W., & Feinberg, A. B. (2005). Whole‐school positive
behaviour support: effects on student discipline problems and academic performance. Educational
Psychology, 25(2-3), 183–198.
Lundahl, B., Risser, H. J., & Lovejoy, M. C. (2006). A meta-analysis of parent training: moderators and
follow-up effects. Clinical Psychology Review, 26(1), 86–104.
Newton, R. R., Litrownik, a J., & Landsverk, J. a. (2000). Children and youth in foster care: distangling
the relationship between problem behaviors and number of placements. Child Abuse & Neglect,
24(10), 1363–74. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11075702
Rappaport, N., & Thomas, C. (2004). Recent research findings on aggressive and violent behavior in
youth: implications for clinical assessment and intervention. The Journal of Adolescent Health :
Official Publication of the Society for Adolescent Medicine, 35(4), 260–77.
Rhoades, K. a, Chamberlain, P., Roberts, R., & Leve, L. D. (2013). MTFC for High Risk Adolescent Girls:
A Comparison of Outcomes in England and the United States. Journal of Child & Adolescent
Substance Abuse, 22(5), 435–449.
Sargent, K., & O’Brien, K. (2004). The Emotional and Behavioural Difficulties of Looked after Children:
Foster Carers’ Perspectives and an Indirect Model of Placement Support. Adoption & Fostering,
28(2), 31–37.
Sheldon, S. B. (2007). Improving Student Attendance With School, Family, and Community
Partnerships. The Journal of Educational Research, 100(5), 267–275.
Smith, D. K., Chamberlain, P., & Eddy, J. M. (2010). Preliminary Support for Multidimensional
Treatment Foster Care in Reducing Substance Use in Delinquent Boys. Journal of Child & Adolescent
Substance Abuse, 19(4), 343–358.
Sugai, G., Horner, R. H., Dunlap, G., Hieneman, M., Lewis, T. J., Nelson, C. M., … Ruef, M. (2000).
Applying Positive Behavior Support and Functional Behavioral Assessment in Schools. Journal of
Positive Behavior Interventions, 2(3), 131–143.
Van Ryzin, M. J., & Leve, L. D. (2012). Affiliation with delinquent peers as a mediator of the effects of
multidimensional treatment foster care for delinquent girls. Journal of Consulting and Clinical
Psychology, 80(4), 588–96.
Westermark, P. K., Hansson, K., & Olsson, M. (2011). Multidimensional treatment foster care
(MTFC): results from an independent replication. Journal of Family Therapy, 33(1), 20–41.
20
Doctorate in Educational and Child Psychology
Rachel Standen
Appendices
Appendix.A. Summary of Studies
Study
Westermark,
Hansson, &
Olsson.
(2011)
Participant Sample
35 young people
aged between 9 and
18 who met the
clinical diagnosis of
conduct
disorder
(according to DSMIV-TR) and were at
risk
of
losing
permanent
home
placement.
Intervention
Multidimensional
Treatment Foster
Care
Design
Randomised
Controlled
Trial
Measures
Childhood
Behaviour
Checklist
(CBCL)
YSR mean total score (pre) 74.7
YSR mean total score (post) 38.6
Youth Self
Report
(YSR)
Intervention group
(N=20)
Control (TAU) group
(N= 15)
Hansson &
Olsson.
(2012)
46 young people
aged between 12
and 17 who met the
clinical diagnosis of
conduct
disorder
(according to DSMIV-TR) and were at
risk
of
losing
permanent
home
placement.
33 young people
aged between 10
and 17 who were
displaying
severe
challenging
behaviour and were
at risk of losing
permanent
home
placement.
TAU:
CBCL mean total score (pre) 74.3
CBCL mean total score (post) 55.1
YSR mean total score (pre) 62.6
YSR mean total score (post) 46.3
Multidimensional
Treatment Foster
Care
Randomised
Controlled
Trial
Childhood
Behaviour
Checklist
(CBCL)
MTFC :
CBCL mean total score (pre) 62.6
CBCL mean total score (post) 39.6
YSR mean total score (pre) 56.6
YSR mean total score (post) 40.4
Youth Self
Report
(YSR)
Intervention group
(N=19)
Control (TAU) group
(N= 27)
Green et al.
(2013)
Outcomes
MTFC :
CBCL mean total score (pre) 60.2
CBCL mean total score (post) 30.7
TAU:
CBCL mean total score (pre) 65.2
CBCL mean total score (post) 46.6
YSR mean total score (pre) 52.9
YSR mean total score (post) 43.9
Multidimensional
Treatment Foster
Care
Randomised
Controlled
Trial
Childhood
Behaviour
Checklist
(CBCL)
Youth Self
Report
(YSR)
This
information
was
integrated,
transcribed,
and
then
converted
into a CGAS
score.
Intervention group
(N=20)
Control (TAU) group
(N= 13)
21
MTFC :
CGAS mean total score (pre) 47.7
CGAS mean total score (post) 56.0
TAU:
CGAS mean total score (pre) 48.3
CGAS mean total score (post) 55.3
Doctorate in Educational and Child Psychology
Chamberlain
et al. (2008)
700 young people
aged between 5 and
12
who
were
displaying
severe
challenging
behaviour resulting
in
current
placement not being
sustainable under
current conditions.
Multidimensional
Treatment Foster
Care
Rachel Standen
Randomised
Controlled
Trial
Parent Daily
Report
Checklist
(PDR)
MTFC :
PDR mean total score (pre) 47.7
PDR mean total score (post) 56.0
Childhood
Behaviour
Checklist
(CBCL)
MTFC :
CBCL mean total score (pre) 79.7
CBCL mean total score (post) 64.8
TAU:
PDR mean total score (pre) 48.3
PDR mean total score (post) 55.3
Intervention group
(N=359)
Control (TAU) group
(N= 341)
Leve,
Chamberlain
& Reid.
(2005)
81 females aged
between 13 and 17
who were displaying
severe challenging
behaviour resulting
in
current
placement not being
sustainable under
current conditions.
Multidimensional
Treatment Foster
Care
Randomised
Controlled
Trial
Intervention group
(N=37)
Control (TAU) group
(N=44)
22
TAU:
CBCL mean total score (pre) 78.7
CBCL mean total score (post) 70.0
Doctorate in Educational and Child Psychology
Rachel Standen
Appendix. B: Excluded Studies
Reference
Source:
Rationale for
exclusion
Excluded
following:
Harold, Kerr, Van, DeGarmo, Rhoades & Leve (2013).
Depressive symptom trajectories among girls in the juvenile
justice system: 24-month outcomes of an RCT of
multidimensional treatment foster care.
Prevention Science. 14, 437-446.
PubMed
Measures were
not of participant
externalising
behaviour
Abstract
Review
Jonkman, Bolle, Lindeboom, Schuengel, Oosterman, Mirjam;
Boer & Lindauer (2012). Multidimensional treatment foster
care for preschoolers: Early findings of an implementation in
the Netherlands. Child and Adolescent Psychiatry and
Mental Health, 6,38 -
PubMed
PsychINFO
participants
described to be
attending
preschool/nursery
settings
Title review
Chamberlain, Roberts, Jones, Sosna & Price. (2012). Three
collaborative models for scaling up evidence-based
practices. Administration and Policy in Mental Health, 39,
278 – 290
PubMed
Does not contain
primary empirical
data
Abstract
Review
Leve, Fisher & Chamberlain. (2009). Multidimensional
Treatment Foster Care as a preventative intervention to
promote resiliency among youth in the child welfare system.
Journal of Personality, 77, 1869 - 18902
PubMed
Does not contain
primary empirical
data
Abstract
Review
PubMed
PsychINFO
Measures were
not of participant
externalising
behaviour
Abstract
Review
Saldana & Chamberlain. (2012). Supporting Implementation:
the role of community development teams to build
infrastructure. American Journal of Community Psychology,
50, 74 – 85
PubMed
Measures were
not of participant
externalising
behaviour
Title Review
Graham, Yockelson, Kim, Bruce, Pears & Fisher, (2012).
Effects of maltreatment and early intervention on diurnal
cortisol slope across the start of school: A pilot study. Child
Abuse & Neglect.36. 666-670.
PsychINFO
Measures were
not of participant
externalising
behaviour
Title review
Holmes, Ward & McDermid. (2012). Calculating and
comparing the costs of multidimensional treatment foster
care in English local authorities. Children and Youth
Services Review.34. 2141-2146.
PsychINFO
Science
Direct
Measures were
not of participant
externalising
behaviour
Abstract
Review
Chamberlain, Saldana, Brown, Hendricks & Leve. (2011).
Implementation of multidimensional treatment foster care
in California: A randomized control trial of an evidencebased practice. In Roberts-DeGennaro & Fogel (Eds.), Using
evidence to inform practice for community and
organizational change. (pp. 218-234). Chicago: Lyceum
Books
PsychINFO
Not published in a
peer-reviewed
journal
Title review
Feil, Edward G; Sprengelmeyer, Peter G; Davis, Betsy;
Chamberlain, Patricia. (2012). Development and testing of
a multimedia Internet-based system for fidelity and
monitoring of multidimensional treatment foster care.
Journal of Medical Internet Research. 14, 239-248.
23
Doctorate in Educational and Child Psychology
Rachel Standen
Van Ryzin, & Leslie. (2012). Affiliation with delinquent peers
as a mediator of the effects of multidimensional treatment
foster care for delinquent girls. Journal of Consulting and
Clinical Psychology. 80, 588-596.
PubMed
PsychINFO
ERIC
Measures limited
to behaviours of
an illegal nature
only
Abstract
Review
Farmer, Burns, Wagner, Murray & Southerland, (2010).
Enhancing "usual practice" treatment foster care: Findings
from a randomized trial on improving youths' outcomes.
Psychiatric Services. 61. 555-561.
PubMed
PsychINFO
Alternative
Intervention
Abstract
Review
Heywood, (2010). Predictors of favourable outcomes among
children in foster care: An analysis of early childhood
variables and their relationship to the development of
assets. Dissertation Abstracts International: Section B: The
Sciences and Engineering. Vol.70, 5866.
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Tininenko. (2009). Actigraphic evaluation of sleep
disturbance in young children. Dissertation Abstracts
International: Section B: The Sciences and Engineering.69.
5061.
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Buchanan. (2009). An investigation of predictors of
educational engagement for severely antisocial girls.
Dissertation Abstracts International Section A: Humanities
and Social Sciences.69.3028.
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Kerr, Leve, Chamberlain, (20099). Pregnancy rates among
juvenile justice girls in two randomized controlled trials of
multidimensional treatment foster care. Journal of
Consulting and Clinical Psychology. 77. 588-593
PubMed
PsychINFO
ERIC
Measures were
not of participant
externalising
behaviour
Abstract
Review
Bruce, McDermott, Fisher & Fox (2009). Using behavioral
and electrophysiological measures to assess the effects of a
preventive intervention: A preliminary study with preschoolaged foster children. Prevention Science. 10. 129-140
PubMed
PsychINFO
participants
described to be
attending
preschool/nursery
settings
Title Review
Price, Chamberlain, Landsverk & Reid, (2009). KEEP fosterparent training intervention: Model description and
effectiveness. Child & Family Social Work. 14, 233-242.
PsychINFO
Does not contain
primary empirical
data
Abstract
Review
Miller, Keith Andrew. (2008). Exploring placement instability
among young children in the multidimensional treatment
foster care preschool study. Dissertation Abstracts
International: Section B: The Sciences and Engineering. 68.
6974.
PsychINFO
participants
described to be
attending
preschool/nursery
settings
Title Review
Fisher, Philip A; Kim, Hyoun K (2009). Intervention effects on
foster preschoolers’ attachment-related behaviors from a
randomized trial. Prevention Science. 8, 161-170.
PubMed
PsychINFO
participants
described to be
attending
preschool/nursery
settings
Title Review
24
Doctorate in Educational and Child Psychology
Rachel Standen
Kazdin, Alan E. (2007). Psychosocial treatments for conduct
disorder in children and adolescents. In Nathan, & Gorman,
(eds.) A guide to treatments that work pp. 71-104. New
York: Oxford University Press
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Powell, & Lochman, (2004). Multidimensional Treatment
Foster Care: A Conceptually Based, Empirically Validated
Approach to Treating Antisocial Behavior in Youth..
PsycCRITIQUES. Vol.49
PsychINFO
Does not contain
primary empirical
data
Abstract
Review
Smith, Dana K. (2004). Risk, Reinforcement, Retention in
Treatment, and Reoffending for Boys and Girls in
Multidimensional Treatment Foster Care. Journal of
Emotional and Behavioral Disorders. 12, 38-48.
PsychINFO
ERIC
Measures limited
to behaviours of
an illegal nature
only
Abstract
Review
Chamberlain, Patricia. (2003). The Oregon Multidimensional
Treatment Foster Care model: Features, outcomes, and
progress in dissemination. Cognitive and Behavioral
Practice.10 303-312.
PsychINFO
Science
Direct
Does not contain
primary empirical
data
Abstract
Review
Chamberlain, Patricia. (2003). Treating chronic juvenile
offenders: Advances made through the Oregon
multidimensional treatment foster care model. In Antisocial
behaviour and delinquency in girls. Pp. 109-127. Washington
DC: American Psychological Association; 2003
PsychINFO
Science
Direct
Not published in a
peer-reviewed
journal
Title Review
Smith, Dana K. (2003) Gender differences in behaviour
change during treatment with chronically delinquent
youths. Dissertation Abstracts International: Section B: The
Sciences and Engineering.63, 1575.
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Chamberlain, Patricia; Fisher, Philip A; Moore, Kevin. (2002)
Multidimensional treatment foster care: Applications of the
OSLC intervention model to high-risk youth and their
families. In Reid, Patterson, & Snyder (Eds.) (2002).
Antisocial behaviour in children and adolescents: A
developmental analysis and model for intervention. Pp. 203218. Washington, DC: American Psychological Association
PsychINFO
Not published in a
peer-reviewed
journal
Title Review
Moore, Sprengelmeyer, & Chamberlain, (2001). Communitybased treatment for adjudicated delinquents: The Oregon
Social Learning Center’s “Monitor” Multidimensional
Treatment Foster Care program. Residential Treatment for
Children & Youth. 18, 87-97.
PsychINFO
Does not contain
primary empirical
data
Abstract
Review
Fisher, Philip A; Chamberlain, Patricia. (2001).
Multidimensional treatment foster care: A program for
intensive parenting, family support, and skill building. In
Walker & Epstein, (Eds). (2001). Making schools safer and
violence free: Critical issues, solutions, and recommended
practices. Pp. 140-149 Austin: PRO-ED
PsychINFO
ERIC
Not published in a
peer-reviewed
journal
Title Review
Eddy, & Chamberlain, (2000). Family management and
deviant peer association as mediators of the impact of
treatment condition on youth antisocial behavior. Journal
of Consulting and Clinical Psychology.68, 857-863.
PubMed
PsychINFO
Measures limited
to behaviours of
an illegal nature
only
Abstract
Review
25
Doctorate in Educational and Child Psychology
Rachel Standen
Chamberlain & Reid (1998). Comparison of two community
alternatives to incarceration for chronic juvenile offenders.
Journal of Consulting and Clinical Psychology, 66, 624 - 633
PubMed
Measures limited
to behaviours of
an illegal nature
only
Abstract
Review
Fisher & Chamberlain, (2000) Multidimensional treatment
foster care: A program for intensive parenting, family
support, and skill building. Journal of Emotional and
Behavioral Disorders.8, 155-164.
PsychINFO
ERIC
Does not contain
primary empirical
data
Abstract
Review
Taylor, Eddy & Biglan (1999). Interpersonal skills training to
reduce aggressive and delinquent behavior: Limited
evidence and the need for an evidence-based system of
care. Clinical Child and Family Psychology Review. 2 169182.
PubMed
PsychINFO
Does not contain
primary empirical
data
Abstract
Review
Smith, Chamberlain & Eddy (2010). Preliminary Support for
Multidimensional Treatment Foster Care in Reducing
Substance Use in Delinquent Boys. Journal of Child &
Adolescent Substance Abuse. 19. 343-358.
ERIC
Measures limited
to behaviours of
an illegal nature
only
Abstract
Review
ERIC
Does not contain
primary empirical
data
Abstract
Review
ERIC
Does not contain
primary empirical
data
Abstract
Review
ERIC
Does not contain
primary empirical
data
Abstract
Review
ERIC
Does not contain
primary empirical
data
Abstract
Review
Leve & Chamberlain. (2005) Association with Delinquent
Peers: Intervention Effects for Youth in the Juvenile Justice
System. Journal of Abnormal Child Psychology 33 339.
Pubmed
ERIC
Measures were
not of participant
externalising
behaviour
Abstract
Review
Vargas & Koss-Chioino. (1992) Working with Culture:
Psychotherapeutic Interventions with Ethnic Minority
Children and Adolescents. Austin: PRO-ED
ERIC
Not published in a
peer-reviewed
journal
Title Review
Henggeler & Sheidow (2012) Empirically Supported FamilyBased Treatments for Conduct Disorder and Delinquency in
Adolescents. Journal of Marital and Family Therapy 38 30-
PubMed
Eric
Does not contain
primary empirical
data
Abstract
Review
Eddy, Bridges Whaley & Chamberlain. (2004). The
prevention of Violent Behavior by Chronic and Serious Male
Juvenile Offenders. A 2-Year Follow-up of a Randomized
Clinical Trial. Journal of Emotional and Behavioral Disorders
12. 2-8.
Chamberlain, Leve ,& DeGarmo. (2007). Multidimensional
Treatment Foster Care for Girls in the Juvenile Justice
System: 2-Year Follow-Up of a Randomized Clinical Trial.
Journal of Consulting and Clinical Psychology 75 187-193.
Antoine & Fisher. (2006). Preparing Foster Children for
School. Education Digest: Essential Readings Condensed for
Quick Review 71. 61-63.
Lee, Bright, Svoboda, Fakunmoju, & Barth. (2011) Outcomes
of Group Care for Youth: A Review of Comparative Studies.
Research on Social Work Practice 2. 177-189.
26
Doctorate in Educational and Child Psychology
Rachel Standen
58.
Jonkman, Schuengel, Lindeboom , Oosterman, Boer &
Lindauer. (2013) The Effectiveness of Multidimensional
Treatment Foster Care for Preschoolers (MTFC-P) for young
children with severe 27behaviour27l disturbances: study
protocol for a randomized controlled trial. Trials. 14, 197
PubMed
participants
described to be
attending
preschool/nursery
settings
Title Review
Palinkas, Fuentes, Finno, Garcia, Holloway & Chamberlain.
(2014). Inter-Organizational
Collaboration in
the
Implementation of Evidence-based Practices Among Public
Agencies Serving Abused and Neglected Youth.
Administration and Policy in Mental Health.41, 74-85.
PubMed
Measures were
not of participant
externalising
behaviour
Title Review
Leve & Chamberlain. (2007). A randomised Evaluation of
Multidimensional Treatment Foster Care: Effects on School
Attendance and Homework Completion in Juvenile Justice
Girls. Research on Social Work Practice,17, 657 – 663
ERIC
Measures were
not of participant
externalising
behaviour
Article
Review
Palinkas, Fuentes, Finno, Garcia, Holloway & Chamberlain.
(2014). Inter-Organizational
Collaboration in
the
Implementation of Evidence-based Practices Among Public
Agencies Serving Abused and Neglected Youth.
Administration and Policy in Mental Health.41, 74-85.
PubMed
PsychINFO
Measures limited
to behaviours of
an illegal nature
only
Article
Review
Fisher. P, Kim. H & Pears. K. (2009). Effects of
Multidimensional Foster Care for Preschoolers (MTFC-P) on
reducing permanent placement failures among children
with placement instability. Children and Youth Services
Review, 31, 541 – 546
Science
Direct
participants
described to be
attending
preschool/nursery
settings
Title Review
Biehal, Ellsion & Sinclair. (2011). Intensive Fostering: An
independent evaluation of MTFC in an English Setting.
Children and Youth Services Review, 33, 2043 – 2049
Science
Direct
Measures limited
to behaviours of
an illegal nature
only
Article
Review
Lunch. F, Dickerson. J, Saldana. L & Fisher. P. (2014).
Incremental net benefit of early intervention for preschoolaged children with emotional and behavioural problems in
foster-care. Children and Youth Services Review, 34, 213 –
219
Science
Direct
participants
described to be
attending
preschool/nursery
settings
Title Review
Westermark. P, Hansson. K & Vinnerljung. B. (2007). Foster
parents in Multidimensional Treatment Foster Care: How do
they deal with omplementation standardized treatment
components? Children and Youth Services Review, 29, 442 –
459
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Saldana. L, Chamberlain. P. Bradford. D, Campbell. M &
Landsverk. J. (2013). The cost of implementing new
strategies (COINS): A method for mapping implementation
resources using the stages of implementation completion.
Children and Youth Services, IN PRESS.
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
27
Doctorate in Educational and Child Psychology
Rachel Standen
Healey. C & Fisher. P (2011) Young children in foster care
and the development of favorable outcomes. Children and
youth Services Review, 33, 1822 – 1830
Science
Direct
participants
described to be
attending
preschool/nursery
settings
Article
Review
Bertram. J, Carter. S, Narendorf, J & McMillen. C. (2013).
Pioneering the Psychiatric Nurse Role in Foster Care.
Archives of Psychiatric nursing, 27, 285 – 292
Science
Direct
Does not contain
primary empirical
data
Title Review
Tripodi. S & Bender. K. (2011). Substance-Abuse treatment
for juvenile offenders: a review of quasi-experimental and
experimental research. Journal of Criminal Justice, 39, 246 –
252
Science
Direct
Does not contain
primary empirical
data
Title Review
Gonzalez. A & Weersing. V. (2011). Modes of Intervention.
In Encyclopedia of Adolescence pp191 – 201. Academic
Press
Science
Direct
does not contain
primary empirical
data
Title Review
Korbin. J, Brazier. S & Swinger. H. (1980). The United States’
fourth national conference on child abuse and neglect.
Achild Abuse and Neglect, 3, 157 – 162
Science
Direct
Does not contain
primary empirical
data
Title Review
Fang. L, Wang. M Li. D & Pan .J. (2014). CPU/GPU near realtime preprocessing to ZY-3 Satellite images: Relative
radiometric
correction.
MTF
compensation,
and
geocorrection. ISPRS Journal of Photogrammetry and
Remote Sensing, 87, 229 – 240
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Carlson. R, Manolis. D, Papamichail. I & Papageorgiou. M.
(2012). Integrated Ramp Metering and Mainstream Traffic
Flow Control on Freeways using Variable Speed Limits.
Procedia – Social and Behavioural Sciences, 48, 1578 – 1588
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Nasuti. A, Pascal, C & Ebbing. J. (2012). Onshore-Offshore
potential field analysis of the More-Trondelag Fault Complex
and adjacent structure of Mid Norway. Tectonophysics, 518,
17 – 28
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Carlson. R Papamichail. I, Papageorgiou. M & Mesmer.
(2010). Optimal mainstream traffic flow control of largescale motorway networks. Transportation Research part C:
Emerging Technologies, 18, 193 – 212
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Lee, Yang, Seo, Song, Chung & Lim. (2011). Image
restoration of the asymmetric point spread function of a
high-resolution remote sensing satellite with time-delayed
integration. Advances in Space Research, 47, 690 - 701
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Redfield. T, Torsvik. T, Andriessen. P & Gabrielson. R. (2003).
Mesozoic and Cenozoic tectonics of the More Trondelag
Fault Complex, central Norway: constraints from new
apatite fission track data. Physics and Chemistry of the
Earth, 29, 673 – 682
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
28
Doctorate in Educational and Child Psychology
Rachel Standen
Ryan. R, Baldridge. B, Schowengerdt. R, Choi. T, Helder. D &
Blonski. S. (2003). IKONOS spatial resolution and image
interpretability characterization. Remote Sensing of
Environment, 88, 37 – 52
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Goward. S, Davis. P, Fleming. D, Miller. L & Townsend J.
(2003). Empirical comparison of Landsat 7 and IKONOS
multispectral measurement for selected Earth Observation
system (EOS) validation sites. Remote sensing of
Environment, 88, 80 – 99
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Redfield. T, Braathan, A, Gabrielson. R, Osmundsen. P,
Torsvik. T & Andriessen. P. (2005). Late Mesozoic to Early
Cenozoic components of vertical serperation across the
More-Trondelag Fault complex, Norway. Tectonophysics,
395, 233 – 249.
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Pagnutti. M, Ryan. R, Kelly. M, Holekap. K, Zanoni. V,Kurtis. T
& Schiller. S. (2003). Radiometric characterisation of IKONOS
multispectral imagery. Remote Sensing of Environment, 88,
53 – 68
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
Skilbrei. J & Oleson. O (2005). Deep structure of the MidNorweigen shelf and onshopre-offshore correlations: Insight
from potential field data. Norweigen Petroleum Society
Special Publications, 12, 43 – 68
Science
Direct
Measures were
not of participant
externalising
behaviour
Title Review
ERIC
Does not contain
primary empirical
data
Abstract
Review
Eddy, Bridges Whaley & Chamberlain. (2004). The
prevention of Violent Behavior by Chronic and Serious Male
Juvenile Offenders. A 2-Year Follow-up of a Randomized
Clinical Trial. Journal of Emotional and Behavioral Disorders
12. 2-8.
29
Doctorate in Educational and Child Psychology
Rachel Standen
Appendix. C: Explanation of Weight of Evidence protocol used
The Kratochwill (2003) coding protocol was used to weight the experimental (Randomised Control
Trial) studies. Each study was weighted on ‘Measures’, ‘Comparison Group’ and ‘Analysis’. These
weightings have numerical values (0 – 3) enabling overall methodological quality to be analysed via
the total points awarded. Descriptors are outlined below.
A: Methodological Quality
Measure
Strong (3 points): studies must;
Promising (2 points): studies
must;
Weak (1 point): studies must;
Unable to Code (0 points):
studies must;

Use measures with reliability coefficient of at least 0.85 and be stated
explicitly.
 Explicitly stated the rationale for choosing the measures
 Used multiple methods of data collection
 Used multiple sources of information

Use measures with reliability coefficient of at least 0.70

Used multiple methods of data collection
and/or

Used multiple sources of information

Use measures with reliability coefficient of at least 0.50

Used multiple methods of data collection
and/or

Used multiple sources of information

Use an unreliable measure (<0.50)

Used single method of data collection
and/or

Used single source of information
Comparison group
Strong (3 points): studies must;
Promising (2 points): studies
must;
Weak (1 point): studies must;
Unable to Code (0 points):
 Use at least one “active” comparison group
 Initial group equivalency must be established
 Evidence the use of change agents
 Meet criteria for equivalent mortality and low attrition
 Use at least one “no intervention” comparison group
And meet two out of the following criteria:
 Initial group equivalency must be established
 Evidence the use of change agents
 Meet criteria for equivalent mortality and low attrition
 Use at least one comparison group
And meet one of the following criteria:
 Initial group equivalency must be established
 Evidence the use of change agents
 Meet criteria for equivalent mortality and low attrition

Studies made no attempt to establish group equivalence.
30
Doctorate in Educational and Child Psychology
Rachel Standen
Appropriate Statistical Analysis
Strong (3 points): studies must;
Promising (2 points): studies
must include three out of the
four criteria;
Weak (1 point): studies must
include two out of the four
criteria;
Unable to Code (0 points):
studies must include less than
two out of the four criteria;
 Use an appropriate model to the research question
 Use an appropriate unit of measurement corresponding to the
research question
 Control for Familywise error rate
 Use a sufficiently large N
 Use an appropriate model to the research question
 Use an appropriate unit of measurement corresponding to the
research question
 Control for Familywise error rate
 Use a sufficiently large N
 Use an appropriate model to the research question
 Use an appropriate unit of measurement corresponding to the research
question
 Control for Familywise error rate
 Use a sufficiently large N
 Use an appropriate model to the research question
 Use an appropriate unit of measurement corresponding to the research
question
 Control for Familywise error rate
 Use a sufficiently large N
Overall Methodological Quality (A)
Calculated by determining the mean score across each criteria, with each criteria holding equal weight.
Overall
Methodological
Quality
(A)
Measures
Comparison
Group
Westermark,
Hansson, &
Olsson.
(2011)
Strong
Promising
Weak
Medium
3
2
1
2
Hansson
Olsson.
(2012)
Strong
Promising
Weak
Medium
3
2
1
2
Promising
Promising
Weak
Medium
3
2
2
2.3
Promising
Promising
Promising
Medium
2
2
2
2
---
Promising
Weak
Low
0
2
1
1
&
Green et al.
(2013)
Chamberlain
et al. (2008)
Leve,
Chamberlain
&
Reid.
(2005)
31
Analysis
Doctorate in Educational and Child Psychology
Rachel Standen
Overall Methodological Quality
Mean Score
High
At least 2.5
Medium
Between 1.5 and 2.4
Low
Less than 1.4
B: Methodological Relevance
This weighting considers whether the design was suitable for evaluating the effectiveness of using
Multidimensional Treatment Foster Care to improve behavioural outcomes for young people.
High (3 points): studies must;
Medium (2 points): studies
must;
Low (1 point): studies must;
Unable to Code (0 points):
studies must;





Randomly assigned participants to the MTFC group and control group
Demonstrated group equivalence
Taken pre and post measures for the MTFC and control group.
Used a control group
Demonstrated group equivalence or matched control pupils using a
behavioural measure
 Taken pre and post measures for the intervention and comparison
group
 Taken pre and post measures
 Studies are descriptive in nature only.
C: Relevance of Evidence to the Review Question.
This weighting considers the contribution of the study to answering the review question
High (3 points): studies must;
Medium (2 points): studies
must;
Low (1 point): studies must;
Unable to Code (0 points):;
 Examined the effects of MTFC on participants between the ages of four
and nineteen with evidence of significant challenging behaviours.
 Explicitly state the initial training programme received by MTFC foster
parents to ensure fidelity.
 Explicitly state the ongoing support and training received by MTFC
foster parents to ensure fidelity
 Took empirical measures of behavioural change from multiple
measures/perspectives
 Examined the effects of MTFC on participants between the ages of four
and nineteen with evidence of significant challenging behaviours.
 Refer to training and/or ongoing support received by MTFC foster
parents without explicitly stating the nature or frequency of this.
 Took empirical measures of behavioural change from multiple
measures/perspectives.
 Examined the effects of MTFC on participants between the ages of four
and nineteen
 No reference to training and/or ongoing support received by MTFC
foster parents.
 Took empirical measures of behavioural change from a single
measure/perspective
 Studies do not meet the criteria for a categorisation of Low.
32
Doctorate in Educational and Child Psychology
Rachel Standen
D: Overall Weight of Evidence
Calculated by determining the mean score across each criteria, with each criteria holding equal weight.
Is Multidimensional Treatment Foster Care an effective intervention for reducing externalising behaviours in children with
Social Emotional Behaviour Difficulties, whose challenging behaviours are resulting in their current placement being at risk?
(A)
(B)
Quality of
Methodology
Relevance of
Methodology
(C)
(D)
Relevance of evidence
to the review
question
Overall Weight of
Evidence
Westermark, Hansson,
& Olsson. (2011)
Medium
High
High
High
2
3
3
2.6
Hansson
(2012)
Medium
High
High
High
2
3
3
2.6
Medium
High
High
High
2.3
3
3
2.8
Medium
High
Medium
Medium
2
3
2
2.3
Low
High
Medium
Medium
1
3
2
2
&
Olsson.
Green et al. (2013)
Chamberlain
(2008)
et
al.
Leve, Chamberlain &
Reid. (2005)
Overall Weight of Evidence
Mean Score
High
At least 2.5
Medium
Between 1.5 and 2.4
Low
Less than 1.4
33
Doctorate in Educational and Child Psychology
Rachel Standen
Appendix.D. Coding Protocol
Name of Coder:
Rachel Standen
Date: 18.01.2014
Full Study Reference in proper format:
Westermark, P. K., Hansson, K., & Olsson, M. (2011). Multidimensional treatment foster care (MTFC): results from an independent
replication. Journal of Family Therapy, 33(1), 20–41.
Intervention Name: Multidimensional Treatment Foster Care
Type of Publication:
Book/Monograph
X Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
X
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
X
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 35
Intervention group sample size: 20
Control group sample size:15
C. Type of Program
X
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
X
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
34
Doctorate in Educational and Child Psychology
Rachel Standen
E. Concurrent or Historical Intervention Exposure
Current exposure
X Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following table for a detailed
breakdown on the outcomes)
X Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
X Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
X Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any cultural
considerations)
X Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
X3
2
1
0
B Comparison Group
B1 Type of Comparison group
X Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
X Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were counter-balanced
across intervention)
X
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
35
Doctorate in Educational and Child Psychology
Rachel Standen
B4 Group equivalence established
X Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
X Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Rating of Evidence:
3
X
2
1
0
C Appropriate Statistical Analysis
Analysis 1 YSR
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Analysis 1 CBCL
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Overall Rating of Analysis:
3
2
X1
0
36
Doctorate in Educational and Child Psychology
Name of Coder:
Rachel Standen
Rachel Standen
Date: 18.01.2014
Full Study Reference in proper format:
Hansson, K., & Olsson, M. (2012). Effects of multidimensional treatment foster care (MTFC): Results from a RCT study in Sweden.
Children and Youth Services Review, 34(9), 1929–1936.
Intervention Name: Multidimensional Treatment Foster Care
Type of Publication:
Book/Monograph
X Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
X
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
X
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 46
Intervention group sample size: 19
Control group sample size: 27
C. Type of Program
X
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
X
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
37
Doctorate in Educational and Child Psychology
Rachel Standen
E. Concurrent or Historical Intervention Exposure
Current exposure
X Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following table for a detailed
breakdown on the outcomes)
X Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
X Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
X Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any cultural
considerations)
X Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
X3
2
1
0
B Comparison Group
B1 Type of Comparison group
X Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
X Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were counter-balanced
across intervention)
X
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
38
Doctorate in Educational and Child Psychology
Rachel Standen
B4 Group equivalence established
X Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
X Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Rating of Evidence:
3
X
2
1
0
C Appropriate Statistical Analysis
Analysis 1 YSR
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Analysis 1 CBCL
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Overall Rating of Analysis:
3
2
X1
0
39
Doctorate in Educational and Child Psychology
Name of Coder:
Rachel Standen
Rachel Standen
Date: 18.01.2014
Full Study Reference in proper format:
Green, J. M., Biehal, N., Roberts, C., Dixon, J., Kay, C., Parry, E., … Sinclair, I. (2013). Multidimensional Treatment Foster Care for
Adolescents in English care: randomised trial and observational cohort evaluation. The British journal of psychiatry : the journal of mental
science.
Intervention Name: Multidimensional Treatment Foster Care
Type of Publication:
Book/Monograph
X Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
X
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
X
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 33
Intervention group sample size: 20
Control group sample size: 13
C. Type of Program
X
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
X
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
40
Doctorate in Educational and Child Psychology
Rachel Standen
E. Concurrent or Historical Intervention Exposure
X
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following table for a detailed
breakdown on the outcomes)
X Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
X Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
X Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any cultural
considerations)
X Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
X3
2
1
0
B Comparison Group
B1 Type of Comparison group
X Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
X Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were counter-balanced
across intervention)
By change agent
Statistical (analyse includes a test for intervention)
41
Doctorate in Educational and Child Psychology
Rachel Standen
Other
X Not reported/None
B4 Group equivalence established
X Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
X Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Rating of Evidence:
3
X
2
1
0
C Appropriate Statistical Analysis
Analysis 1 CGAS
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Overall Rating of Analysis:
3
2
X1
0
42
Doctorate in Educational and Child Psychology
Name of Coder:
Rachel Standen
Rachel Standen
Date: 18.01.2014
Full Study Reference in proper format:
Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. a, & Reid, J. B. (2008). Prevention of behavior problems for children in
foster care: outcomes and mediation effects. Prevention Science : The Official Journal of the Society for Prevention Research, 9(1), 17–27.
Intervention Name: Multidimensional Treatment Foster Care
Type of Publication:
Book/Monograph
X Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
X
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
X
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 700
Intervention group sample size: 359
Control group sample size: 341
C. Type of Program
X
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
X
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
43
Doctorate in Educational and Child Psychology
Rachel Standen
E. Concurrent or Historical Intervention Exposure
Current exposure
X Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following table for a detailed
breakdown on the outcomes)
X Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
X No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
X No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any cultural
considerations)
X
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
X
0
B Comparison Group
B1 Type of Comparison group
X Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
X Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were counter-balanced
across intervention)
X
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
44
Doctorate in Educational and Child Psychology
Rachel Standen
B4 Group equivalence established
X Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
X Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Rating of Evidence:
3
X
2
1
0
C Appropriate Statistical Analysis
Analysis 1PDR
X Appropriate use of analysis
X
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Overall Rating of Analysis:
3
X
2
1
0
45
Doctorate in Educational and Child Psychology
Name of Coder:
Rachel Standen
Rachel Standen
Date: 18.01.2014
Full Study Reference in proper format:
Leve, L. D., Chamberlain, P., & Reid, J. B. (2005). Intervention outcomes for girls referred from juvenile justice: effects on delinquency.
Journal of Consulting and Clinical Psychology, 73(6), 1181–5.
Intervention Name: Multidimensional Treatment Foster Care
Type of Publication:
Book/Monograph
X Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
X
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
X
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 81
Intervention group sample size: 37
Control group sample size: 44
C. Type of Program
X
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
X
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
46
Doctorate in Educational and Child Psychology
Rachel Standen
E. Concurrent or Historical Intervention Exposure
X
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following table for a detailed
breakdown on the outcomes)
X Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
X No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
X No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any cultural
considerations)
X Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
X
2
1
0
B Comparison Group
B1 Type of Comparison group
X Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
X Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were counter-balanced
across intervention)
X
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
47
Doctorate in Educational and Child Psychology
Rachel Standen
B4 Group equivalence established
X Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
X Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Rating of Evidence:
3
X
2
1
0
C Appropriate Statistical Analysis
Analysis 1CBCL
X Appropriate use of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Required N = 64 per group.
No error rate control
Overall Rating of Analysis:
3
2
X1
0
48