THE EFFECTIVENESS OF APPLIED
BEHAVIOUR ANALYSIS
INTERVENTIONS FOR PEOPLE
WITH AUTISM SPECTRUM
DISORDER
- AN UPDATE OF SECONDARY
LITERATURE
Report to the New Zealand Ministry of Health
30 September 2009
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
2
PRINCIPAL AUTHOR
Marita Broadstock
ACKNOWLEDGEMENTS
Acknowledgements
This systematic review was undertaken by staff of the New Zealand Guidelines
Group, and completed in September 2009. The scope of the review was agreed with
the Living Guidelines Group project’s sponsor, Joanna Curzon (Ministry of Health)
and was funded by the Ministry of Health to contribute to the work of the Living
Guideline Group.
Marita Broadstock (Senior Researcher) conducted the review and prepared the
report and Evidence Tables. Margaret Paterson (NZGG Information Specialist)
conducted the search strategy and managed document retrieval and referencing. A
list of excluded publications is available upon request. Anne Lethaby (Acting
Manager, Research Services) provided methodological input and peer review.
Please cite this report as:
New Zealand Guidelines Group. The effectiveness of applied behaviour analysis
interventions for people with autism spectrum disorder – an update of secondary literature.
Wellington; 2009.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
3
SYNTHESIS OF RECENTLY PUBLISHED SECONDARY
LITERATURE ON APPLIED BEHAVIOUR ANALYSIS
BACKGROUND AND SCOPE
In 2008, the New Zealand Ministries of Health, and Education, commissioned two
technical reviews on interventions and strategies for people with Autism Spectrum
Disorder (ASD) grounded in the principles of applied behaviour analysis (ABA)
(Ministries of Health and Education, 2007). These two reviews were completed in
parallel by Uniservices and New Zealand Guidelines Group (NZGG) respectively and
based on research published to 31 December 2007. These reports represent the
prime evidence for the Living Guideline Group (LGG) to consider in determining
whether any revision of Recommendations relevant to ABA is needed in the New
Zealand Autism Spectrum Disorder Guideline (2008). However, as this field has been
the subject of significant research interest, it was considered valuable to supplement
the New Zealand commissioned research with a synthesis of other high level
evidence that has emerged since the reviews’ search cut-off period.
In a hierarchy of evidence for studies of intervention effectiveness, the highest level
evidence (Level I) consists of systematic reviews and meta analyses which include
(but are not limited to) randomised controlled trials (RCTs) (level II evidence)
(National Health and Medical Research Council, 2008). This current update is limited
to level I evidence. Such secondary evidence offers the opportunity to consider a
range of approaches to reviewing and synthesising the evidence relating to ABA and
compare conclusions with those of the two comprehensive parallel technical reviews.
OBJECTIVES AND RESEARCH QUESTION
The objective of this update is to critically appraise secondary research evidence
published since December 2007 relating to the effectiveness of interventions and
strategies grounded in the principles of applied behaviour analysis for people with
autism spectrum disorder.
ABA-based interventions can be defined as ‘those in which the principles of learning
theory are applied in a systematic and measurable manner to increase, reduce,
maintain and/or generalise target behaviours’ (Ministries of Health and Education,
2007). Well-established principles and techniques of ABA include (a) reinforcement,
(b) shaping, (c) chaining, (d) fading, (e) response and stimulus prompting, (f)
discrimination training, (g) programming, and (h) functional assessment (Broadstock
& Lethaby, 2008).
The clinical question employed for this update of secondary literature was that used
for the parallel technical reviews (Ministries of Health and Education, 2007):
To what extent are interventions and strategies based on the principles of applied
behaviour analysis effective in leading to the following outcomes for people with
autism spectrum disorders:
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
4

social development and relating to others

development of cognitive (thinking) skills

development of functional and spontaneous communication which is used in
natural environments

engagement and flexibility in developmentally appropriate tasks and play and
later engagement in vocational activities

development of fine and gross motor skills

prevention of challenging behaviours and substitution with more appropriate
and conventional behaviours

development of independent organisational skills and other behaviours

generalisation of abilities across multiple natural environments outside the
treatment setting

maintenance of effects after conclusion of intervention

improvement in behaviours considered non-core ASD behaviours, such as
sleep disturbance, self mutilation, aggression, attention and concentration
problems.
REVIEW METHODS
A systematic method of literature searching, selection and appraisal was employed in
the preparation of this report, consistent with New Zealand Guidelines Group (NZGG)
review processes (Broadstock & Lethaby, 2008).
The search was limited to articles published in the English language between
January 1 2008 and August 6 2009 (when the search was conducted). Sources
included general bibliographic databases (Medline, PsychINFO, EMBASE, CINAHL,
ERIC, Cochrane Library) as well as various health technology assessment/guideline
databases, and cross-checking of references from retrieved references.
Selection criteria were based on those used for the NZGG technical review
(Broadstock & Lethaby, 2008), modified to solely identify on secondary studies.
Studies were included if they:

were secondary research (systematic reviews and meta-analyses)

were published on or after January 1st 2008

had a clear review question

used at least two searching sources
and reported on studies:

of eligible interventions (studies which evaluated interventions which were
predominantly based on the principles of applied behaviour analysis)

considering comparators including usual care, another intervention or
application of interventions

considering individuals with a diagnosis of autism spectrum disorder or where
results are reported separately for this group.
Research papers were excluded if they:
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
5

were non-systematic reviews, correspondence, editorials, expert opinion
articles, comments, case reports, book chapters, articles published only in
abstract form, conference proceedings, news items, unpublished work

were primary studies (except where reported in included systematic reviews

did not provide separate analyses/syntheses of results for eligible
interventions and eligible population

were not deemed appropriate to the research question or nature of review,
including those reporting on outcomes solely relating to safety; the
acceptability of, or ethical, economic or legal considerations relating to ABA;
or the impact on persons other than those diagnosed with ASD.
STUDY SELECTION AND APPRAISAL
Selection criteria were applied by a single reviewer to abstracts/titles identified by the
search strategy, and again to those retrieved as full text, to identify the final set of
included papers for critical appraisal. All included studies represented level I
evidence (National Health and Medical Research Council, 2008) as specified in the
selection criteria, being systematic reviews or meta analyses of studies that included
randomised controlled trials.
Included studies were formally critically appraised using NZGG’s quality checklists
which are based on the GATE Frame tools designed by Effective Practice,
Informatics and Quality improvement (EPIQ), within the School of Population Health
at the University of Auckland. Twenty questions addressed methodological quality,
including whether the review’s methods were internally valid, precision of results, and
the applicability/external validity of the review. An overall quality code was applied
based on the appraisal to summarise the quality of each included study. Overall
study quality was categorised qualitatively based on summary questions on the
GATE checklist and using one of the following descriptors: “very good”, “good”, or
“fair”.
Details of each study were entered into Evidence Tables, including aspects of
methodology, results, authors’ conclusions, reviewer’s additional comments, and the
summary study quality descriptor. The included studies are presented alphabetically
by first author within two intervention type categories, (1) applied behaviour analysis
interventions (defined broadly), and (2) Picture Exchange Communication System
(PECS) interventions (see Evidence Tables).
DATA SYNTHESIS
Studies were narratively synthesised to determine the strength of evidence. Strength
of evidence is determined by three domains (West, King, & Carey, 2002):
-
quality (the extent to which bias was minimised);
-
quantity (magnitude of effect, numbers of studies, sample size or power);
-
consistency (the extent to which similar findings are reported).
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
6
KEY FINDINGS AND CONCLUSIONS
Overview
Of 461 articles identified by the search strategy, 35 publications were retrieved as full
text. Twelve of these met selection criteria and were deemed eligible for critical
appraisal and inclusion in this synthesis.
Of the 12 included secondary studies, 10 reported on evaluations of interventions
broadly based on applied behaviour analysis, and two reported on evaluations of the
Picture Exchange Communication System (PECS) intervention specifically (see
Table).
The 10 reviews reporting on ABA-based interventions (in general) used varying
descriptive terms for included interventions, reflecting that there is no universally
accepted definition of ABA-based interventions. For example, some studies refered
specifically to Early Intensive Behavioural Interventions (EIBI) as being restricted to
interventions based on the work of Lovaas (1987), whereas others included as EIBI
any intensive behavioural intervention directed at children in pre-school years. In this
report the broader definition is used.
Of the 10 reviews, four included meta analyses (Sigmund Eldevik et al., 2009;
Ospina et al., 2008; Reichow & Wolery, 2009; Spreckley & Boyd, 2009b); all four
were rated as being of good (Reichow & Wolery, 2009) or very good quality
(Sigmund Eldevik et al., 2009; Ospina et al., 2008; Spreckley & Boyd, 2009b).
Four of the 12 reviews (33%) were classified as being of “very good” quality (Blue
Cross and Blue Shield Association, 2009; Sigmund Eldevik et al., 2009; Ospina et al.,
2008; Spreckley & Boyd, 2009b), another third were of “good” quality (Svein
Eikeseth, 2009; Howlin, Magiati, & Charman, 2009; Reichow & Wolery, 2009; Rogers
& Vismara, 2008), and the remaining final four systematic reviews were of “fair”
quality, including two considering ABA in general (Case-Smith & Arbesman, 2008;
Seida et al., 2009), and two reporting on PECS (Ostryn, Wolfe, & Rusch, 2008;
Sulzer-Azaroff, Hoffman, Horton, Bondy, & Frost, 2009).
It should be noted that as these reviews were all published across an 18 month
period on the same broad topic area, they tend to report on the same studies, and
also overlap with the studies included in the parallel technical reviews of NZGG and
Auckland Uniservices. For this reason it is important to be aware that the results and
the studies appraised should not be summated as independent sources of evidence
as this would misrepresent the “quantity” of studies and give the individual studies
undue weight.
No additional primary studies published post December 2007 and eligible for
inclusion in either of the two technical reviews were identified by the 12 reviews, and
so the evidence “catchment area” is broadly the same as that considered by the
parallel technical reviews, although some included studies prior to 1998, the earliest
date for publications considered in the parallel technical reviews. In considering
secondary reviews which consider overlapping but different selection criteria and
methodological approaches, attention should be given to the consistency of
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
7
conclusions. More weight/emphasis should be placed on the reviews that are of
higher quality. For this reason, the following synthesis will consider the findings and
conclusions of the “very good” quality reviews first.
Reviews of “very good” quality
Four reviews were rated as being of “very good” quality using the systematic critical
appraisal process outlined above. All considered ABA interventions in general,
though the focus was on studies of early intensive behavioural analysis.
The systematic review published by Blue Cross and Blue Shield Association’s
(2009) Technical Evaluation Centre considered evidence published 1987 - July 2008.
The review considered the use of EIBI based on applied behaviour analysis. It
excluded single-case experimental studies (SCED), arguing that such designs lack
generalisability across individuals, and are of limited value in evaluating EIBI due to
methodological constraints.
The reviewers included 16 studies, including 2 RCTs. They concluded that,
“weaknesses in research design, differences in the treatments and outcomes
compared, and inconsistent results, mean that the impact of EIBI versus other
treatments on outcomes for children with autism cannot be determined.” They argued
that the weakness of the evidence base and variability in findings meant that they
could also not reach conclusions about the impact of EIBI on any specific
domains/outcomes, or on whether greater intensity of EIBI can lead to better
outcomes.
The authors called for research to identify what aspects of an intervention and its
delivery lead to greatest positive effect, including content, technique, intensity,
starting and ending age. A greater emphasis was said to be needed on randomized,
controlled trials with substantially larger sample sizes, uniformity of outcomes and
instruments, and consistent treatments.
The systematic review by Eldevik and Hastings (2009) considered nine controlled
group studies of EIBI in their meta analyses. They reported moderate to large effect
sizes for adaptive behaviour composite (ABC) and full-scale IQ scores, respectively,
compared with controls or eclectic comparators. It is concluded that, “in the absence
of other interventions with established efficacy, EIBI should be an “intervention of
choice for children with autism.” However the authors cautioned that the improved
outcomes for people receiving EIBI may be due to the EIBI group having received, in
general, more frequent and more total hours of supervision and training than
comparison groups. They noted that this remains a threat to the validity of their
conclusions and called for RCTs where the comparison intervention is of similar
intensity and where staff receive similar training and supervision.
Another SR with meta analysis rated as being very good quality was authored by
Ospina et al (2008). It considered literature published to May 2007 and included 101
controlled studies across a range of interventions ranging from behavioural to
developmental. Those interventions relating to ABA specifically included those based
on Direct Trial Training (DTT) and Lovaas (1987) therapy. With respect to ABA, the
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
8
reviewers concluded from descriptive synthesis that there was “some support” for
discrete trial training in terms of motor and functional skills but not for communication
skills. From meta analyses of 3 controlled clinical trials the authors reported that
Lovaas’ therapy appeared to show benefits when compared to special education on
several domains (adaptive behaviour, communication and interaction,
comprehensive language, daily living skills, expressive language, overall intellectual
functioning, socialisation) with the exception of non-verbal intellectual functioning.
Based on a meta-analysis of two RCTs, they found no difference in communication
skill outcomes for children receiving EIBI compared with a Developmental Individual
Difference Relationship based intervention. Lower level evidence from meta-analysis
of two retrospective cohort studies showed greater effects for “high intensity” versus
“low intensity” Lovaas.
These findings were tempered by the limitations of the evidence base, with the
authors noting that, “these findings are based on pooling of a few, methodologically
weak studies with few participants and relatively short-term follow-up”. The
heterogeneity of the study populations, interventions and outcomes was also noted,
raising questions as to the appropriate use of meta analysis to synthesise these
results. The overall conclusion from across interventions was that there is no clear
answer regarding the most effective therapy to improve symptoms associated with
ASD. The authors called for replication in RCTs to substantiate the use of Lovaas
(1987) and to assess the effect of treatment intensity on outcomes of children with
ASD. They further recommended that clinical management be guided by individual
needs and availability of resources.
The fourth review of very good quality also included a meta analysis (Spreckley &
Boyd, 2009b), and reported on applied behaviour interventions (ABI) for preschool
children with ASD. Randomised or quasi-randomised trials were included, and of the
13 studies meeting initial selection criteria, only six met minimum quality criteria and
four had adequate data to be included in the formal meta analysis. Results
suggested no significant improvement on outcomes for those in ABI programs
compared with standard care on cognitive outcomes, expressive language, receptive
language, or adaptive behaviour. Low to moderate heterogeneity existed across
outcomes which prompts uncertainty about the suitability of meta analytic techniques
here, a limitation the authors acknowledged given high variability in participant
characteristics and programme content. Other limitations of the evidence base
disussed by the authors included difficulty in establishing genuine control groups,
lack of strict selection criteria, and limited information on participant retention. The
authors concluded that there is inadequate evidence that applied behaviour
intervention has better outcomes than standard care for children with autism.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Summary
These high quality reviews offer some agreement and some divergence in their
interpretations of the evidence base. Two reviews, which included meta analyses,
gave tentative evidence of positive benefits of ABA-based interventions over
comparison treatments from a limited evidence base. EIBI was reported as having
beneficial effects on adaptive behaviour and IQ compared with control or eclectic
comparators (Sigmund Eldevik et al., 2009). Some support was reported for discrete
trial training in terms of motor and functional skills but not for communication skills.
Lovaas’ (1987) therapy was presented as having benefit over no treatment, and
greater effect for higher intensity treatment (Ospina et al., 2008). However these
conclusions are qualified by methodological weaknesses of the individual studies and
the small number of studies and participants. In particular, the lack of control for the
intensity of treatment delivered between intervention and comparator is problematic.
In contrast, the two other reviews suggested there was inadequate evidence of
effectiveness of ABA-based early interventions. These conclusions were based on a
lack of significant results in the meta analysis of four studies by Spreckley and Boyd
(2009b). The Blue Cross and Blue Shield Association (2009) report argued that the
limitations and inconsistent results of the evidence base precluded making any
conclusions about the relative effectiveness of EIBI compared with alternative
treatments, about its effectiveness for particular outcome domains, or about the
impact of intensity on outcomes. The report also argued that study heterogeneity
ruled out the use of meta analysis.
All four reviews emphasised the need for additional research addressing
methodological weaknesses, and suggested the need for appropriately powered
RCTs to systematically investigate what characteristics of treatment lead to the best
outcomes. Particularly called for was the use of comparison interventions of similar
intensity and quality to those of EIBI.
Reviews of “good” quality
Four reviews were rated as being of “good” quality, having more flaws and limitations
than the “very good” reviews discussed above, often in terms of a limited search
strategy, lack of systematic quality assessment, and/or lack of detail and precision in
reporting study characteristics and synthesising results. However these studies are
still of reasonably good quality and of value in contributing to the evidence base.
A systematic review of comprehensive psycho-educational research (Eikeseth,
2009) included 20 studies on ABA-based interventions. The reviewers concluded that
intensive ABA treatment carried out by trained therapists is effective in enhancing
“global functioning” in pre-school children with autism, and those with PDD-NOS.
In Howlin et al’s (2009) review of 11 controlled (any comparator) group EIBI studies,
in 9 of 10 studies, IQ scores were significantly higher for children at follow-up in the
EIBI group compared with those in the control/comparator group, with moderate to
large effect sizes in the majority, but not all studies. However the authors also noted
that children receiving EIBI received significantly more (mean) hours of intervention
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
10
per week than those in the comparison group (30 c.f. 19 hours, respectively). It was
concluded that whilst EIBI is highly effective for some children, there is wide
individual variation in response, and some failed to make progress at all. Howlin et al
(2009) also noted that chronological age and length of intervention did not appear to
be related to outcome, with varying results for initial language level as a predictor.
Initial IQ strongly predicted improvement after EIBI treatment in four studies, but was
unassociated with outcome in a fifth.
A systematic review and meta analysis by Reichow and Wolery (2009) evaluated 13
controlled group studies on EIBI interventions based on the Lovaas (1987) model
(notably, Eldevik and Hastings, 2009, criticised this as too restrictive a definition for a
review of EIBI’s effectiveness). Meta analyses of 12 studies considering change in IQ
within treatment group suggested a moderately large effect size for improvement at
followup, however the authors cautioned that this finding was limited by excluding
reference to a comparison group. As such, improvements cannot be attributed to
EIBI exclusively; for example, they could relate to maturation effects. As there was
significant variability between study characteristics, the appropriateness of using
meta analytic techniques for synthesising results is questionable. Descriptive and
summative syntheses suggested that EIBI can be an effective treatment, on average,
for some children with autism compared with control/various comparators. However
individual data typically were not presented and the authors noted that the
intervention has not worked for all children, and gains in one domain may not be
accompanied by gains in another domain within individuals. Moreover, comparison
groups were of limited quality and poorly described. The authors concluded that it is
not possible to determine whether EIBI is more or less effective than other treatment
options without comparisons between EIBI and empirically validated treatment
programs.
The systematic review by Rogers and Vismara (2008) considered early
interventions including psychopharmacological ones for young children with autism,
reporting on seven ABA based interventions. From four controlled studies evaluating
Lovaas’s (1987) treatment, the authors argued that this intervention meets criteria for
a ‘‘well-established’’ psychosocial intervention for improving the intellectual
performance (specifically IQ scores) of young children with ASD. “Well established”
interventions were defined as those requiring treatment manuals, clearly specified
participant groups, and either (a) two independent well-designed group studies
showing the treatment to be better than placebo or alternative treatment, or
equivalent to an established effective treatment, or (b) nine or more single-subject
design studies using strong designs and comparison to an alternative treatment
(Chambless et al., 1998). There was less consistency in the data for improvements in
behavior, adaptive skills, and language skills. Overall, the review concluded that
focused daily early intervention programs of several different kinds are beneficial for
young children with autism, however the lack of comparative studies precludes
determining which comprehensive treatment approach is best. Further, the authors
suggested that other well-known interventions may be as or more efficacious as
Lovaas’s (1987) model but they have not been rigorously evaluated.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Summary
The review by Eikeseth (2009) concluded that intensive ABA treatment carried out by trained
therapists is effective in enhancing global functioning in pre-school children with autism, and those
with PDD-NOS. Rogers and Vismara (2008) suggested that Lovaas-based (1987) EIBI met criteria
for a ‘‘well-established’’ intervention for improving the IQ of young children with ASD, but that there
was less consistency for improvements in behavior, adaptive skills, and language. Howlin et al
(2009) and Reichow and Wolery (2009) both concluded that whilst EIBI is highly effective in
increasing IQ for some children, some appear to make no progress, suggesting wide individual
variation in response. The former reviewers also found that age and length of intervention do not
appear to predict response, but that initial language level did in some studies, and initial IQ
appears to be related to impact on outcome in most studies evaluated.
Despite these broadly positive conclusions about the potential for benefit among (at least some)
children with ASD, the reviewers tempered their findings with reference to the methodological
limitations of the evidence base. Major concerns centred around the need to disentangle the
effects of programme content from those of programme intensity, which has not been well
controlled for. To address this, Howlin et al (2009) called for equivalence trials, where high quality
interventions are compared witho other high quality interventions of a similar intensity. Other calls
for further research echoed those of the “very good” studies reported earlier, including the need to
systematically assess characteristics of children for whom EIBI is most effective, optimal duration
and intensity of treatment, and age of commencement of treatment, so that better targetted and
cost-effective interventions could be developed (Howlin et al., 2009).
Another repeated theme concerned the limitations of the comparator or control groups with which
ABA based interventions are compared. These included their lack of standardization, being poorly
defined, lack of measures of procedural fidelity, and lack of data on any supplemental treatments
participants may also receive. The apparent superiority of EIBI may be an artifact of the lack of
rigorous empirical evaluation of alternative interventions of good quality.
As fully randomised controlled studies in this area were extremely rare, the nonrandom assignment
of participants to treatment groups was also said to severely limit conclusions about treatment
effectiveness. This is because factors affecting group allocation (such as parental chioce) may
have contributed to improved outcomes (such as increased parental motivation and support), thus
confounding any effects attributed to the treatment.
Only one review (Rogers & Vismara, 2008) made specific reference to cultural factors, observing
that treatment programmes for children with autism have been developed for and evaluated
primarily with children from European American backgrounds, and generalization of effects across
ethnic groups from this research is premature. It was suggested that cultural issues may moderate
the effects of autism intervention programs.
Reviews of “fair” quality
Four reviews were rated as being of fair quality, two considering ABA interventions broadly
defined, and the only two reviews considering a specific ABA based intervention, both relating the
the Picture Exchange Communication System. These intervention categories will be considered
separately, as they are in the Evidence Tables.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Applied Behaviour Analysis interventions in general
Two reviews on ABA interventions were classified as being of “fair” quality due to flaws in
methodology and reporting such that results are likely to be susceptible to bias. It should be noted
that some of these studies had supplementary or alternative goals to providing comprehensive
systematic reviews of the literature. Whilst the narrative critiques of the literature in these reviews
may be of interest, conclusions about the evidence base from these studies may not be reliable.
The review by Case-Smith and Arbesman (2008) reviewed ASD interventions relevant to
occupational therapy. The authors concluded that behavioural interventions are successful in
training children in basic academic and life skills, but noted that generalisation of skills to natural
environments was unproven. They also commented on the lack of studies into the promotion of
effective transition to work and independent living. Discussing barriers to widespread application
of behavioural interventions, the reviewers pointed to the time and resources required to
implement a 30- to 40-hour/week intervention as well as the lack of definition of the most
appropriate candidates.
Seida et al (2009) conducted an “umbrella review” of systematic reviews of psychosocial
interventions for people with ASD. Broadly favourable outcomes from behavioural interventions
versus no treatment were reported. Critiquing the literature, the authors noted the absence of data
on the relative effectiveness of treatment options, and the lack of research attention to factors in
choice of treatment including cost, convenience and family burdens.
Picture Exchange Communication System
Two systematic reviews of fair quality specifically considered interventions based on Picture
Exchange Communication System (PECS).
The review by Ostryn et al (2008) considered 15 PECS studies concluded that whilst PECS is
widely implemented it lacked a strong empirical base. Criticisms of the literature included lack of
reporting of statistical significance and precision of results, lack of scope and definition of key
outcomes, and lack of data on maintenance. The authors recommended that PECS was best used
as an initial intervention to teach manding as part of a multimodal system but is not recommended
as a long term intervention.
The other PECS review by Sulzer-Azaroff et al (2009) concluded from 34 studies that
professionals and parents can teach individuals to successfully initiate exchanges of pictures for
tangible and non tangible reinforcers.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
REFERENCES
The references for this report and the appended Evidence Tables are presented below.
Publications included for critical appraisal in this review are indicated by ** prefacing the citation.
Bassett, K., Green, C. J., & Kazanjian, A. (2000). Autism and Lovaas treatment: A systematic
review of effectiveness evidence.
Retrieved 17 June 2008, from
http://www.chspr.ubc.ca/node/351
** Blue Cross and Blue Shield Association. (2009). Special report: early intensive behavioral
intervention based on applied behavior analysis among children with autism spectrum
disorders. Technology Evaluation Centre Assessment Program, 23(9), 1-61.
Broadstock, M., & Lethaby, A. (2008). The Effectiveness of Applied Behavioural Analysis
interventions for people with autism spectrum disorder, Systematic Review. Wellington,
New Zealand: New Zealand Guidelines Group.
Carr, D., & Felce, J. (2007). Brief report: Increase in production of spoken words in some children
with autism after PECS teaching to phase III. Journal of Autism and Developmental
Disorders, 37(4), 780-787.
** Case-Smith, J., & Arbesman, M. (2008). Evidence-based review of interventions for autism used
in or of relevance to occupational therapy. American Journal of Occupational Therapy,
62(4), 416-429.
Chambless, D. L., Baker, M. J., Baucom, D. H., Beutler, L. E., Calhoun, K. S., Crits-Christoph, P.,
et al. (1998). Update on empirically validated therapies, II. The Clinical Psychologist, 51(1),
3-16.
Cohen, H., Amerine-Dickens, M., & Smith, T. (2006). Early intensive behavioral treatment:
Replication of the UCLA model in a community setting. Journal of Developmental and
Behavioral Pediatrics, 27(2 SUPPL. 2), S145-S155.
Drew, A., Baird, G., Baron-Cohen, S., Cox, A., Slonims, V., Wheelwright, S., et al. (2002). A pilot
randomised control trial of a parent training intervention for pre-school children with autism:
Preliminary findings and methodological challenges. European Child and Adolescent
Psychiatry, 11(6), 266-272.
Eikeseth, S. (2009). Outcome of comprehensive psycho-educational interventions for young
children with autism. Research in Developmental Disabilities, 30(1), 158-178.
Eikeseth, S., Smith, T., Jahr, E., & Eldevik, S. (2002). Intensive behavioral treatment at school for
4- to 7-year-old children with autism: A 1-year comparison controlled study. Behavior
Modification, 26(1), 49-68.
Eikeseth, S., Smith, T., Jahr, E., & Eldevik, S. (2007). Outcome for children with autism who began
intensive behavioral treatment between ages 4 and 7: A comparison controlled study.
Behavior Modification, 31(3), 264-278.
Eldevik, S., Eikeseth, S., Jahr, E., & Smith, T. (2006). Effects of low-intensity behavioral treatment
for children with autism and mental retardation. Journal of Autism and Developmental
Disorders, 36(2), 211-224.
** Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2009). Metaanalysis of early intensive behavioral Intervention for children with autism. Journal of
Clinical Child & Adolescent Psychology, 38(3), 439-450.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
14
Horner, R. H., Carr, E. G., Strain, P. S., Todd, A. W., & Reed, H. K. (2002). Problem behavior
interventions for young children with autism: A research synthesis. Journal of Autism and
Developmental Disorders, 32(5), 423-446.
Howard, J. S., Sparkman, C. R., Cohen, H. G., Green, G., & Stanislaw, H. (2005). A comparison of
intensive behavior analytic and eclectic treatments for young children with autism.
Research in Developmental Disabilities, 26(4), 359-383.
Howlin, P., Gordon, R., Pasco, G., Wade, A., & Charman, T. (2007). The effectiveness of Picture
Exchange Communication System (PECS) training for teachers of children with autism: A
pragmatic, group randomised controlled trial. Journal of Child Psychology and Psychiatry,
48(5), 473-481.
** Howlin, P., Magiati, I., & Charman, T. (2009). Systematic review of early intensive behavioral
interventions for children with autism. American Journal on Intellectual & Developmental
Disabilities, 114(1), 23-41.
Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with
autism: A randomized controlled trial to evaluate a caregiver-based intervention program in
community day-care centers. Journal of Developmental & Behavioral Pediatrics, 19(5),
326-334.
Lovaas, O. I. (1987). Behavioral treatment and normal educational and intellectual functioning in
young autistic children. Journal of Consulting and Clinical Psychology, 55(1), 3-9.
Magiati, I., Charman, T., & Howlin, P. (2007). A two-year prospective follow-up study of
community-based early intensive behavioural intervention and specialist nursery provision
for children with autism spectrum disorders. Journal of Child Psychology and Psychiatry
and Allied Disciplines, 48(8), 803-813.
Ministries of Health and Education. (2007). Request for tender (RFT). Review of published
research on applied behaviour analysis (ABA) interventions for people with autism
spectrum disorder (ASD). Wellington, New Zealand.
Ministries of Health and Education. (2008). New Zealand autism spectrum disorder guideline.
Wellington.
Morris, S. B. (2000). Distribution of the standardized mean change effect size for meta-analysis on
repeated measures. Br J Math Stat Psychol, 53 ( Pt 1), 17-29.
Nathan, P. E., & Gorman, J. M. (2002). A guide to treatments that work (2nd ed.). New York, NY:
Oxford University Press.
National Health and Medical Research Council. (2008). NHMRC additional levels of evidence and
grades for recommendations for developers of guidelines. Pilot program 2005-2007.
NHMRC.
** Ospina, M. B., Krebs Seida, J., Clark, B., Karkhaneh, M., Hartling, L., Tjosvold, L., et al. (2008).
Behavioural and developmental interventions for autism spectrum disorder: a clinical
systematic review. PLoS ONE [Electronic Resource], 3(11), e3755.
** Ostryn, C., Wolfe, P. S., & Rusch, F. R. (2008). A review and analysis of the Picture Exchange
Communication System (PECS) for individuals with autism spectrum disorders using a
paradigm of communication competence. Research & Practice for Persons with Severe
Disabilities, 33(1/2), 13-24.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
15
** Reichow, B., & Wolery, M. (2009). Comprehensive synthesis of early intensive behavioral
interventions for young children with autism based on the UCLA young autism project
model. Journal of Autism & Developmental Disorders, 39(1), 23-41.
Remington, B., Hastings, R. P., Kovshoff, H., degli Espinosa, F., Jahr, E., Brown, T., et al. (2007).
Early intensive behavioral intervention: Outcomes for children with autism and their parents
after two years. American Journal on Mental Retardation, 112(6), 418-438.
** Rogers, S. J., & Vismara, L. A. (2008). Evidence-based comprehensive treatments for early
autism. Journal of Clinical Child & Adolescent Psychology, 37(1), 8-38.
Sallows, G. O., & Graupner, T. D. (2005). Intensive behavioral treatment for children with autism:
Four-year outcome and predictors. American Journal on Mental Retardation, 110(6), 417438+497.
** Seida, J. K., Ospina, M. B., Karkhaneh, M., Hartling, L., Smith, V., & Clark, B. (2009).
Systematic reviews of psychosocial interventions for autism: an umbrella review.
Developmental Medicine & Child Neurology, 51(2), 95-104.
** Smith, T., Eikeseth, S., Sallows, G. O., & Graupner, T. D. (2009). Efficacy of applied behavior
analysis in autism. The Journal of Pediatrics, 155(1), 151-152.
Smith, T., Groen, A. D., & Wynn, J. W. (2000). Randomized trial of intensive early intervention for
children with pervasive developmental disorder. American Journal of Mental Retardation,
105(4), 269-285.
Spreckley, M., & Boyd, R. (2009a). Efficacy of applied behavior analysis in autism. Reply. Journal
of Pediatrics, 155(1), 152-153.
** Spreckley, M., & Boyd, R. (2009b). Efficacy of applied behavioral intervention in preschool
children with autism for improving cognitive, language, and adaptive behavior: a systematic
review and meta-analysis. Journal of Pediatrics, 154(3), 338-344.
** Sulzer-Azaroff, B., Hoffman, A. O., Horton, C. B., Bondy, A., & Frost, L. (2009). The Picture
Exchange Communication System (PECS): what do the data say? Focus on Autism &
Other Developmental Disabilities, 24(2), 89-103.
West, S., King, V., & Carey, T. S. (2002). Systems to rate the strength of scientific evidence. (Vol.
Evidence Report/Technology Assessment): Research Triangle Institute-University of North
Carolina Evidence-based Practice Center under Contract No. 290-97-0011. AHRQ
Publication No. 02-E016. .
Yoder, P., & Stone, W. L. (2006). A randomized comparison of the effect of two prelinguistic
communication interventions on the acquisition of spoken communication in preschoolers
with ASD. Journal of Speech, Language and Hearing Research, 49(4), 698-711.
Yoder, P., & Stone, W. L. (2006). Randomized comparison of two communication interventions for
preschoolers with autism spectrum disorders. Journal of Consulting and Clinical
Psychology, 74(3), 426-435.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
16
EVIDENCE TABLES
Evidence tables for included Level I secondary studies assessing the effectiveness of applied behaviour analysis (ABA) for people with
autism spectrum disorder.
1. Applied Behaviour Analysis interventions in general
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Blue Cross
Blue Shield
(2009)
Technology
Evaluation
Centre
Evidence level: I
Inclusion criteria:

reported on the use of
EIBI compared to another
treatment strategy;

attempted to identify
features of EIBI that had
the most impact on its
effectiveness; or

sought to identify children
most likely to benefit from
EIBI.
16 studies met selection criteria, including 2 RCTS, 9 nonrandomized,
comparative studies, and 5 single-arm studies. No studies were found that
included children
with Asperger’s disorder; 4 studies explicitly included children with PDD or
PDD-NOS.
US
SR
Aim:
To conduct a systematic
review of the research
literature on the use of
early intensive
behavioural interventions
based on applied
behavioural analysis
(EIBI) among young
children with ASD. Asked:
(1) How effective is EIBI
in improving the
functioning of children
with autism spectrum
disorders, and how does it
compare to other early
intervention approaches?
(2) Can patient
characteristics be
identified that predict
better outcomes from
EIBI?
(3) Does the effect of EIBI
vary with the intensity of
treatment?
Search period:
From 1966 (varied, see
below) – July 2008.
A more narrowly defined
update was performed in
January 2009 using
Medline but identified no
additional eligible studies.
Exclusion criteria:

sample size <10,
including single-subject
studies;

interventions were very
poorly described;

interventions were not
comprehensive, where a
number of domains
affected by ASD were
addressed;

intervention within a
treatment group was
heterogeneous,
combining a variety of
methods;

experimental intervention
was not intensive (< 20
hrs/wk);

study did not directly
measure outcomes
through a direct
assessment of the child’s
achievement;



The strongest evidence was provided by two randomized, controlled trials
(Smith et al. 2000; Sallows and Graupner 2005) and they compared
different interventions, had small sample sizes, and came to different
conclusions.
3 of 4 studies examining the impact of pretreatment cognitive functioning
found that it significantly predicted outcomes, while one (a randomized,
controlled trial) did not. Some studies suggested that younger age at the
start of therapy is a predictor of better outcomes (e.g., Howard et al.
2005), while others found no difference based on initial age (e.g., Magiati
et al. 2007).
The findings on whether more intense treatment leads to better outcomes
were inconsistent, and no conclusions were drawn.
Authors’ conclusions:

Unfortunately, only two randomized, controlled trials have been
conducted. Weaknesses in research design, differences in the treatments
and outcomes compared, and inconsistent results mean that the impact of
EIBI versus other treatments on outcomes for children with autism cannot
be determined.

The body of evidence overall is too weak to reach conclusions regarding
any of the domains/outcome areas, and too variable to assess whether
greater intensity of EIBI can lead to better outcomes.

The heterogeneity of the interventions used in studies on EIBI and the
significant methodological weaknesses preclude the use of meta-analysis
and of reaching any general conclusions about the effectiveness of EIBI
among children with ASD.

About half of the studies followed children for approximately 2 years or
less, and some for only 1 year. This is not sufficient follow-up time to
assess the potential impact of an intervention over a lifetime.

Research is needed to identify those characteristics of treatment - content,
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score







Clearly defined
methodology though
only two databases
considered
Study quality coded and
rated, apparently by
single reviewer.
Justification given for
excluding SCED studies
including lack of
generalisability across
individuals, inability to
use blinding, variability
of the ASD condition
over time, and potential
for carryover effect
between intervention
and comparator (given
that interventions are
intended to have long
lasting effects)
Comprehensive Tables
including details of
study characteristics,
outcomes, results, and
aspects of quality.
Thorough critical
discussion of included
studies and of recent
systematic reviews
Discussion of the
methodological
limitations of studies
Suggestions made for
future research
17
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Note that selection criteria
then excluded studies
published <1987.

Search strategy:
Medline (from 1966)
PsycINFO (from 1970)
(search terms provided).
References of retrieved
papers were cross
checked to identify
additional papers.
study was
published before
1987, when the
seminal article on
EIBI by Lovaas
was released.
Study quality was appraised
and graded (as good, fair or
poor) using detailed criteria
and formal grading systems.
Results and authors’ conclusions

technique, intensity, starting and ending age, etc. - that maximize its
effectiveness. (Need) a greater emphasis on randomized, controlled trials;
substantially larger sample sizes; uniformity of outcomes and instruments;
and consistent treatments that do not vary widely within the experimental
or control group.
The cost of continuing the current course of assuming that EIBI works may
not be obvious. EIBI is costly financially for society and requires a large
time commitment from children, their families, and their teachers or
therapists. However, these programs may not appear to pose any harm
for the children themselves. Nevertheless, the opportunity costs could be
high, indeed, of providing suboptimal care to these children, simply
because we as a society do not know what works best. The children may
be treated with an intervention that is not as effective as the alternatives.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score
Overall quality: Very good
18
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Case-Smith
& Arbesman
(2008)
Evidence level: I
Inclusion criteria:

Provided evidence
for an intervention
approach used with
children or
adolescents with
ASD;

Peer reviewed;

Addressed a
performance area
or intervention
approach within the
domain of
occupational
therapy;

RCTs, non
randomized clinical
trials, and beforeand-after, one
group designs.
49 studies met selection criteria. Six categories of research topics were
identified, including intensive behavioral intervention, which is reported on here.




USA
SR
Aim:
To identify, evaluate, and
synthesize the research
literature on interventions
for ASD of relevance to
occupational therapy.
Search period:
1987 –2007
Search strategy:
CINAHL
Cochrane SRs database
ERIC
Medline
PsychINFO
Social Science Abstracts
Sociological Abstracts
RehabData
Latin American and
Caribbean Health
Sciences Literature
EBSCOHost
(search terms not
provided).
Exclusion criteria:
 Case series, single
subject design, case
reports and expert
opinion, narrative reviews,
consensus statements;
 Used qualitative methods
to the exclusion of
quantitative methods;
 Had serious design
limitations.

EIBI using one-on-one discrete trial training is widely applied to children with
ASD and has evidence of moderate to strong effects (Lovaas, 1987; Sallows
& Graupner, 2005; Smith, Groen, & Wynn, 2000).
Positive behavioral support has moderate to strong positive effects in
reducing problem behaviors in children with ASD. A systematic approach
that includes prevention of problem behaviors by applying consistent
instruction and consequences to behavior, modifying the environment to
promote appropriate behaviors, and collecting data to monitor children’s
progress appears to be highly effective (Horner et al., 2002).
Authors’ conclusions:

Although behavioral interventions are successful in training children in
basic academic and life skills concepts, it is not known how well these skills
transfer into the natural environment, and the findings are mixed regarding
changes in behavior (Smith et al., 2000). Barriers to widespread application
are as follows: (1) time and resources required to implement a 30- to 40hr/wk intervention and (2) definition of the most appropriate candidates
(Bassett, Green, & Kazanjian, 2000).

When children exhibit problem behaviors, functional analysis is essential to
determine the basis for the behavior.

“Level I” studies of interventions to promote adolescents’ and young adults’
success in work and independent living were virtually absent from the
research literature.
Single reviewer extracted
data. Used a hierarchy of
evidence based on study
design to rank studies.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER

Search strategy
restricted to databases,
no search terms given.
No Tables
Narrative discussion of
results with minimal
methodological critique
and minimal detail of
quantitative results.
Framed selection and
discussion of studies in
terms of relevance to
practice of occupational
therapists.
Overall quality: Fair
19
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Howlin et al.
(2009)
Evidence level: I
Inclusion criteria:

Case-control comparison
study of EIBI for children
with autism;

UCLA affiliated
(home or clinic
based) or UCLAmodeled (homebased) EIBI
program;

At least 10 people
in EIBI group;

age < 6 years at
start of treatment;

intervention at least
12 hrs/wk;

intervention for 12
mths or longer

adequate data on
IQ or standard
outcome measures.
11 studies (reported in 13 reports) met selection criteria, including one RCT
(Smith et al, 2000).
UK
SR
Aim:
To examine the findings
from controlled early
intensive behavioural
intervention (EIBI) studies
published in peer
reviewed journals.
Search period:
1985 – May 2007
Search strategy:
CINAHL
Cochrane library
ERIC
Medline
PsychINFO
Embase
(search terms provided).
Searched websites of
several major health,
autism and research
organisations. References
of review papers were
cross checked to identify
additional papers.
Exclusion criteria:
 SCED or case series
study without comparison
group, studies focused on
a specific behaviour, use
of pharmacological or non
ABA interventions,
outcomes for/views of
therapists or parents.
Summaries/analyses based on
comparisons of the published
group means.
Intervention intensity
 The mean intervention period for the 9 studies which reported intensity
clearly was 27.4 months (SD=10.7, range 14-48 mths).
 Intensity (number of hours of intervention per week) varied greatly by
individual within study. Estimated that EIBI children received significantly
more hours of intervention per week than controls (EIBI Mean=29.8 hrs/wk
cf Control Mean=19.1 hrs/wk, p=.007).
Impact of intervention on IQ
IQ did not vary at baseline between treatment groups, but at final follow up
was significantly higher for EIBI group compared to the Control group in 9 of
11 studies, with estimated effect sizes moderate to large in the majority of
studies, but not all studies.
Variability in outcome
There was considerable variation between individuals, with a minority
achieving marked improvement and even educational independence, whilst
the majority had less dramatic improvements. Some failed to make progress
at all.
Predictors of outcome
 Length of intervention did not appear to be related to outcome, and most
impact on IQ occurred in the first year of the intervention after which the
effect tended to plateau.
 Initial (baseline) IQ strongly predicted improvement in 4 studies, but was
unassociated in another study that investigated this.
 Chronological age did not predict outcome (though age range tended to
be limited to 3-5.5 yrs) which does not support suggestions that EIBI must
start at 3 years.
 Mixed results were found for initial language level which was associated
with outcome in 4 of 7 studies investigating this.
Methodological limitations
 Comparisons were made difficult by varying time lag of follow-up after
ending of intervention, and use of different instruments and scoring
reported (raw, standard, age equivalents).
 Few researchers have assessed outcome in terms of behavioural
difficulties/severity of autism, or impact on family life.
 Few studies reported on quality of alternative therapies in the Control arm,
which tended not to be autism specific.
Authors’ conclusions:
 There is little question that EIBI is highly effective for some children,
however gains are not universal.
 Crucial need to systematically assess for which children is EIBI most and
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score






Clearly defined
methodology and
comprehensive search
strategy
No rating of study
quality
Detailed Tables and
discussion of results
Good attempt to draw
out patterns of results
and explain findings.
Discussion of the
methodological
limitations of included
studies
Suggestions made for
future research
Overall quality: Good
20
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions

least effective. Clearer evidence concerning the optimal duration of
therapy and the age at which is should begin could result in the
development of better targeted, more cost effective programmes.
Need for more RCTs, and need for equivalence trials, where high quality
interventions are compared with each other, rather than comparing high
quality interventions with low quality/low intensity interventions. Failure to
control for time in intervention means that for some studies, EIBI may be
more effective due to differences in intensity, not quality.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score
21
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Eikeseth
(2009)
Evidence level: I
Inclusion criteria:

study was
published in a peerreviewed journal;

children had a
mean age of 6
years or less at
intake;

participants
received
comprehensive
psycho-educational
interventions;

studies contained
outcome data.
25 studies met selection criteria, including 20 which evaluated interventions
based on Applied Behavior Analysis (ABA) treatment, 3 studies evaluated
Project TEACCH, and 2 studies evaluated the Colorado Health Sciences
Model. Only those relating to ABA are discussed here.
Norway
SR
Aim:
Evaluates comprehensive
psycho-educational
research on early
intervention for children
with autism.
Search period:
Not reported
Search strategy:
ERIC
Medline
Psyclit
(search terms not
provided).
References of “recent
papers” were cross
checked to identify
additional papers.
Researchers known in the
field were contacted by
email and asked for
relevant articles recently
published or “in press”.
Comprehensive psychoeducational interventions were
defined as interventions
addressing social behaviors,
communication and
ritualistic/stereotyped
behaviors.
Exclusion criteria:
None reported.
Outcome studies were graded
according to their (i) scientific
value and (ii) according to the
magnitude of results
documented in the studies.
Scientific Merit was evaluated
based on: (a) diagnosis, (b)
study design, (c) dependent
variables and (d) treatment
fidelity. Four levels were used.
Comments and quality
score


Scientific merit

1 ABA study, an RCT, received Level 1 scientific merit (Smith, Groen, &
Wynn, 2000)

4 ABA studies received Level 2 rating (Cohen, Amerine-Dickens, &
Smith, 2006; Eikeseth, Smith, Jahr, & Eledevik, 2002, 2007; Howard,
Sparkman, Cohen, Green, Stanislaw, 2005; Remington et al., 2007)

9 received Level 3 evidence support

6 ABA outcome studies were classified as having insufficient (level 4)
scientific value.
Magnitude of treatment effects

4 ABA studies received Level 1 rating demonstrating that children
receiving ABA made significantly more gains than control group children
on standardized measures of IQ, language and adaptive functioning
(Cohen et al., 2006; Eikeseth et al., 2002, 2007; Howard et al., 2005;
Sallows & Graupner, 2005). Several studies also included data on
maladaptive behavior, personality, school performance and changes in
diagnosis.

3 ABA studies received Level 2 rating (Eldevik et al., 2006; Lovaas, 1987;
Smith, Groen, & Wynn, 2000), demonstrating that ABA treated children
made significantly more gains than the comparison group on one
standardized measures of IQ or Adaptive Functioning.

5 ABA studies received Level 3 rating.
Authors’ conclusions:
Recommended practice parameter

ABA treatment is demonstrated effective in enhancing global functioning
in pre-school children with autism when treatment is intensive and carried
out by trained therapists.

ABA treatment is demonstrated effective in enhancing global functioning
in children with PDD-NOS (one Level 1 study; Smith, Groen, & Wynn,
2000).
Guideline practice parameter

ABA can be effective for children who are up to 7 years-of-age at intake
(one Level 2 study; Eikeseth et al., 2002, 2007).
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER




Comprehensive search
strategy, but no search
terms or dates given to
allow replication
Very detailed Tables,
describing studies,
interventions, and
discussion of results
Lacks detail on
participant
characteristics at
baseline.
Discussion of the
methodological
limitations of included
studies
Suggestions made for
future research
directions
Insufficient quantitative
information on the
extent of the gains
experienced, the clinical
meaningfulness of these
gains and whether there
were subgroups of
children who did not
improve.
Overall quality: Good
22
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Eldevik et al
(2009)
Evidence level: I
Inclusion criteria:

intervention generally adhered to
defined common EIBI elements;

aged between 2-7 years at intake;

diagnosed with autism, PDD NOS;

full-scale measure of intelligence
and/or standardized measure of
adaptive behaviour at intake and
post intervention;

intervention lasted 12-36 mths;

included Control (no intervention,
considerably less intensive one, or
poorly described intervention) or
Comparison (non EIBI intervention
of similar duration and intensity)
group.
9 controlled designs met selection criteria, including one RCT
(Smith et al, 2000). Howard et al (2005) included both a
Control and Comparison group and so 10 effect sizes were
calculated. Four studies had a Comparison group, and six had
a Control group.

Wales &
Norway
SR/MA
Aim:
To conduct a systematic
literature search for
studies reporting effects
of Early Intensive
Behavioral Intervention
(EIBI). Also aimed to
replicate and extend (with
methodological
improvements) the MA of
Reichow and Wolery
(2009).
Search period:
- March 2008
Search strategy:
ERIC
PsycINFO
PubMed
(search terms provided).
References of retrieved
papers were cross
checked to identify
additional papers.
Exclusion criteria:

Primarily using nonverbal intelligence
measure;

Case study or case series;

Data from people in another included
study.
Inter-rater reliability was high for selection and
coding of articles. Approached authors for raw
individual data which was re-analysed.
Completed a meta-analysis yielding a
standardized mean difference (SMD) effect
size for two available outcome measures:
change in full-scale intelligence and/or
adaptive behavior composite. Effect sizes
were computed using Hedges’s g.
The average effect size was 1.10 for change in full-scale
intelligence (95% CI .87, 1.34) and .66 (95% CI=.41, .90) for
change in adaptive behavior composite. These effect sizes are
generally considered to be large and moderate, respectively.
As the two sets of comparative data from Howard et al were
compared to the same intervention group and not independent,
an analysis excluding the control group from Howard et al
(2005) was also conducted, with effect sizes not significantly
altered.
Homegeneity was tested and the Q statistic was not significant,
indicating that studies could be combined, and a fixed effect
model was used. No statistical or visual evidence of publication
bias was observed.





Authors’ conclusions:

EIBI produces large to moderate effect sizes for changes
in IQ and ABC scores for children with ASD when
compared with no intervention controls and eclectic
provision. The results support the clinical implication that
at present, and in the absence of other interventions with
established efficacy, Early Intensive Behavioral
Intervention should be an intervention of choice for
children with autism.

The difference in outcome between EIBI and the
comparison intervention may be due to differences in the
amount and frequency of supervision and training. There
was insufficient data to control for this in the present
study, however the EIBI group in general received more
frequent and more total hours of supervision and training.
This remains a threat for the validity of conclusions about
the superiority of EIBI in relation to comparison
intervention.

Randomized controlled trials comparing EIBI to other
interventions are still needed. In particular, studies are
needed where the comparison intervention is of similar
intensity and where staff receive similar training and
supervision.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER



Moderately broad
search strategy
Excluded Sallows and
Graupner (2005) as it
compared two EIBI
groups of somewhat
different intensity
varying in programme
delivery (clinic- or
parent-directed)
Used more inclusive
definition of EIBI than
Reichow and Wolery
(2009)
Cited papers of those
retrieved which were
excluded and noted why
Detailed account of
outcome measures
considered
Approached authors for
individual data
Fully detailed Tables,
describing design,
allocation to group
method, interventions,
outcome scores
Moderator variables
were not considered.
One author is from
Lovaas institute.
Overall quality: Very good
23
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and exclusion criteria
Ospina et al.
(2009)
Evidence level: I
Inclusion criteria:

RCTs, non randomized controlled
trials, and prospective and
retrospective cohort studies with a
control group;

published in English;

reporting on the efficacy of any
behavioural or developmental
intervention;

for individuals with ASD (including
those with dual diagnoses);

reporting on outcomes relating to
the core features of ASD (though
additional outcomes may also be
reported).
Canada
SR/MA
Aim:
To identify, appraise and
synthesise the evidence
on the effectiveness of
behavioural and
developmental
interventions for
improving core symptoms
of ASD.
Search period:
– May 2007
Search strategy:
22 electronic databases,
including:
BioSYS Previews
CINAHL
Cochrane central register
of controlled trials
Embase
ERIC
Medline
ProQuest Dissertations
and theses
PsychINFO
Web of Science
(search terms provided).
Hand searching journals,
cross-checking of
reference lists, databases
of theses and
dissertations, and
contacting experts in the
field were used to identify
additional papers.
Exclusion criteria:

Individuals with Rett’s Disorder, or
Childhood Disintegrative Disorder.
Two independent reviewers made the
final study selection, extracted data
using pre-tested checklists, and reached
consensus on study quality. Metaanalyses of the study results were
conducted when two or more trials
assessed the same intervention, used
similar comparison groups, and had data
for common outcomes of interest. If the
same measure was reported, reported
weighted mean differences (WMD);
otherwise, reported standardised mean
differences (SMD). SMD of 0.2 indicated
a small effect, 0,5 a medium, and 0.8 a
large effect.
Results and authors’ conclusions
101 studies met selection criteria, and evaluated 8 broad types of
intervention defined on a continuum of behavioural to
developmental interventions, the most relevant reported here being
the ABA category, which included discrete trial training (DTT) and
UCLA/Lovaas therapy as sub-categories. There were 31 ABA
studies, including 12 trials and 9 cohort/observational studies.
Descriptive analyses

The reviewers judged studies to be at predominantly high risk
of bias and reported inconsistent results across various
interventions.

Discrete Trial Training effects were inconsistent, but motor and
functional outcomes were often positive compared to speechrelated outcomes which were generally negative. For Lovaas, 7
of 8 non-RCTs reported positive findings, whereas 3 of 4 RCTs
reported no significant findings. The reviewers note that as
non-RCT evidence can be more prone to bias and overestimate
treatment effects, this observation has serious implications for
interpreting the results from non-RCTs.
Heterogeneity was moderate for some outcomes (for overall
intellectual functioning, non verbal intellectual functioning,
socialization).
Meta Analyses

Meta-analyses of 3 controlled clinical trials showed that Lovaas
treatment was superior to special education on measures of
adaptive behaviour (WMD = 11.8; 95%CI, 6.94 to 16.67),
communication and interaction (WMD = 16.63; 95% CI, 11.25
to 22.01), comprehensive language (WMD = 12.84; 95% CI,
6.38 to 19.30), daily living skills (WMD = 5.61; 95% CI, 0.54 to
10.67), expressive language (WMD = 15.05; 95% CI, 6.19 to
23.90), overall intellectual functioning (SMD = 0.95; 95% CI,
0.44 to 1.46), and socialization (WMD = 9.17; 95% CI, 2.16 to
16.19).

High-intensity Lovaas was superior to low-intensity Lovaas on
measures of intellectual functioning in two retrospective cohort
studies (SMD = 0.92; 95% CI, 0.61 to 1.24).

No statistically significant differences were found for: Lovaas
versus special education for non-verbal intellectual functioning
in a meta analysis of three controlled clinical trials (SMD = 7.83;
95% CI, 22.86 to 18.52); or for Lovaas versus Developmental
Individual-difference Relationship-based intervention for
communication skills based on two RCTs (SMD = 0.73; 95%
CI, 20.26 to 1.72).
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score






Clearly defined
methodology and
extensive and
comprehensive search
strategy
Review reports
extensive details of
individual studies
including sample
characteristics, as well
as assessments for
each study across
several quality domains.
Careful synthesis of
results.
Discussion of the
methodological
limitations of review
approach and studies.
In particular, notes risk
of “expectancy bias” for
studies with non active
controls (eg, wait list)
where participants (and
their families) in control
group would not expect
improvement.
Moderate heterogeneity
evident for some
outcomes.
Suggestions made for
future research
Overall quality: Very good
24
Reference,
country,
design
Evidence level, aim and
search method
Results and authors’ conclusions
Inclusion and
exclusion criteria
Authors' conclusions:

There is no clear answer regarding the most effective therapy to improve
symptoms associated with ASD.

The evidence seems to provide some support for discrete trial training in
terms of motor and functional skills but not for communication skills.

Lovaas' therapy showed benefits when compared to ''no treatment'' and
evidence from meta-analysis of retrospective cohort studies showed greater
effects for High versus Low intensity Lovaas. Whilst Lovaas may improve
some core symptoms of ASD compared to special education, these findings
are based on pooling of a few, methodologically weak studies with few
participants and relatively short-term follow-up. Replication in RCTs is
needed to substantiate the use of Lovaas and to assess the effect of
treatment intensity on outcomes of children with ASD.

There is considerable potential for heterogeneity in the population,
intervention, comparator and outcomes of interest, as ASD is a spectrum
disorder, therapy is not always reported in detail, comparators are difficult to
control for, and outcomes are somewhat subjective.

As no definitive behavioural or developmental intervention improves all
symptoms for all individuals with ASD, it is recommended that clinical
management be guided by individual needs and availability of resources.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score
25
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Reichow and
Wolery
(2009)
Evidence level: I
Inclusion criteria:

evaluated EIBI based on
the UCLA/Lovaas (1987)
model;

participants had
diagnoses of autistic
disorder, ASD, PDD, or
PDD-NOS;

participant samples
receiving EIBI had a
mean chronological age <
84 months at the
beginning of treatment;

mean duration of EIBI
was 12 months;

at least one child outcome
measure was reported;

experimental (e.g., pre/post-test multiple-group
design) or quasiexperimental designs (i.e.,
nonequivalent control
group, one-group pre/post-test design);

publication in English in a
peer-reviewed journal.
sample adjustment.
14 studies met selection criteria, although due to patient overlap, only 13
publications were appraised, reporting on 14 distinct samples. The studies
included 10 studies allowing between group analyses, including two RCTs
(Smith et al, 2000; Sallows and Grupner, 2005).

USA
SR/MA
Aim:
To provide a 3-part
comprehensive synthesis
of the early intensive
behavioral intervention
(EIBI) for young children
with autism based on the
Lovaas University of
California at Los Angeles
Young Autism Project
method (Lovaas, 1987).
The synthesis included:
(a) descriptive analyses,
(b) effect size analyses,
and (c) a meta-analysis.
Search period:
Not reported
Search strategy:
An “electronic database
search”, databases not
identified. (Search terms
not provided).
References from review
articles and eligible
reports were cross
checked to identify
additional papers.
Descriptive analyses

EIBI is an effective intervention for children with autism. On average,
children present fewer or less severe autism symptoms after intervention.
Effect size differences between treatment groups

The between group (standardized mean difference) effect sizes suggest
children receiving EIBI made more gains than children receiving minimal
behavioral intervention, eclectic treatment, or treatment as usual.

Children receiving EIBI made large gains on multiple domains of
behavior, and made better progress than children with autism who
receive less intense behavioral intervention or other treatments.
Meta analysis for change in IQ within treatment group

A meta-analysis was conducted on 12 samples using standardized mean
change effect size for IQ within the intervention group (because
comparison groups varied across studies). Using a random effects model,
the mean effect size was 0.69. This is statistically significant (p < 0.001)
and represents a large effect. This suggests EIBI is, on average, an
effective intervention for increasing IQ scores for children with autism,
though the effect is likely to be inflated by publication bias.

Tests of homogeneity suggest that there was significant variability
between studies.

Moderating variables identified a priori suggest the greatest results on IQ
change might be seen when supervisory staff were trained using the
UCLA model (p<0.01).
Hand search of selected
journals.
Contact with experts.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER







Limited detail provided
on search strategy
restricting ability for
replication.
Detailed Tables, figures
Precision details
(confidence intervals)
given in Figure only
Discussion of the
methodological
limitations of included
studies and of approach
of reviewers.
Studies used a range of
measures of IQ
Included quasi
experimental studies
without an equivalent
control group in the MA.
Heterogeneity present,
which raises questions
about whether
combining results
through a meta analysis
was appropriate.
Suggestions made for
future research
directions
Overall quality: Good
26
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Study characteristics and
effect size data were coded
using a manual. Two
independent coders for 4 of
the studies had mean
interobserver agreement of
92%, range 85-93%. Studies
rated on experimental rigour.
Meta analysis conducted for
studies reporting within-group
comparisons using
standardized mean change
effect size with small sample
adjustment.
Authors’ conclusions:

While these findings were strong, the nonrandom assignment of
participants to groups limit conclusions about the superiority of EIBI to
other treatments. Further, the comparison groups often lacked
standardization within the group, were poorly defined, had no measures
of procedural fidelity, and had no data on whether participants received
supplemental treatments.

The findings suggest EIBI can be an effective treatment, on average, for
some children with autism. However the intervention has not worked for
all children, and individual data typically were not presented. It is unclear
if individuals making change in one domain (e.g., IQ) also made gains in
another (e.g., adaptive behavior).

These results should be taken with caution because of the small number
of studies, and because the standardized mean change effect size is
calculated without reference to a comparison or control group. Thus, the
threats to internal validity of history, maturation, lack of procedural fidelity,
and instrumentation threats cannot be eliminated. Thus, while the effect
sizes were often large, they cannot be attributed to EIBI exclusively.
Effect sizes tend to be inflated when using standardised mean change
effect sizes (Morris, 2000). Without comparisons between EIBI and
empirically validated treatment programs, it is not possible to determine if
EIBI is more or less effective than other treatment options.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
Comments and quality
score
27
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Rogers and
Vismara
(2008)
Evidence level: I
Inclusion criteria:

study involved
comprehensive
(addressing core
deficits in autism)
treatment
approaches for
children with
autism;

predominantly ages
5 years or younger;

controlled group
designs or singlesubject multiple
baseline designs
using 3 or more
subjects with
published data.
22 studies met selection criteria. Studies of non ABA interventions, including
psychopharmacological interventions, are not discussed here.

5 RCTs, included 4 relating to ABA were identified: Jocelyn et al (1998),
Drew et al (2002), Smith et al (2000), and Sallows and Graupner (2005).

Non-randomized controlled group design studies included 2 of Lovaasbased interventions: Eikeseth et al (2002), Cohen et al (2006); and one of
an EIBI approach using a mixture of didactic and naturalistic behavioral
teaching approaches by Howard et al (2005).

US
SR
Aim:
What is the empirical
evidence supporting
efficacy of early
intervention or young
children with autism.
Search period:
1998 - 2006
Search strategy:
PsycINFO
(search terms provided).
References of all
reviewed articles were
cross checked to identify
additional papers.
Exclusion criteria:

studies that did not report
as outcomes analyses of
child progress using
general measures of
children’s language or
intellectual development;

studies targeting only one
domain (eg; unwanted
behaviours);

case reports;

studies whose data were
published only in book
chapters;

studies included in review
by Rogers (1998).
Studies were graded
according to methodological
criteria based on Chambless
et al (1998) as being (i) “well
established” or (ii) probably
efficacious. Studies also
graded using Nathan and
Gorman (2002) criteria as
Type 1 through to Type 6,
ranging from most to least
rigorous.




Results suggest that young children with autism, as a group, demonstrate
accelerated developmental gains in response to focused daily
interventions of several different kinds. Significant increases in language
and communication abilities in the treated group occurred in most studies
and interventions with many targeted hours per week resulted in
increases in IQ at the group level as well.
The best-designed, controlled studies evaluating Lovaas’s treatment met
criteria for a ‘‘well-established’’ psychosocial intervention for improving
the intellectual performance of young children with autism spectrum
disorders, based on the significant increase in IQ reported in these four
studies compared to control groups. Although some of these studies also
reported significant improvements in behavior, adaptive skills, and
language skills, reviewers argue that there is less consistency in the data
in these areas.
No treatment met the ‘‘probably efficacious’’ criteria, although 3
treatments met criteria for being ‘‘possibly efficacious’’.
Most studies were either Type 2 or 3 in terms of their methodological rigor
based on Nathan and Gorman’s (2002) criteria.





Limited search
strategy, 1 database
and cross-checking
Detailed Tables,
describing studies,
interventions, and
discussion of results
Discussion of
methodological
limitations of included
studies and findings.
Detailed suggestions
made for future
research directions,
especially around
mediators and
moderators of effect
Practice
recommendations are
offered for
psychologists.
Discussed the value of
both RCTs and SCED
studies in the pathway
of intervention
development.
Overall quality: Good
Authors’ conclusions:

The evidence suggests that early intervention programs are beneficial for
children with autism, often improving developmental functioning and
decreasing maladaptive behaviors and symptom severity at the level of
group analysis.

Lack of comparative studies prevents determining which comprehensive
treatment approach is best for young children with autism. Other wellknown interventions may be as or more efficacious as Lovaas’s model but
they have not been rigorously evaluated.

Because the majority of studies did not include ethnically diverse
participant groups, generalization of effects across groups is premature.

Given the few randomized controlled treatment trials that have been
carried out, the few models that have been tested, and the large
differences in interventions that are being published, it is clear that the
field is still very early in the process of determining (a) what kinds of
interventions are most efficacious in early autism, (b) what variables
moderate and mediate treatment gains and improved outcomes following
intervention, and (c) the degree of both short-term and long-term
improvements that can reasonably be expected for a child with autism.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
28
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Seida et al
(2008)
Evidence level: I
Inclusion criteria:

Systematic reviews
(defined as having
defined a search
strategy, and
attempted to
synthesise data of
primary studies
quantitatively or
qualitatively);

Data reported for
participants with
autistic disorder,
Asperger
syndrome, atypical
autism, high
functioning autism,
PDD-NOS, and/or
suspected autism;

Have covered a
psychosocial
intervention aimed
at improving the
functioning of
individuals in any of
the impairments
characteristic of
ASD.
30 studies met selection criteria representing five intervention domains,
including one of interventions based on behavioural theory (n=9) reported on
here.


Canada
SR (of SRs)
Aim:
To present an “umbrella
review” of the clinical
findings of systematic
reviews of psychosocial
interventions for ASDs.
Search period:
– May 2007
Search strategy:
25 electronic databases
including Medline
(listed with search terms
in an online supplement).
References of articles and
of “personal collections”
were cross checked to
identify additional papers.
Exclusion criteria:

editorials,
correspondence,
abstracts and review
summaries

All 9 SRs on interventions based on behavioural theory were rated as having
low methodological quality.


The reviews with meta-analyses showed favourable outcomes for the
behavioural intervention. Reductions in problem behaviour were found in
3 reviews and an increase in adaptive, cognitive, and language skills was
observed in 1 review.
In the reviews without meta-analysis, positive findings were reported for
intelligence, developmental gains, functional skills, and communication
outcomes. The authors of 3 reviews concluded that there is uncertainty
about whether behavioural interventions produce ‘normal functioning’ and
improvement on various intelligence and developmental measures. The
remaining 2 reviews provided no information on efficacy outcomes;
instead trends in the use of behavioural treatments over time were
described.
Authors’ conclusions:

The reviews reported positive outcomes for many of the interventions,
suggesting that some form of treatment is favourable over no treatment.

Little evidence for the relative effectiveness of these treatment options.

Many of the systematic reviews had methodological weaknesses that
make them vulnerable to bias.

Even if differences in the therapeutic effectiveness of the interventions
exist, differences in cost, convenience and family burdens associated with
the interventions are likely to be important factors in individual decisionmaking.

Future studies and reviews that break down the characteristics of the
individuals with autism and the components of the programmes are
needed in order to provide more meaningful and stronger conclusions.
Abstracts scanned by single
reviewer but eligibility criteria
applied on full text
independently by two
reviewers. Studies were
graded for methodological
quality by two reviewers using
the Overview Quality
Assessment Questionnaire.
Data extracted by single
reviewer, and verified by
another.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER


Broad search strategy
Use of two reviewers
and quality checklist
Only secondary studies
considered
Only sparsely detailed
Tables, listing
intervention domains
and the outcome areas
where positive,
negative or unclear
results were found. No
data provided to allow
estimation of effect or
precision of findings.
Suggestions made for
future research,
including on reporting
and conduct of SRs
Overall quality: Fair
29
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Spreckley &
Boyd (2009)
Evidence level: I
Inclusion criteria:

Systematic reviews,
randomized or quasirandomized controlled
trials (RCT);

Preschool children with
diagnosis of ASD or
PDD;

Interventions focused on
ABI approaches to
behavioural
management;

Interventions delivered to
parents/carers and/or
directly to the child by
special educators,
teachers, speech
pathologists,
psychologists, or allied
health professional
students;

Interventions occurred
between 18 mths and 6
years;

Cognitive, language,
and/or adaptive
behaviour outcomes
measured.
13 studies met selection criteria, 6 were trials with adequate methodologic
quality (PEDro6), and 4 of these had adequate data to be included in the
meta analysis (Smith et al, 2000; Sallows and Graupner, 2005; Eikeseth et al
2002; and Eikeseth et al, 2007).

Australia
SR/MA
Aim:
To review the
effectiveness of applied
behavior intervention
(ABI) programs for
preschool children with
autism spectrum disorder
(ASD) in their cognitive,
adaptive behavior, and
language development.
Search period:
From 1982 but varied
(see below) - Nov 2007
Search strategy:
Cochrane Library
Medline (from 1996)
PsycINFO (from 1985)
CINAHL (from 1982)
AMED (from 1985)
(search terms provided).
Meta-analysis of 4 studies concluded that, compared with standard care, ABI
programs did not significantly improve outcomes of children in the
experimental group compared with those who received standard care:

For cognitive outcomes, a standardized mean difference (SMD) of 0.38
(95%CI �-0.09 to 0.84; P=.1)

For expressive language; SMD of 0.37 (95%CI - 0.09 to 0.84; P=.11)

For receptive language; SMD of 0.29 (95%CI �- 0.17 to 0.74; P=.22)

For adaptive behavior; SMD of 0.30 (95%CI �-0.16 to 0.77; P=.20).
Heterogeneity (measured by I2) was low for cognitive and receptive language
outcomes, and moderate for expressive language and adaptive behaviour
outcomes.





Authors’ conclusions:

Currently there is inadequate evidence that ABI has better outcomes than
standard care for children with autism. Appropriately powered clinical
trials with broader outcomes are required.

Limitations of the meta-analysis and evidence base include: high
variability in the studies included (in ages of children involved, programme
content), difficulty in establishing control groups all received some form of
intervention), poor homogeneity, limited information on retention in the
interventons groups, and lack of strict selection criteria for participants.

What is too often forgotten is that the overwhelming majority of children
with ASD change over time as part of their development.
Exclusion criteria:

Studies with PEDro score
<6;

RCTs with no useful data
for meta-analysis; and/or

those that did not include
discrete trial training
(DTT) as part of their
intervention.
2 reviewers independently
applied the Physiotherapy
Evidence Database (PEDro)
Scale of quality assessment to
critically appraise the studies.
Data synthesis used RevMan,
reported effect sizes and
standardised mean differences
(SMD).
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER

Broad database search
strategy, but no other
search technique
Applied minimum
quality criteria for
inclusion in the MA
Tables described
design, interventions,
comparators, and
intensity.
Estimates of precision
provided.
Too few studies
included to consider
sub-group or sensitivity
analyses or to consider
moderator variables.
Follow-up
correspondence about
this article from Smith
et al (2009) suggested
that the Sallows and
Graupner (2005) study
be excluded from the
MA as its comparison
group was a lower
intensity form of ABA.
Spreckley & Boyd’s
(2009b) reply argued
that they were
evaluating standard
(i.e., high intensity) ABA
with any standard or
eclectic comparator.
Limitations and
heterogeneity of the
evidence base raises
questions about
whether a meta
analysis was
appropriate.
Overall quality: Good
30
2. PECS
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
Results and authors’ conclusions
Comments and quality
score
Ostryn et al.
(2009)
Evidence level: I
15 studies met selection criteria. Most studies were single subject within
subjects or multiple baseline designs, with one RCT (Yoder and Stone, 2006)
and one non-randomised, between-group experimental study (Carr and Felce,
2007).


Aim:
To examine research
involving the use of PECS
with individuals with ASD
to identify dependent
variables and outcomes
reported in the research,
and to apply the
communication
competence paradigm to
the PECS research.
Inclusion criteria:

English language;

Used experimental
or quasiexperimental
design;

Included individuals
aged under 18
years old with
ASDs;

evaluated PECS;

published in a peerreviewed Journal.
Search period:
1985 –2007
Exclusion criteria:
Not reported
USA
SR





Search strategy:
ERIC
PsychINFO
PubMed
(search terms provided).
Also searched PECS
website. References of
retrieved papers were
cross checked.
Narrative synthesis
Noted that whilst “positive outcomes” were reported in all but four
studies, though these were not defined in terms of statistical significance.
Noted that acquisition rates (through phases of PECS 1-6) were reported
as being “fast” by several studies, though “fast” was not defined.
Commented that outcomes such as spontaneous communication were
only defined in two studies in relation to visual prompts.
Outcomes associated with joint attention were measured in only three
studies.
Notes that few studies examined maintenance of PECS outcomes.
Where they are reported, results are encouraging.


The authors applied the communication competence paradigm to PECS
outcomes and argue that additional strategies to those of manding or tacting
could enhance PECS. The authors suggest that PECS strategies may be too
limited for some individuals and people with ASD should be encouraged to
use any communication attempts, including speech, gestures, and vocal
approximations.
Authors’ conclusions:
 Results of this study reveal that the PECS is widely implemented with
individuals having ASDs but without a strong empirical base.
 Suggest that PECS is best used as an initial intervention to teach manding
and the basic elements of communicative exchange, but that other
communication systems may be needed to permit communication beyond
this.
 PECS is not recommended as a long term intervention and is best
implemented as part of a multimodal system for when picture
communications are more socially appropriate.
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER



Broad search strategy
Detailed Tables
describing outcome
measures and results,
but lacking data on
participant
characteristics and
interventions.
Narrative discussion of
results framed around
how outcomes
measures in PECS
studies relate to the
communication
competence paradigm.
No formal checklists
used to assess or grade
quality.
Minimal synthesis of
findings, although note
that this was not
specifically the aim of
the review.
Suggestions made for
future research.
Suggestions for how to
use PECS as part of a
multimodal approach
including other
communication systems
Overall quality: Fair
31
Reference,
country,
design
Evidence level, aim and
search method
Inclusion and
exclusion criteria
SulzerAzaroff et al.
(2009)
Evidence level: I
Inclusion criteria:

Articles including
data and protocol
containing the key
features of PECS
(as specified by
Frost and Bondy).
UK
SR
Aim:
To synthesise key
features of the published,
peer-reviewed, databased research reports of
Picture Exchange
Communication System
(PECS) applications, to
answer the question “how
solid is the scientific
evidence supporting
PECS effectiveness?”
Search period:
Not reported
Search strategy:
ERIC
PsycINFO
Science Direct
Google scholar. Minimal
search terms used
including PECS, Bondy,
Frost (who developed
PECS).
References of retrieved
papers were cross
checked to identify
additional papers.
Exclusion criteria:
 Papers written in
languages the authors
couldn’t read (n=3)
 Studies describing
different pictorial
interventions to the
original PECS.
The review summarised the
studies in a narrative way, with
little quantification.
Results and authors’ conclusions
Comments and quality
score
34 studies met selection criteria, including two RCTs (Howlin et al. 2007)
(Yoder and Stone 2006b) (Yoder and Stone 2006a) and one non randomised
experimental study (Carr and Felce 2007). Studies reported on 386 study
participants, the “majority” of whom had received diagnoses of ASD.


The authors presented a number of summarising statements in the Results
section asserting positive research findings, however these were not attended
by quantitative supporting data, analysis or synthesis of study findings. For
example, “researchers reported improvement in communication among the
vast majority of their participants”. Also, “several investigators provided
evidence that learning to use PECS was associated with some of their
participants increasing their speaking and social approaching” and “a number
of investigators cited decreases in disruptive or dangerous behaviours”.
Thus whilst these and similar statements stated that there were examples of
positive findings, the authors did not demonstrate the extent to which such
findings occurred or the precision of this effect, how effects varied between
studies and individuals, or why.





The authors urged PECS researchers to report more methodological details,
and identified questions to be explored in future research.
Authors’ conclusions:
 Analysis of the available research leads to the conclusion that the majority
of participants who lacked functional comunication skills did acquire
extensive functional vocabularies.
 Analysis of the studies strongly supports the conclusion that by adhering
to the PECS protocol, professionals and parents can teach individuals to
successfully initiate exchanges of pictures for tangible and non tangible
reinforcers.
Limited search
strategy, particularly
with respect to search
terms and reporting.
Narrow definition of
PECS.
Lacks detail of outcome
measures and follow-up
periods of inidivudal
studies.
Results and Discussion
focussed on narrative,
imprecise description of
studies.
Detailed Tables
Emphasised the need
for additional reporting
in future research
Note that two of the
authors were the
original developers of
PECS.
Overall quality: Fair
Key:
ABI
CI
EIBI
IQ
PEDro
PECS
SCED
SMD
TEACCH
WMD
Applied behaviour intervention
Confidence interval
Early intensive behavioural intervention
Intelligence quotient
Physiotherapy evidence database
Picture exchange communication system
Single case experimental design
Standardised mean difference
Treatment and education of autistic and related communication-handicapped children
Weighted mean difference
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER
ABA
DTT
Hrs/wk
MA
PDD NOS
RCT
SD
SR
UCLA
Applied behaviour analysis
Discrete trial training
Hours per week
Meta analysis
Pervasive developmental disorder – not otherwise specified
Randomised controlled trial
Standard deviation
Systematic review
University of California, Los Angeles
32
UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER