Doctorate in Educational and Child Psychology
Tracey Atkins
Case Study 1: An Evidence-Based Practice Review Report
Theme: Interventions for children with Special Educational Needs
Are Working Memory Training programmes effective interventions for young
people with ADHD?
Summary
Working Memory Training (WMT) programmes vary in presentation but all are
intended to improve the capacity and efficiency of an individual’s working memory in
order to increase cognitive ability and academic attainment (Melby- Lervag & Hulme,
2013).
WMT programmes usually comprise consistent practice of short but regular
computerised activities that can be carried out in schools or at home for around 30
minutes a day and for up to 10 weeks. For young people with cognitive or
behavioural disabilities, WMT has been seen more as a treatment than a teaching
tool. The range of international research currently available for WMT in a variety of
populations is extensive (Melby-Lervag & Hulme, 2013; Rapport, Orban, Kofler &
Freidman, 2013); Chacko, Feirson, Bedard, Marks, Uderman & Chimiklis, 2013). As
well as hyperactivity and impulsivity, young people with Attention Deficit
Hyperactivity Disorder (ADHD) have been identified as having impairments of
executive function with a particular weakness in working memory (Barkley, 1997);
Makris, Beiderman, Monuteaux & Seidman, 2009; Willcutt, Doyle, Nigg, Faraone &
Pennington, 2005).
This systematic review explores whether WMT is an effective intervention for
young people with ADHD and whether WMT impacts on working memory functions
(as defined by Baddeley, 2000). The WMT programmes selected for this review
aimed to develop verbal, visuo-spatial, short-term memory and digit span skills,
1
Doctorate in Educational and Child Psychology
Tracey Atkins
some of which showed more potential than others. However, critical analysis of
methodological issues, differences in WMT programmes and the heterogeneity of
ADHD itself, recommends that results must be interpreted with caution.
Introduction
Working Memory Training
Three different working memory training programmes were included in the
studies chosen for this review. Each one will be described separately.
Cogmed Working Memory Training (CWMT)
Cogmed Working Memory Training (www.cogmed.com) is a computerised
programme combining software with coaching for children, adolescents and adults.
Sessions can be administered on computers at school or at home and used as a tool
for teaching or treatment to develop attention skills for people with working memory
difficulties, brain injury or learning difficulties. It is intended that participants will be
better able to stay focused, ignore distractions, plan next steps, remember
instructions, and complete tasks.
Cogmed.com suggests that the programme requires qualified trainers to teach
users how to administer the product. However, there was no indication in the studies
selected for this review that any qualification was utilised. Studies that used
‘Cogmed’ were Chacko, Bedard, Marks, Feirsen, Uderman Chmilkis…& Ramon
(2013); Green, Long, Green, Iosif, Dixon, Miller… & Schweitzer (2012) and Holmes,
Gathercole, Place, Dunning, Hilton & Elliot (2010).
Chacko et al. (2013) administered Cogmed to young people aged between 7
and 11 years old. Sessions lasted around 40 minutes a day for around 25 days.
2
Doctorate in Educational and Child Psychology
Tracey Atkins
Green et al. (2012) administered CWMT to young people with a mean age of 9.9
years old. Here, 90 tasks were completed in a twenty five day period. Holmes et al.
(2010) administered CWMT to young people aged between 8 and 11 years old and
spread 20-25 sessions for over a 6–10 week period.
CWMT includes the temporary storage and manipulation of 10 verbal and
visuo-spatial WM span tasks in both auditory and visual form. The participant
chooses any number as specified by the researchers such as 6 out of the 8 tasks or
changeable tasks after a period of time. The programme is adapted to the
functioning level of the child. If the child answers correctly on consecutive tasks, the
level increases accordingly. Placebo groups engage in the same tasks without the
responding increase in levels. CWMT programmes used by both Chacko et al.
(2013) and Holmes et al. (2010) were conducted at the pupil’s home, whereas, the
programme in Green et al. (2012) was conducted at school. CWMT was used here
for research purposes but is a commercially available for educational settings or
individuals.
Individual computer training programme
Klingberg, Forssberg & Westerberg (2002) used a computerised training
programme with young people aged between 7 and 15 years old. Three working
memory tasks per session were presented to the participants. These included tasks
on a) visuo-spatial working memory visual stimuli that had to be remembered; b)
numbers shown for participant to repeat backwards; and c) letters read aloud then
repeated back according to its place in the sequence. Sessions lasted 25 minutes
and took place on a daily basis for 5–6 weeks. This study was conducted for
research purposes in a children’s hospital. Klingberg et al. (2002) aimed to
3
Doctorate in Educational and Child Psychology
Tracey Atkins
investigate whether working memory capacity could be increased and the
hyperactive motor symptoms of ADHD improved.
Braingame Brian
Braingame Brian is also a computerised programme aiming to address three
aspects of executive function: visuo-spatial working memory; inhibition and ability to
switch attention (www.gamingandtraining.nl/beschrijving-braingame-brian/).
Braingame Brian is used in the study by Van der Oord, Ponsioen, Geurts,
Brink & Prins (2012) for research purposes with young people aged between 8 and
12 years old. Working memory tasks within this programme included activities to
target: a) short term memory; b) short term memory, updating and retaining
information; c) short term memory and manipulation; d) short term memory and
keeping information after a delay and e) short term memory, keeping information
after a delay, manipulation and updating. Visual stimuli were provided in 4 x 4 grids
where shapes light up randomly and in sequence, after a pause, they had to be
repeated. The sequence length is adapted as the participant continues and
increased according to correct answers. Participants were provided with computers
and the programme was conducted at home under the supervision of parents.
Young people were required to complete twenty five sessions of forty minutes each.
Psychological Theory
Working memory function has significant educational and functional
importance in developing young people. Rapport et al. (2013) claim that young
people with ADHD are in desperate need of an effective treatment that helps to
improve academic and interpersonal outcomes. Working memory training is
4
Doctorate in Educational and Child Psychology
Tracey Atkins
considered to be hopeful because, for the first time, it addresses the cognitive issues
that are now associated with ADHD. Rapport et al. (2013) argue that whilst
medication and behaviour management work in the short term, they are not
accompanied by gains in academic or learning outcomes (Molina, Hinshaw,
Swanson, Arnold, Vitiello, Jenson… & Houck, P. R. (2009), and it is here that
research for ADHD needs to concentrate its efforts.
Baddeley (2000) describes a model of working memory as one that consists
of components that help to store information through inner voice rehearsal
(articulatory control system); sound patterns (phonological store) or visual
information (visuo-spatial sketchpad). The ‘Episodic Buffer’ binds information
together, ready for storage as memories. The ‘Central Executive’ acts as a
coordinator for all other components and can store all modalities but has a limited
capacity. It is no wonder that deficits in working memory have been associated with
difficulties at school including following instructions, prioritising and organising work
schedules (Gathercole & Alloway, 2008). By improving working memory, it is hoped
that it will play a role in building capacity to perform complex tasks such as learning,
comprehension, reasoning and planning (Baddeley, 2007).
With 66% of permanently excluded children and 75% of children in PRUs
quoted has having special educational needs (DfES, 2008) and that fact that children
with ADHD are more than 100 times more likely to be permanently excluded from
school than other children (O’Regan, 2009), Educational Psychologists are uniquely
placed to draw on the psychological theory of working memory and the
effectiveness of evidence-based research to identify and recommend supportive
interventions to individuals and schools. Programmes that are easily accessible
from school or at home are attractive propositions but it is important for the
5
Doctorate in Educational and Child Psychology
Tracey Atkins
Educational Psychology profession to know which particular programmes suit which
particular profile of individual or group. This paper therefore aims to explore the
evidence available for working memory training on a specific age group of young
people with ADHD in order to inform EP practice in the future.
Rationale
To obtain a diagnosis of ADHD, people need to fulfil the criteria of difficulties
in attention, hyperactivity and impulsivity, as set out by the Diagnostic Statistical
Manual of Mental Disorders (DSM-IV, APA,1994). However, particular difficulties in
executive functioning have also been put forward as a description of the condition
(Barkley, 1997) as well as impaired prefrontal cortex (Schwietzer, Faber, Grafton,
Tune, Hoffman, & Kilts, 2000). Impaired working memory affects social development
as well as academic achievement, for example, logical reasoning and problem
solving (Klingberg, 2000).
The prevalence of young people with ADHD today may amount to between
3% and 9% in the UK alone (NICE, 2008) and the link between improved working
memory skills and young people with ADHD has fundamental implications.
Difficulties identified in young people with ADHD leads us to the review question of
whether working memory training programmes are effective interventions for young
people with ADHD?
6
Doctorate in Educational and Child Psychology
Tracey Atkins
Critical Review of the Evidence Base
Literature Search
Ideally, as far as resources will allow, studies should be screened by at least
two reviewers to minimise errors of judgement, however, on this occasion, these
studies were screened and appraised by one reviewer only.
Electronic databases PsycINFO; ERIC (ProQuest); and MEDLINE were
searched on Sunday, February 2nd and Monday, 3rd February 2014 resulting in the
location of 316 articles. Sequential phrases and terms for each of the above
databases can be found in Table 1 below.
Table 1
Database search terms
Sequence Terms
1
Working
Memory
AND
Training
AND
ADHD
2
Working
Memory
AND
Training
AND
Attention Deficit Disorder
3
Working
Memory
AND
Intervention
AND
ADHD
4
Working
Memory
AND
Training
AND
Attention Deficit Hyperactivity
Disorder
Inclusion and Exclusion Criteria
Studies were included in the review if they met the criteria outlined in
Appendix 1. For this systematic review, inclusion and exclusion criteria were
narrowed simply to look at whether practice in working memory skills can alter the
7
Doctorate in Educational and Child Psychology
Tracey Atkins
way working memory functions. Likewise, programmes measuring predominantly
“attention” difficulties (such as “Pay Attention!”) were excluded at the title stage.
The criteria for participants have been quite stringent so that groups between
studies can be matched as closely as possible. For example, as far as information
has been available, groups have a similar proportion of those taking medication to
those who are not. The diversity of a diagnosis of ADHD and the range of its comorbidities has also meant that research has been chosen where as little of known
co-existing behavioural difficulties are included as possible. Comorbidities that may
have a perceptible bearing on the results of the training programme are Oppositional
Defiance Disorder (ODD), Autistic Spectrum Disorder (ASD) and Conduct Disorder
(CD).
Research has also been specifically chosen where data on changes in
working memory can be extracted separately in order to consider its specific impact.
Although peer reviewed research does not guarantee validity, only peer reviewed
research was included on the basis that some measure of quality assurance had
been implemented. All studies include young people who fit within the school age
range where the implications of such training have significant educational
importance.
There were no restrictions on the date of the studies and studies were also
open to international research that could be obtained in English. Subsequent to the
initial search, 47 articles were excluded as duplications followed by 187 by title
alone. A further 55 articles were excluded by abstract leaving 27 articles to be
examined at full text.
On closer inspection, it was found that 10 studies that had been included in
prior meta-analyses (Rapport, Orban, Kofler, & Friedman,2013 and Melby-Lervag &
8
Doctorate in Educational and Child Psychology
Tracey Atkins
Hulme, 2013) or systematic reviews (Chacko, Fiersen, Bedard, Marks, Uderman, &
Chimiklis, 2013) relating to working memory training, did not fit with the participant
profile. These studies were excluded on the grounds that some participants with
clinical diagnoses of ADHD were mixed with participants identified with a variety of
special needs, some participants were selected through teacher/parent ratings of
attention difficulties (Beck, 2010; Dahlin, 2011, 2013;) and some groups were not
matched in clinical diagnosis with comparison control groups (Dahlin, 2013).
Two promising recent studies were found (Hovik, Saunes, Aarlien, & Egeland,
2013; Egeland, Aarlien, & Saunes, 2013), but were excluded on the basis that
participants were diagnosed with Hyperkinetic Disorder. According to the
International Classification of Diseases and Related Health Problems criteria (ICD10, World Health Organisation, 1992), Hovik et al. (2013) and Egeland et al. (2013)
suggest that a diagnosis of Hyperkinetic Disorder is equivalent to a diagnosis of
ADHD but only to ADHD-C (combined type). Whilst supporting the view that there
may be overlaps in diagnosis, Tripp, Luk, Schaughency, & Singh (1999) also report
that this overlap is imperfect. A diagnosis of ADHD using the DSM-IV criteria
identifies a broader range of young people spanning two further subtypes (ADHDInattentive and ADHD-Hyperactive). This would imply, therefore, that the ICD-10
may fail to diagnose young people who do not have the combined symptoms of
hyperactivity and inattention.
Furthermore, Lahey, Pelham, Loney, & Willcutt (2005) found that although a
diagnosis of ADHD was enduring, the severity of symptoms and the identification of
an original subtype were changeable over time. For the purposes of obtaining as
close a match as possible regarding the effectiveness of an intervention, it would
9
Doctorate in Educational and Child Psychology
Tracey Atkins
therefore be appropriate to exclude this group as a partial representation of the
ADHD population.
Of the remaining 17 articles at full text, one was found to be a study protocol;
another a Master’s level thesis that was not peer reviewed; three were found to be
reviews only; seven were found to investigate either particular conditions within a
single working memory training programme (such as a gaming element for
motivation) or compare two types of Working Memory Training programme rather
than the effectiveness of one.
A detailed list of articles excluded according to the inclusion criteria from full
text stage onwards is summarised in Appendix 2 and a summary of the five chosen
studies can be found in Appendix 3.
Critical Review of Studies
The critical review will outline how the individual studies in this systematic
review were judged in terms of weight of evidence. It will critically consider issues
with design, participant selection, intervention, measures and finally, analysis.
Three types of published study are identified as randomized groups, pretestposttest designs (Barker, Pistrang, & Elliot, 2002). These studies included Green, et
al. (2012); Van der Oord, et al. (2012) and Chacko et al. (2013). Another study is of
a similar design (Klingberg et al. 2002) but information on the randomisation of
participants is not provided in their study. The 5th study is a nonrandomized onegroup pretest-post-test design (Barker, Pistrang, & Elliot , 2002) by Holmes, et al.
(2010).
10
Doctorate in Educational and Child Psychology
Tracey Atkins
Each study was examined for methodological, comparison group and
statistical analysis quality using an adapted version of coding from the APA Task
Force on Evidence-Based Interventions in School Psychology (Kratochwill, 2003).
Coding protocols can be found in Appendix 4. In order to determine the
compatibility and contribution of the chosen studies to the review question, Gough’s
(2007) Weight of Evidence Framework was used, a summary of which can be found
in table 2 The Weight of Evidence Framework allows studies to be rated in terms of
quality of methodology; relevance of methodology; and relevance of evidence to the
review question. It needs to be noted, however, that Weight of Evidence scales are
subjective and while judgements may indicate a quality study overall, there may still
be methodological flaws contained within it (Wright, Brand, Dunn, & Spindler, 2007).
As a result, all studies contained within the Weight of Evidence Framework and
criteria for this review will be used, regardless of the result of quality, in order for
readers to draw their own conclusions (Gough, Oliver, & Thomas, 2012).
Table 2
Weight of Evidence Framework (Gough, 2007)
Weight of Evidence
A
Weight of Evidence
B
Weight of Evidence
C
Weight of Evidence
D
Quality of
Methodology:
Relevance of
Methodology:
Overall weight of
evidence:
The accuracy,
coherency and
transparency of
evidence.
The appropriateness
of the methodology
for answering the
review question.
Relevance of
evidence to the
review question:
The relevance of the
focus of the evidence
for answering the
review question.
11
Overall assessment
of the extent to which
the study provides
evidence to answer
the review question
Doctorate in Educational and Child Psychology
Tracey Atkins
An average of these weightings was taken to establish the study’s Overall Weight of
Evidence (see Appendix 5).
Participants
Studies were carried out in different countries. These included the
Netherlands (Van der Oord et al. 2012), Sweden (Klingberg et al. 2002), the UK
(Holmes et al. 2010) and two in the USA (Chacko et al. 2013; Green et al. 2012).
This range of International studies may provide a good representation of results
within the mainstream school age range . The mean age of participants ranged from
8 years, 4 months old (Chacko et al. 2013) to 11 years old (Klingberg et al. (2002).
As the criteria of this review included children clinically diagnosed with ADHD,
preliminary recruitment had to be restricted to institutions where a diagnosis was
known, therefore, initial recruitment ranged from community or institute
advertisements (Chacko et al. 2013; Green et al. 2012); or outpatient mental health
clinics via paediatrician and psychiatrist registers (Van der Oord et al. 2012; Holmes
et al. 2010). Klingberg et al. (2002) do not indicate their selection procedure.
There are obvious limitations attached to the initial voluntary recruitment of ADHD
candidates before randomisation. It may be helpful to view ADHD as a “cluster”
group in order to account for the variety of symptoms, subtypes and confounding
variables that may affect the results of the study.
Potential generalizability may also be limited due to four out of five of the
studies failing to provide adequate sample sizes. Based on power analyses using GPower, Green et al. (2012) was the only study that met the required sample size of
22 participants by reporting on 26 participants. Van der Oord et al. (2012) was
calculated to require 350 participants while only reporting on 40. Where the effect
12
Doctorate in Educational and Child Psychology
Tracey Atkins
sizes were available for different dependent variables within one study (Klingberg et
al. 2002; Chacko et al. 2013), the median effect size was used to calculate
appropriate sample size through G-Power (Rosenthal & Rubin, 1986). This is due
to the fact that effect sizes for each of the dependent variables varied greatly.
Rosenthal & Rubin (1986) suggest that although this is likely to be a more
conservative estimate than using the mean, the mean is only a good representation
if the dependent variable effect sizes are virtually equal. Klingberg et al. (2002)
required a sample size of 46 while only reporting on 14; and Chacko et al. (2013)
required a sample size of 390 while reporting only on 85. Inadequate samples sizes
undermine the effect size results and compromise generalizability further.
Information on the matching of treatment to control groups was also varied.
Klingberg et al. (2002) do not provide details but suggest that there were no
significant differences between groups. Lack of information on how the groups differ
is reflected in a ‘low’ Weight of Evidence rating for this study. Green et al. (2012),
Van der Oord et al. (2012) and Chacko et al. (2013) illustrate group equivalence by
providing details on demographic status, gender, age, and ADHD subtype with Van
der Oord et al. (2012) and Chacko et al. (2013) also providing equivalence in IQ, and
medication use. Further details are provided by Van der Oord et al. (2012) on
experience of gaming and Chacko et al. (2013) on percentage of comorbidities and
marital status of parents.
Methodology
The type of working memory intervention varied across the studies. Three
studies use “CWMT” (Holmes et al. 2009; Green et al. 2012; Chacko et al. 2013).
One study (Klingberg et al. 2002), reported the use of an unnamed specifically
13
Doctorate in Educational and Child Psychology
Tracey Atkins
designed computerised intervention; and the remaining study used a gaming
intervention called “Braingame Brian” (Van der Oord et al. 2012). Although Chacko
et al. (2013) intended to measure more generalised outcomes from the training
programme, the study was awarded medium for Weight of Evidence within measures
category of Weight of Evidence (A) due to the fact that the reliability of the measure
was noted, results for working memory changes were able to be specifically
extracted, and effect sizes for each were recorded.
Van der Oord et al. (2012) received a rating of ‘high as the results and effect
size for the working memory element of study were all providedalong with
randomization of groups and a control group.. Green et al. (2012) received a
medium rating as the working memory element of the study, although explicit in its
results, was secondary to the main measurement of on-task behaviour and only
measured using the ‘Working Memory Index’ subtest in the Weschler Intelligence
Scale for Children (WISC-IV, 2003). Effect sizes for Green et al. (2012) and
Klingberg et al. (2002) were calculated for the purposes of this review using Cohen’s
d formula (Cohen, 1992).
A wait-list control design was used by Van der Oord et al. (2012), this allows
for a direct comparison with the same intervention, but there is the possibility of
“leakage”, for example, information about the computerised games being disclosed
by participants in the treatment group to participants in the wait-list group.
The use of placebo groups was used by Green et al. (2012), Klingberg et al.
(2002) and Chacko et al. (2013). Although three studies used double-blind placebo
conditions (Chacko et al. 2013; Klingberg et al. 2002; Green et al. 2010), the fact that
all of them used computerized programmes with adjustability to test the impact on
working memory, means that the element of motivation could have contributed to the
14
Doctorate in Educational and Child Psychology
Tracey Atkins
results. ‘Adjustability’ refers to the automated increase or decrease in level of game
difficulty according to the scores obtained in previous games played in the
intervention programme. Chacko et al. (2013) found that without adjustability in the
games, motivation decreased and participants completed less games. Without the
acknowledgment of motivation as a confounding variable, the weight of evidence for
the methodological quality of Klingberg et al. (2002) and Green et al. (2010) is limited
(see Appendix 5). All randomized controlled designs did not suffer from reported
attrition rates at posttest assessment.
Klingberg et al. (2002) was rated low for overall weight of evidence. Although
the intervention used by Klingberg et al. (2002) was primarily intended to measure
the effect of a working memory programme, sample sizes were small and lack of
detail on limitations and allocation of participants reduced the opportunity to evaluate
the study adequately. Reviews that limit information may be subject to publication
bias and should be approached with caution (Wright et al. 2007). Although rated a
medium for overall weight of evidence, Holmes et al. (2010) also report a small
sample size and the lack of a concurrent control group. Limitations of the nonrandomised one-group pretest-posttest design may include threats to construct
validity through potential confounding variables, and expectancy effects, and internal
validity such as maturation and endogenous change (Barker, Pistrang, & Elliot,
2012). There is no sign in this study of an attempt to mitigate the effects of such
limitations.
Measures
Klingberg et al. (2002) used a variety of measures for working memory.
These included tests for visuo-spatial skills; a forward and backward span board
15
Doctorate in Educational and Child Psychology
Tracey Atkins
using blocks, and the stroop test. Each of these were scored separately for
independent analysis, however, no reliability scores were given for the measures
used.
Van der Oord et al. (2012) used the Behaviour Rating Inventory of Executive
Functioning (BRIEF) and reported test-retest reliability was “good” (p4), however, the
“working memory” component was scored separately providing an accessible and
appropriate measure of working memory on its own. Unfortunately though, details
on the exact breakdown of the “working memory” component are not stated and so
can’t be compared to other measures of “working memory”.
For the working memory section in the studies reported by Chacko et al.
(2013) and Holmes et al. (2010), the Automatic Working Memory Assessment
(AWMA) was used. Chacko et al. (2013) measured dot matrix; spatial recall; digit
recall and listening recall. Holmes et al. (2010) measured and reported test-retest
reliability scores for dot matrix (.85); digit recall (.89); backward digit recall (.86) and
Mr X (.84). Green et al. (2012) used the Working Memory Index from the WISC-IV
but do not report reliability scores.
All studies provided information from which to calculate effect sizes relating
directly to working memory performance and as such, specifically relating to the
research question for this study. In this respect, weight of evidence ratings of high
were awarded to Green et al. (2012) Van der Oord et al. (2012) and Holmes et al.
(2010); and medium to Klingberg et al. (2002) and Chacko et al. (2013). Despite the
provision of some reliability scores, most of the data to measure impact on working
memory alone only included one instrument of measure. The use of more than one
instrument of measurement increases measurement validity and reliability.
16
Doctorate in Educational and Child Psychology
Tracey Atkins
Findings
Table 3 (below) shows a summary of the effect sizes and overall quality of the
studies included in this review. All studies except (Klingberg et al. 2002) looked at
the impact of working memory training programmes on a range of executive
functions but it was possible for this systematic review to extract particular
information on the changes in working memory function alone.
Chacko et al. (2013) obtained an overall weight of evidence” of medium due
to the standard of information available for analysis; and recognition of limitations.
Effect sizes were supplied showing the most improvement in non-verbal storage
using the ‘Dot Matrix’ element of the test (d=1.17), small effect sizes reported were
reported for ‘Spatial Recall’ and ‘Digit Recall’ (d=0.29 and 0.28 respectively).
Van der Oord et al. (2012) received an overall weight of evidence of high
based on the information available and a large effect size ( 2p =0.05). Although the
authors report improvements in executive functions in general, they also report that
the lack of positive effects of this training programme on working memory and setshifting were unexpected. However, praise for the authors needs to be
acknowledged as they critique the use of a parent scale (BRIEF) as potentially
biased; and the part parents play in the selection of participants. They also critique
the effect parents have as supervisors in the home programme.
17
Doctorate in Educational and Child Psychology
Tracey Atkins
Table 3
Effect sizes and overall quality ratings
Study
Green et al.
(2012)
Klingberg
et al. (2002)
Outcomes
Measures
Generalisation to ‘on task’
behaviour in class
WM composite of WISC IV
Digit span and letter number sequence tests
Decrease symptoms of
ADHD & increase WM
capacity
Visuo-spatial task
(author’s own design)
Stroop test accuracy
Span Board task
(author’s own design)
Time for completion
Effect Size Descriptors
Large
Effect Size
Cohen’s d (calculated from given
data)
d = 1.3
Medium
d = 0.59
Medium – Large
Large
d = 1.11
d = 0.72
Small
d = 0.16
Overall Weight of
Evidence
Medium
Low
Cohen’s d (calculated from given
data)
Van der Oord
et al. (2012)
Increase in cognitive
flexibility, inhibition & WM
capacity
Medium
2p = 0.05
High
(Cohen, 1998)
Decrease symptoms of
ADHD & increase academic
achievement
Automatic Working Memory Assessment (Alloway, 2007)
Impact of WMT on WM
Automatic Working Memory Assessment (Alloway, 2007)
WASI (Weschler, 1999) (different tests used pre and post to control for
re-test effects)
Chacko et al.
(2013)
Holmes et al.
(2010)
BRIEF (Gioia, Isquith, Guy & Kenworthy, 2000)
Dot Matrix
Spatial Recall
Digit Recall
Listening Recall
Large
Small - Medium
Small - Medium
Small
Verbal STM:
Medium
Visuo-spatial STM:
Large
Verbal WM:
Medium
Visuo-spatial WM:
Small - Medium
d = 1.17
d = 0.29
d = 0.28
d = 0.07
2p = 0.27
2p = 0.66
2p =0.29
2p .= 0.15
(Ferguson, 2009)
18
Medium
Medium
Doctorate in Educational and Child Psychology
Key:

2
p
d
Small
Medium
Large
(Cohen, 1998)
0.01
0.06
0.14
(Ferguson, 2009)
0.04
0.25
0.64
(Cohen, 1998,
1992)
0.2
0.5
0.8
SEM was calculated into SDs by the following: SEM x square root of N; Cohen’s d = post-test mean for
intervention – post-test mean for control divided by (averaged) SD of both post-test intervention and control
(Cohen, 1992). STM = short term memory; WM = Working Memory
Green et al. (2012) received an overall weight of evidence of medium. They
report statistical significance for working memory improvement through the Working
Memory Index (WMI) of the WISC-IV exclusively to the treatment group, however, they
were looking for changes in ADHD symptoms such as on-task behaviours and
characteristics from the Connors Ratings Scale and draw attention to the limitations of
measuring groups, such as ADHD, where there is a wide range of symptoms, subtypes
and comorbid disorders. The effect size calculated from the data available was rated as
Large (d=1.3).
Holmes et al. (2010) received an overall weight of evidence of medium. Authors
claim that significant gains in working memory were found from working memory training.
Effect sizes show that verbal short-term memory was medium ( 2p =0.27); visuo-spatial
short-term memory was large ( 2p =0.66); verbal working memory was medium ( 2p
=0.29) and visuo-spatial working memory was small-medium ( 2p =0.15). Credit must be
accorded to the authors as they comment on the limitations of a one-group design in that
it is not clear whether improvement is due to the training programme or the programme
supporting the development of “strategies to compensate for weakness” (p833).
19
Doctorate in Educational and Child Psychology
Claims by Klingberg et al’. (2002) that the programme increased working memory
capacity was not consistently borne out by the calculation of the effect sizes taken from
both the control group and the treatment group. The strongest effect was for stroop test
accuracy which was rated as large (d=1.11); followed by span board task which was
rated as medium-large (d=0.72); visuo-spatial working memory which was rated as
medium (d=0.59) and finally time for stroop test completion which was rated as small
(d=0.16).
Although the study itself was relevant to the review question and test-retest
activities were varied so as to avoid reactivity, information to analyse the effectiveness of
the intervention independently, was extremely limited. This information further supported
a rating of low for overall weight of evidence.
Conclusions and Recommendations
Strengths
Based on the best available evidence from the studies examined, it appears that
working memory training, delivered through a range of computerised programmes, can
have a positive effect on working memory function but it is unclear whether this is due to
practice effects or the intervention alone. Many studies have commented on the fact that
there is generalizability potential but this is yet to be conclusive. Effect sizes where
different aspects of working memory data could be isolated showed that stroop test
accuracy and span board task (Klingberg et al. 2002) seem to be effective (however, the
lack of reliability for these results has already been discussed). A large effect size was
also found for visuo-spatial short-term memory by Holmes et al. (2010) who were given
an overall weight of evidence rating of medium. Chacko et al. (2013) also found a large
effect size for dot matrix tasks and also received a rating of medium foroverall weight of
evidence. Finally, Green et al. (2012) found an overall large effect size for his study and
20
Doctorate in Educational and Child Psychology
received a rating of medium for overall weight of evidence. This may draw us to the
conclusion that there is potential for working memory training to confirm that working
memory function is plastic; and that it may be especially useful for developing visuospatial and sequencing skills.
Limitations
It is difficult to get a true representation of participants with ADHD due to the
range and severity of symptoms. Some researchers feel that certain subtypes are
separate disorders entirely (Lahey, Pelham, Loney, Lee, & Willcutt2005 and Milich,
Balentine, & Lynam, 2001). Non-randomisation (Holmes et al. 2010) and high
occurrences of comorbidities (such as ODD), especially compared to the treatment group
(Van der Oord et al. 2012), can be subject to regression to the mean and distort results.
Lack of a control group can also interfere with representative results (Holmes et al.
(2010) and fails to provide convincing evidence for the effects of one particular
intervention over another.
Limitations such as small sample sizes (Klingberg et al. 2002; Holmes et al. 2010;
Van der Oord et al. 2012; Chacko et al. 2013) and studies that use automated
adjustments in the treatment condition but none in the placebo condition;, may mean that
effects for interventions are overestimated.
‘Adjustability’ in computer interventions (such as CWMT) contains a highly
motivating component to the programme (Chacko et al. 2013) the lack of which, for
example, in a placebo condition, may be seen to favour the treatment group and boost
results.
21
Doctorate in Educational and Child Psychology
Recommendations
Given the range of findings in the studies included in this systematic review, it is
possible to suggest that WMT can help to improve working memory function for some
participants (Green et al. 2012) and for some areas of working memory (visuo-spatial
and sequencing skills). It is important for Educational Psychologists to inform schools
about the fact that research in the area is in one respect, diverse, offering the client an
aesthetically attractive intervention with a choice of game to play; but in another respect,
the client appears to be rather limited to certain working memory programmes (such as
CWMT and Braingame Brian). A greater choice of activities that schools could use to
improve working memory may be an area of development for research. Further
recommendations highlight the need to address the wide variety of assessment tools
used to measure the impact of working memory training and the difficulties with selfreport (Van der Oord et al. 2012). Forthcoming research on working memory training
may benefit from the use of a universal scale, such as the AWMA either on its own or
alongside other measures; and for researchers to report the reliability data for each
measure as a matter of course.
The question of motivation must also be taken into account. As computer
programmes with adjustability appear to sustain the attention and interest of young
people, future research should aim to restrict comparison groups to programmes with a
similar advantage.
22
Doctorate in Educational and Child Psychology
References
Baddeley, A. (2000). The episodic buffer: A new component of working memory?
Trends in Cognitive Sciences, 4, 417-423.
Baddeley, A. (2007). Working Memory, thought, and action. Oxford University Press
Barker, C., Pistrang, N., & Elliot, R. (2012). Research methods in clinical and
counselling psychology. London: John Wiley & Sons.
Barkley, R. A. (1997). Behavioural inhibition, sustained attention, and executive
functions: Constructing a unifying theory of ADHD. Psychological Bulletin,
121, 65-94.
Beck, S. J., Hanson, C. A., Puffenberger, S. S., Benninger, K. L., & Benninger, W. B.
(2010). A controlled trial of working memory training for children and
adolescents with ADHD. Journal of Clinical Child & Adolescent Psychology,
39(6), 825-836.
Braingame Brian (2014). Retrieved from
http://www.gamingandtraining.nl/beschrijving-braingame-brian/
Chacko, A., Bedard, A. C., Marks, D. J., Feirsen, N., Uderman, J. Z., Chimiklis, A.,
Rajwan, E., Cornwell, M., Anderson, L., Zwilling, A., & Ramon, M. (2013). A
randomized clinical trial of Cogmed Working Memory Training in school‐age
23
Doctorate in Educational and Child Psychology
children with ADHD: a replication in a diverse sample using a control
condition. Journal of Child Psychology and Psychiatry. doi:10.1111/jcpp.12146
Chacko, A., Feirsen, A., Marks, D., Uderman., & Chimklis. (2013). Cogmed
Working Memory Training for Youth with ADHD: A Closer Examination of
Efficacy Utilizing Evidence-Based Criteria. Journal of Clinical Child &
Adolescent Psychology, 42(6), 769-783.
Cohen, J. (1992). "A power primer". Psychological Bulletin, 112 (1),155–159.
doi:10.1037/0033-2909.112.1.155
Cohen, J. (1998). Statistical Power Analysis for Behavioural Sciences (2nd Ed).
New York, USA: Lawrence Erlbaum Associates, Publishers.
Dahlin, K. I. (2011). Effects of working memory training on reading in children with
special needs. Reading and Writing, 24(4), 479-491.
Dahlin, K. I. (2013). Working Memory Training and the Effect on Mathematical
Achievement in Children with Attention Deficits and Special Needs. Journal of
Education & Learning, 2(1).
Department for Children, Schools and Families. (2008) Back on track: a strategy for
modernizing alternative provision for young people. Command Paper; Cm 7410
30. HM Government (2008) Youth Crime Action Plan 2008. London: Central Office
24
Doctorate in Educational and Child Psychology
of Information.
Egeland, J., Aarlien, A. K., & Saunes, B. K. (2013). Few effects of far transfer of working
memory training in ADHD: a randomized controlled trial. PloS one, 8(10).
Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers.
Professional Psychology: Research and Practice, 40(5), 532.
Gathercole, S. E., & Alloway, T. P. (2008). Working memory and learning: A
teacher’s guide. California, USA: Sage Publishing.
Green, C. T., Long, D. L., Green, D., Iosif, A. M., Dixon, J. F., Miller, M. R., ... &
Schweitzer, J. B. (2012). Will working memory training generalize to improve
off-task behavior in children with attention-deficit/hyperactivity disorder?.
Neurotherapeutics, 9(3), 639-648.
Gough, D. (2007). Weight of Evidence: A Framework for the Appraisal of the Quality
and Relevance of Evidence. Research Papers in Education, 22, 213-228.
Gough, D., Oliver, S., & Thomas, J. (2012). An Introduction to Systematic Reviews.
London, England: Sage Publications Ltd.
Holmes, J., Gathercole, S. E., Place, M., Dunning, D L., Hilton, K. A., & Elliott, J. G.
25
Doctorate in Educational and Child Psychology
(2010). Working memory deficits can be overcome: Impacts of training and
medication on working memory in children with ADHD. Applied Cognitive
Psychology, 24(6), 827-836.
Hovik, K. T., Saunes, B. K., Aarlien, A. K., & Egeland, J. (2013). RCT of Working
Memory Training in ADHD: Long-Term Near-Transfer Effects. PloS one, 8(12).
Klingberg, T. (2000). Limitations in information processing in the human brain:
Neuroimaging of dual task performance and working memory tasks. Progress
in Brain Research, 126, 95-102.
Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Training of working memory
in children with ADHD. Journal of Clinical and Experimental Neuropsychology,
24(6), 781-79.
Klingberg, T. (2010). Training and plasticity of working memory. Trends in
CognitiveScience, 14, 177-186.
Kratochwill, T.R., & Shernoff, E.S. (2003). Evidence-Based Practice: Promoting
Evidence-Based Interventions in School Psychology. School Psychology
Quarterly, 18(4), 389-408.
Lahey, B. B., Pelham, W. E., Loney, J., Lee, S. S., & Willcutt, E. (2005). Instability of
the DSM-IV subtypes of ADHD from preschool through elementary school.
Archives of General Psychiatry, 62(8), 896-902.
26
Doctorate in Educational and Child Psychology
Lahey, B. B., & Willcutt, E. G. (2010). Predictive validity of a continuous alternative
tonominal subtypes of attention-deficit/hyperactivity disorder for DSM–V.
Journal of Clinical Child & Adolescent Psychology, 39(6), 761-775.
Lee, S. I., Schachar, R. J., Chen, S. X., Ornstein, T. J., Charach, A., Barr, C., &
Ickowicz, A. (2008). Predictive validity of DSM‐IV and ICD‐10 criteria for
ADHD and hyperkinetic disorder. Journal of child psychology and Psychiatry,
49(1), 70-78.
Makris, N., Biederman, J., Valera, E. M., Bush, G., Kaiser, J., Kennedy, D. N., ... &
Seidman, L. J. (2007). Cortical thinning of the attention and executive function
networks in adults with attention-deficit/hyperactivity disorder. Cerebral Cortex,
17(6), 1364-1375.
Melby-Lervåg, M., & Hulme, C. (2013). Is working memory training effective? A
meta-analytic review. Developmental Psychology, 49(2), 270.
Milich, R., Balentine, A. C., & Lynam, D. R. (2001). ADHD combined type and ADHD
predominantly inattentive type are distinct and unrelated disorders. Clinical
Psychology: Science and Practice, 8(4), 463-488.
27
Doctorate in Educational and Child Psychology
Molina, B. S., Hinshaw, S. P., Swanson, J. M., Arnold, L. E., Vitiello, B., Jensen, P. S., .
.. & Houck, P. R. (2009). The MTA at 8 years: prospective follow-up of children
treated for combined-type ADHD in a multisite study. Journal of the American
Academy of Child & Adolescent Psychiatry, 48(5), 484-500.
National Institute for Health and Clinical Excellence. (2008). Attention Deficit
Hyperactivity Disorder: Diagnosis and Management of ADHD in Children, Young
People and Adults. London, England: NICE Clinical Guidelines, CG72.
O’Regan, F. (2009) Persistent disruptive behaviour and exclusion. ADHD in Practice,
1(1): 8–11
Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013). Do programs
designed to train working memory, other executive functions, and attention
benefit children with ADHD? A meta-analytic review of cognitive, academic,
and behavioral outcomes. Clinical psychology review, 33(8), 1237-1252.
Schweitzer, J. B., Faber, T. L., Grafton, S. T., Tune, L. E., Hoffman, J. M., &
Kilts, C. D. (2000). Alterations in the functional anatomy of working memory in
adult attention deficit hyperactivity disorder. American Journal of
Psychiatry, 157(2), 278-280.
28
Doctorate in Educational and Child Psychology
Tripp, G., Luk, S. L., Schaughency, E. A., & Singh, R. (1999). DSM-IV and ICD-10: A
Comparison of the Correlates of ADHD and Hyperkinetic Disorder. Journal of
the American Academy of Child & Adolescent Psychiatry, 38(2), 156-164.
Tzuriel, D. (2001). Dynamic assessment of young children. (pp. 63-75). New York:
Kluwer Academic/Plenum Publishers.
Van der Oord, S., Ponsioen, A. J. G. B., Geurts, H. M., Ten Brink, E. L., & Prins, P.
J. M. (2012). A pilot study of the efficacy of a computerized executive
functioning remediation training with game elements for children with ADHD in
an outpatient setting: outcome on parent-and teacher-rated executive
functioning and ADHD behaviour. Journal of Attention Disorders,
1-14. doi:10.1177/1087054712453167
Special Educational Needs: Report of the Committee of Enquiry into the
Education and Handicapped Children and Young People. (2008). London,
England: Her Majesty’s Stationery Office. Retrieved March 2014 from
http://www.educationengland.org.uk/documents/warnock/warnock1978.html
Willcutt, E. G., Doyle, A. E., Nigg, J. T., Faraone, S. V., & Pennington, B. F. (2005).
Validity of the executive function theory of attention-deficit/hyperactivity disorder: a
meta-analytic review. Biological psychiatry, 57(11), 1336-1346.
29
Doctorate in Educational and Child Psychology
Woolfson, L. M. (2011). Educational Psychology: The Impact of Psychological
Research on Education. United Kingdom: Prentice Hall.
Wright, R. W., Brand, R. A., Dunn, W., & Spindler, K. P. (2007). How to Write a
Systematic Review. Clinical Orthopaedics and Related Research,
455, 23-29.
30
Doctorate in Educational and Child Psychology
Appendix 1
Inclusion and exclusion criteria for studies
Inclusion Criteria
Exclusion Criteria
Justification
1. Type of
Publication
Peer reviewed journal
Material not in peer review journal
To ensure
methodological rigour
2. Language
Written in English or translated into English
Articles not written in English
Translation services
were not available
3. Intervention
The study must specifically look at training
programmes or interventions intended to
measure the effectiveness of working memory
performance compared to non-working memory
training programmes.
Studies where the working memory component
can be extrapolated from a general Executive
Function training programme.
Interventions that use the same working memory programme to compare
two aspects of working memory; Interventions looking at one particular
aspect of Executive Function, such as inhibition or attention alone.
To be able to ensure
validity and consistency
across studies
4. Primary source
Must be reporting on primary empirical research
evidence
Citing secondary data; reviews of previous research.
To ensure originality of
findings
5. Participants
Young people of compulsory school age: 5 –
17yrs;
Young people with a primary clinical diagnosis of
ADHD or ADD (any subtype).
Pre 5 years of age and Post 17 years of age; adults; pre-school; young
people not of statutory school age; ratings of attention difficulties by
parental or teacher self-report; self-diagnosed ADHD/ADD; ADHD with
severe learning difficulties; Young people who are all taking medication;
young people with a diagnosis of Hyperkinetic Disorder.
To ensure consistency
across the population.
To be able to examine
the implications for
learning and education
6. Methodology
Designs that include a waitlist; control group;
alternative comparison programme; information
from which to calculate effect sizes; pre and post
measures evaluating outcomes of working
memory training.
Designs without the following: control groups; alternative comparison
programmes; waitlist groups; pre or post measures; non-empirical
reviews; Single case designs; interventions focussing on attention.
To ensure as far as
possible the effects of
the intervention alone.
7. Dates
Research up to and including 2014 to date.
N/A
To include the latest
research available.
8. Setting
Mainstream / special / state schools or home
settings for young people aged between 5 and
17 years.
Pre-schools; nurseries; universities.
To consider the
implications for learning
and statutory education.
31
Doctorate in Educational and Child Psychology
Appendix 2
Excluded Studies
Study
Exclusion Criteria
1. Gropper, R. J., Gotlieb, H., Kronitz, R., &
Tannock, R. (2014). Working Memory Training
in College Students with ADHD or LD. Journal
of attention disorders, 1087054713516490.
Excluded at full text
Criteria 5 – Participants at University aged 19
– 52 years old.
2. Johnstone S, J., Roodenrys S., Blackman R.,
Johnston E., Loveday K., Mantz S., Barratt M.
F. (2012) Neurocognitive training for children
with and without AD/HD. Attention Deficit and
Hyperactivity Disorders. 4(1):11-23.
3. Johnstone S, J., Roodenrys S., Phillips, E.,
Watt, A. J., Mantz S. (2010) Neurocognitive
training for children with and without AD/HD.
Attention Deficit and Hyperactivity Disorders.
4(1),11-23.
4. Wass, S. V., Scerif, G., & Johnson, M. H.
(2012). Training attentional control and working
memory–Is younger, better?. Developmental
Review, 32(4), 360-387.
5. Rutledge, K. J., Van den Bos, W., McClure, S.
M., Schweitzer, J. B. (2012). Training Cognition
in ADHD: Current Findings, Borrowed
Concepts, and Future Directions.
Neurotherapeutics. Vol 9, 542-558
6. Lineweaver, T. T., Kercood, S., O'Keeffe, N. B.,
O'Brien, K. M., Massey, E. J., Campbell, S. J.,
Pierce, J. N. (2012). The effects of distraction
and a brief intervention on auditory and visualspatial working memory in college students with
attention deficit hyperactivity disorder. Journal
of Clinical and Experimental Neuropsychology.
Vol.34(8), 791-805.
7. Beck, S. J., Hanson, C. A., Puffenberger, S. S.,
Benninger, K. L., & Benninger, W. B. (2010). A
controlled trial of working memory training for
children and adolescents with ADHD. Journal of
Clinical Child & Adolescent Psychology, 39(6),
825-836.
8. Dahlin, K. I. (2011). Effects of working memory
training on reading in children with special
needs. Reading and Writing, 24(4), 479-491.
Excluded at full text
Criteria 3 – intervention was a combined
Working Memory and Impulse Control
programme.
9. Roberds, J. A. (2005). The Effect of Interactive
MetronomeRTM Training on Response
Excluded at full text
Criteria 3 – focussed on inhibition only.
32
Excluded at full text
Criteria 3 – intervention was a combined
Working Memory and Impulse Control
programme
Excluded at full text
Criteria 4 – review of mixed interventions for
developing control and working memory.
Excluded at full text
Criteria 4 – review of mixed interventions for
developing control and working memory
Excluded at full text
Criteria 5 – College students aged 19 – 20
years old.
Excluded at full text
Criteria 5 – participants were not officially
stated to be clinically diagnosed with ADHD.
Selection based on rating scales that fitted
within DSM-IV criteria. Also known
comorbidities.
Excluded at full text
Criteria 5 – control group 100% diagnosed
with ADHD whilst treatment group only 33%
ADHD. It was not possible to separate the
clinically diagnosed participants in the data.
Doctorate in Educational and Child Psychology
Inhibition Within ADHD Children. ProQuest.
10. Dahlin, K. I. (2013). Working Memory Training
and the Effect on Mathematical Achievement in
Children with Attention Deficits and Special
Needs. Journal of Education & Learning, 2(1).
Excluded at full text
Criteria 5 – Out of 57 participants, 17 in
treatment group had a prior diagnosis of
ADHD and only 3 in the control group.
11. Holmes, J., Gathercole, S. E., & Dunning, D. L.
(2009). Adaptive training leads to sustained
enhancement of poor working memory in
children. Developmental science, 12(4), F9F15.
Excluded at full text
Criteria 3 – focused on differentiating
between adaptive and non-adaptive working
memory training therefore, not directly the
effect compared to a non-working memory
training comparison.
Excluded at full text
Criteria 5 – 100% participants on medication.
Research cannot be matched to results from
those who include a mixture or nonmedicated ADHD participants.
Excluded at full text
Criteria 5 – participants were mixed with
small minority diagnosed with ADHD.
12. Hovik, K. T. (2010). Can PC-based training
boost working memory in ADHD
preadolescents on medication?:a clinical
intervention study.
13. Shavelson, R. J., Yuan, K., Alonzo, A. C.,
Klingberg, T., & Andersson, M. (2008). On the
Impact of Computer Cognitive Training on
Working Memory and Fluid Intelligence1.
Fostering change in institutions, environments,
and people: A festschrift in honor of Gavriel
Salomon, 35.
14. Steiner, N. J., Sheldrick, R. C., Gotthelf, D., &
Perrin, E. C. (2011). Computer-based attention
training in the schools for children with attention
deficit/hyperactivity disorder: a preliminary trial.
Clinical Pediatrics, 50(7), 615-622.
15. Mawjee, K. (2013). ‘Working Memory Training
in post-secondary students with AttentionDeficit/Hyperactivity Disorder: Pilot study of the
differential effects of training session length’.
(Doctoral dissertation, University of Toronto).
16. Gray, S. A., Chaban, P., Martinussen, R.,
Goldberg, R., Gotlieb, H., Kronitz, R.,
Hockenberry, M., Tannock, R. (2012). Effects of
a computerized working memory training
program on working memory, attention, and
academics in adolescents with severe LD and
comorbid ADHD: a randomized controlled trial.
Journal of Child Psychology and Psychiatry,
53(12), 1277-1284.
17. Klingberg, T., Fernell, E., Olesen, P. J.,
Johnson, M., Gustafsson, P., Dahlström, K.,
Gillberg, C. J., Forssberg, H., Westerberg, H.
(2005). Computerized training of working
memory in children with ADHD-a randomized,
controlled trial. Journal of the American
Academy of Child & Adolescent Psychiatry,
44(2), 177-186.
33
Excluded at full text
Criteria 3 – Only ’Attention’ Training.
Excluded at full text
Criteria 3 – study to measure the effects of
the length of a working memory training
programme rather than its effects.
Criteria 5 – participants were aged 18-25
years old.
Excluded at abstract
Criteria 5 – participants included those with
comorbid ADHD and severe learning
difficulties.
Excluded at full text
Criteria 3 – Study aimed to compare the
difficulty level of the same Working Memory
programme.
Doctorate in Educational and Child Psychology
18. Gibson, B. S., Gondoli, D. M., Johnson, A. C.,
Steeger, C. M., Dobrzenski, B. A., & Morrissey,
R. A. (2011). Component analysis of verbal
versus spatial working memory training in
adolescents with ADHD: A randomized,
controlled trial. Child Neuropsychology, 17(6),
546-563.
19. Gropper, R. J., Gotlieb, H., Kronitz, R., &
Tannock, R. (2011). Working Memory Training
in College Students With ADHD or LD. Journal
of attention disorders, 1087054713516490.
20. Prins, P. J., Dovis, S., Ponsioen, A., Ten Brink,
E., & Van der Oord, S. (2011). Does
computerized working memory training with
game elements enhance motivation and
training efficacy in children with ADHD?
Cyberpsychology, behavior, and social
networking, 14(3), 115-122.
Excluded at full text
Criteria 3 – Study aimed to compare two
aspects of working memory within one
working memory programme.
Excluded at full text
Criteria 5 - College students aged 28 - 30
years old.
21. Ivarsson, M., & Strohmayer, S. (2010). Working
memory training improves arithmetic skills and
verbal working memory capacity in children with
ADHD.
22. Prins, P. J., Brink, E. T., Dovis, S., Ponsioen,
A., Geurts, H. M., de Vries, M., & van der Oord,
S. (2013). “Braingame Brian”: Toward an
Executive Function Training Program with
Game Elements for Children with ADHD and
Cognitive Control Problems. GAMES FOR
HEALTH: Research, Development, and Clinical
Applications, 2(1), 44-49.
34
Excluded at full text
Criteria 3 – Study aimed to measure the
difference in motivation by manipulating one
working memory training programme. Effects
of gaming elements to the programme were
compared to non-gaming elements thereby
restricting both groups from accessing the
whole programme.
Excluded at full text
Criteria 1 – Masters Thesis / not a peer
reviewed journal.
Excluded at full text
Criteria 4 - Review
Doctorate in Educational and Child Psychology
Appendix 3
Summary of Studies
Study & Aims
Programme
Green et al
(2012)
90 trials of
(COGMED
(CWMT) (2006)
To explore
whether effects of
computerized
Working Memory
Training would
generalize to “ontask” behaviour in
lessons.
www.cogmed.com
Design
Participants
Control
Measures
Outcome
Between subjects
Randomized
Pretest-postest
controlled design.
Total no. of
participants: 26
Treatment group: 12
Placebo group: 14
Mean age for
treatement: 9.9yrs
Mean age for placebo:
9.6yrs.
Total mean age: 9.7yrs
Total boys: 65%
Total girls: 35%
Placebo
(non-adaptive
working memory
training)
Timed sampling every 30
seconds of RAST
(Restricted Academic
Setting Task) at pre and
post measures using
academic worksheet l level
below functioning level for
15 mins.
nd
2 rater scored for
reliability.
Raters: Blind
Inter-rater reliability: >0.95
(with kappa)
Results were controlled for the use
of medication and medication was
not associated with improved
performance.
Tasks per day for
25 days at home.
Mean IQ for treatment:
107.0
Mean IQ for placebo:
105.4
Working Memory Index
(WMI) (from WISC IV)
assessed change in WM
performance.
Improvements at post test for
treatment group for:
“Off task” behaviours:
Mean12.3 ±4.6 SE; p=0.01
And “Plays with objects” measured
in tallies, mean score 3.4 ±1.5 SE,
p=0.04
Treatment group tallies reduced
from 9 (pre-test) to 1 (post-test).
Treatment group improved on the
WMI at pre-test with a difference of
8.4±3.3 SE, p=0.02.
CPRS-R (Connors Parent
Rating Scale) additional
measure
Klingberg et al
(2002)
To investigation
into whether a)
working memory
Computerized
training
programme for 56 weeks,
including each of
the following
Between subjects
Double-Blind
Pretest-postest
controlled design
Diagnosis of ADHD
Ages 7 – 15.
Treatment group mean
age 11.0. Control
group mean age 11.4
Treatment 2/7 on
35
Placebo (lower
dose) – 10 trials
a day and
difficulty
remained stable
with only two
Pre and Post testing
involved:
Visuo-spatial WM task;
span board; Stroop Test;
Raven’s coloured
progressive matrices
Test-retest comparisons showed
head movements significantly
decreased in treatment group
compared to control group:
Control increase of 6%; treatment
decrease of 74% (p=0.00008).
Doctorate in Educational and Child Psychology
can be improved
by training, b) the
impact would
decrease the
symptoms of
ADHD.
Hypothesising
that adaptability
would increase
WM capacity
rather than just
lead to faster
processing.
tasks for 30 trails
a day (around
25mins each
session) :
Visuo spatial task;
backwards digit
span task; letter
span task;
reaction time task.
Number of stimuli
adjusted to
increase difficulty.
medication; control 2/7
on medication.
stimuli to be
remembered.
(Raven, 1996); Choice
reaction time task
(Psychology Software
Tools, Pittsburg); head
movement task (Teicher et
al, 1996).
Statistically significant improvement
in visuo-spatial (p=0006); span
board (p=0.001); Raven’s
Progressive Matrices (p=001);
Stroop test accuracy (p=0.02). No
significance for other tasks.
Diagnosed with ADHD.
Treatment group: 18;
wait-list group: 22.
Randomised to either
group; stratified by
gender and medication
use.
Participants included
29/40: ADHD-C; 7/40:
ADHD-In; 4/40: ADHDIm; 16/40: comorb idity
with ODD
Randomized –
wait-list.
Pre and post measures
included:
Parent and teacher
questionnaires;
Post-test after 6 weeks.
And 9 weeks for wait-list
group. Followed by
ANOVA and chi-square for:
Treatment versus waitlist;
Treatment conditions pre
and post; treatment
conditions intervention
versus waitlist; intervention
within group pre and post
follow-up.
Using Cohen’s partial Eta Squared.
Treatment group showed large
effect size reduction in parent rated
ADHD symptoms and medium
effect sizes for parent rated ODD
and inattention symptoms, both
(p=<0.08). Authors report large
effect size for improvement in EF
conditions (total score) compared to
wait-list group.
Total comparisons between
medicated participants and nonmedicated showed only a
significant effect for parent rated
ODD symptoms (p=<0.05). There
was a range of medium to large
effects within this group in EF
conditions. Authors report large
effect sizes within group at followup tests.
Study conducted
at a Children’s
hospital.
Van der Oord et
al (2012)
To measure the
effectiveness of
an Executive
Function
Training
Programme using
gaming elements
on cognitive
flexibility,
response
inhibition and
WM capacity.
Training of 3
Executive
Functions: WM;
inhibition and
cognitive flexibility
using
“Brainggame
Brian” (Prins et al,
2010).
25 sessions of 40
mins each.
Difficulty level
adjusted.
Participants given
computers for use
at home.
Between subjects
Randomized
Pretest-postest
controlled design
36
Doctorate in Educational and Child Psychology
Chacko et al
(2013)
COGMED
(CWMT) (2006)
www.cogmed.com
To replicate a
previous studies
(Green et al,
2012; Klinberg et
al, 2005) and
measure the
efficacy of a WM
programme using
a more diverse
range of
participants with
ADHD of WM;
ADHD symptoms
and academic
achievement.
30-45mins, 1 x
day for 25 days.
Adapted to level
of ability as the
training
progresses.
Compliance –
through
completion of
20/25 sessions,
either compliant
or non-compliant.
Active training
time monitored
and adjusted
accordingly.
Between subjects
Randomized
Pretest-postest
controlled design
85 children ranging
from between 7-11
years recruited
through community
advertisements.
Mean age for both
groups 8.4 years.
Treatment Group: 44
Placebo group: 41
Placebo version
of CWMT
including no
adaptation for
difficulty level;
reduced number
of trials per
session and
alternative
games.
53 with ADHD-C; 32
with ADHD-In.
66/85 males evenly
distributed between
groups.
Mean IQ: 104
Both groups matched
for characteristics and
demographic profiles.
Training took
place at home.
DBD Rating Scale (Pelham
et al 1992) used to
measure ADHD symptoms.
Parents and Teachers on 4
point scale.
AWMA (Alloway, 2007)
used to measure WM
capacity.
CWMT improvement index
score, baseline compared
to top two maximum
performance scores).
Improvement index score =
≥17.
Movement was monitored
using actigraphs on ankles
and waists. (Reichenbach
et al 1992).
Impulsivity and inattention
measured by A-X
Continuous Performance
Test (CPT), (Halperin et al.
1991).
Academic achievement
measured by WRAT4-PMV
(Roid & Ledbetter, 2006)
Assessors blind to
treatment randomization.
37
Intent-To-Treat approach to
compare treatment effects.
Compliance:
Treatment: 35/44 met compliance
standard;
Placebo: 31/41 met compliance
standard.
Notable difference in time spent
training between treatment and
placebo (p=<0.0001)
Treatment mean39.1mins
compared to Placebo mean 26
mins, lower than minimum
recommended: 30 mins.
ADHD symptoms:
No significant main effect for
diminution of symptoms from
teacher ratings; no significant main
effects between groups for parent
ratings.
Significant differences for Working
Memory were reported as follows:
between treatment conditions for
non –verbal storaage/ dot matrix
tasks (d=1.17),(p=<0.00009) and
verbal storage/digital recall
(d=0.28), (p=<0.0050); but no
significant effects for verbal or nonverbal storage, manipulation and
processing (complex working
memory function).
No significant effects were found for
assessments on attention and
impulse control.
Doctorate in Educational and Child Psychology
There was a significant effect of
time for both but not on separate
aspects of academic achievement.
Holmes et al
(2010)
COGMED (2006)
(CWMT)
www.cogmed.com
To compare the
impacts of
specific WM
training and drug
treatments on
Working Memory
in children with
ADHD.
Computer training
school-based
intervention.
With medication
pretest measured
without
medication.
Programme
Administered by
TA.
Non-randomized
one group
pretest-posttest
design.
25 children aged
between 8-11 years
(mean 9.9 years); with
a diagnosis of ADHD;
recruited through
psychiatrists and
paediatricians;
Medication
alone.
AWMA was used to assess
verbal STM; visuo-spatial
STM; verbal WM; visuospatial WM.
Standard and composite
scores were taken.
Medication is able to improve
working memory function but not to
the extent of medication and
working memory training.
Medication showed significant
interaction between visuo-spatial
elements of WM only.
Working Memory Training led to
significant effects in more WM
components : visuo-spatial STM;
verbal WM and visuo-spatial WM.
20-25 sessions
over 6-10 weeks
Key:
WM – Working Memory; statistical significance: p=<0.05; ADHD-C = Combined subtype; ADHD-In = Inattentive subtype; ADHD-Im = Impulsive subtype; ODD
= Oppositional Defiance Disorder; EF = Executive Function; CWMT = COGMED Working Memory Training; DBD = Disruptive Behaviour Disorders Rating
Scale; AWMA = Automatic Working Memory Assessment; WRAT4-PMV = Wide Range Achievement Test Progress Monitoring Version; STM = short term
memory; TA = Teaching Assistant;
38
Doctorate in Educational and Child Psychology
Appendix 4
Coding Protocol
University College London, Educational Psychology, Literature Review Coding Protocol
adapted from Task Force on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill (2003).
Name of Coder:
Date: 21.2.14
Full Study Reference in proper format:
Holmes, J., Gathercole, S. E., Place, M., Dunning, D L., Hilton, K. A. & Elliott, J. G. (2010).
Working memory deficits can be overcome: Impacts of training and medication on working
memory in children with ADHD. Applied Cognitive Psychology. Vol.24(6), Sep 2010, pp.
827-836.
Intervention Name: Working Memory Training
Study ID Number: 1
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
This was a non-randomised one group pretest-posttest design.
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
39
Doctorate in Educational and Child Psychology
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 25
Intervention group sample size:_______
Control group sample size:________
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
40
Doctorate in Educational and Child Psychology
AWMA and WASI used to compare conditions (with medication and without medication and
training)
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
Same group tested at pretest in two conditions (with medication and without medication).
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
41
Doctorate in Educational and Child Psychology
Statistical matching
Post hoc test for group equivalence
Adjusted scores due to one or more covariates.
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence : 2
3= Strong Evidence / 2=Promising Evidence / 1=Weak Evidence / 0=No Evidence
C Appropriate Statistical Analysis
Analysis 1 : WMT found to significantly improve all aspects of WM measured,
compared to only Visuo-spatial on medication alone.
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Expectations for a medium effect size for correlational analysis estimated that sample size
was under powered.
No - Minimum of 46 participants based on a median effect size of 0.66
Overall Rating of Analysis:
3
2
1
0
42
Doctorate in Educational and Child Psychology
Coding Protocol
University College London, Educational Psychology, Literature Review Coding Protocol
adapted from Task Force on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill (2003).
Name of Coder:
Date: 21.2.14
Full Study Reference in proper format:
Chacko, A., Bedard, A. C., Marks, D. J., Feirsen, N., Uderman, J. Z., Chimiklis, A., Rajwan,
E., Cornwell, M., Anderson, L., Zwilling, A., & Ramon, M. (2013). A randomized clinical trial
of Cogmed Working Memory Training in school‐age children with ADHD: a replication in a
diverse sample using a control condition. Journal of Child Psychology and Psychiatry, doi:
10.1111/jcpp.12146.
Intervention Name:
Study ID Number:
Cogmed Working Memory Training
2
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
43
Doctorate in Educational and Child Psychology
Unknown/unable to code
B Participants
Total size of sample (start of study):
85
Intervention group sample size:
44
Control group sample size:
41
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
However, 25 on medication but details of dose, individual effect are not known.
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
However, for Working Memory alone, only on method used.
44
Doctorate in Educational and Child Psychology
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
As mentioned above, in general at least two sources used but not for Working Memory
measures.
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
Due to the fact that Working Memory measures were being examined in the review only.
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
Time on the training programme was adapted to even out and compare between
groups.
B4 Group equivalence established
45
Doctorate in Educational and Child Psychology
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence : 3
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
C Appropriate Statistical Analysis
Analysis 1 - Working Memory was measured using AWMA (Alloway, 2007)
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
No - Minimum of 390 participants based on a median effect size of 0.285
Overall Rating of Analysis:
3
2
1
0
46
Doctorate in Educational and Child Psychology
Coding Protocol
University College London, Educational Psychology, Literature Review Coding Protocol
adapted from Task Force on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill (2003).
Name of Coder:
Date: 21.2.14
Full Study Reference in proper format:
Van der Oord, S., Ponsioen, A. J. G. B., Geurts, H. M., Ten Brink, E. L., & Prins, P. J. M.
(2012). A pilot study of the efficacy of a computerized executive functioning remediation
training with game elements for children with ADHD in an outpatient setting: outcome on
parent-and teacher-rated executive functioning and ADHD behavior. Journal of attention
disorders, xx(x) 1-14, doi: 10.1177/1087054712453167.
Intervention Name:
“Braingame Brian”
Study ID Number: 3
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
47
Doctorate in Educational and Child Psychology
Unknown/unable to code
B Participants
Total size of sample (start of study):
43
Intervention group sample size:
21
Control group sample size:
22
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
However, not for Working Memory alone.
48
Doctorate in Educational and Child Psychology
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
Unknown/unable to code
However, not for Working Memory alone.
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
Due to measures for Working Memory alone only using one element of BRIEF.
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
Where items were missing in analysis, item was replaced by the mean of the other
items in the subscale, if more than one item missing, the subscale was not used.
49
Doctorate in Educational and Child Psychology
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Report states that missing items in analysis were replaced by the mean of other
items in the subscale and that if more than one item missing, subscale was not used.
Report also indicated that those who did not complete 20/25 sessions were not used. This
implies that there may be some attrition bias (Wright et al, 2007).
Overall Level of Evidence
2
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
C Appropriate Statistical Analysis
Analysis 1 - ANOVA
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Expectations of a medium effect size due to some confounding variables, indicating that 43
participants may be under powered.
Analysis
2__________________________________________________________________
__________
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Analysis
3__________________________________________________________________
__________
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
No - Minimum of 350 participants based on a median effect size of 0.05
Overall Rating of Analysis:
3
2
1
0
50
Doctorate in Educational and Child Psychology
Coding Protocol
University College London, Educational Psychology, Literature Review Coding Protocol
adapted from Task Force on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill (2003).
Name of Coder:
Date: 21.2.14
Full Study Reference in proper format:
Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Training of working memory in
children with ADHD. Journal of clinical and experimental neuropsychology, 24(6), 781-791.
Intervention Name:
Study ID Number:
Computerized Working Memory Training Programme
4
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
Although not specifically stated except that the design was a “double-blind, placebo,
controlled design” and that there were no significant differences between groups.
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
51
Doctorate in Educational and Child Psychology
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): 14
Intervention group sample size:
7
Control group sample size:
7
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
52
Doctorate in Educational and Child Psychology
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
Not enough information to be able to judge. Despite reporting no difference between group,
method of measurement is not indicated.
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
53
Doctorate in Educational and Child Psychology
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence - 1
3= Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence
C Appropriate Statistical Analysis
Analysis 1 Correlational Analysis
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Expectation for effect size is medium (64) indicating that sample size is greatly under
powered.
No - Minimum of 38 participants based on a median effect size of 0.95
Overall Rating of Analysis:
3
2
1
0
54
Doctorate in Educational and Child Psychology
Coding Protocol
University College London, Educational Psychology, Literature Review Coding Protocol
adapted from Task Force on Evidence-Based Interventions in School Psychology, American
Psychology Association, Kratochwill (2003).
Name of Coder:
Date: 21.2.14
Full Study Reference in proper format:
Green, C. T., Long, D. L., Green, D., Iosif, A. M., Dixon, J. F., Miller, M. R., Fassbender, C.,
& Schweitzer, J. B. (2012). Will working memory training generalize to improve off-task
behavior in children with attention-deficit/hyperactivity disorder? Neurotherapeutics, 9(3),
639-648.
Intervention Name:
Cogmed Working Memory Training (CWMT)
Study ID Number: 5
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the
following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
55
Doctorate in Educational and Child Psychology
B Participants
Total size of sample (start of study):
26
Intervention group sample size:
12
Control group sample size:
14
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No
Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary
outcomes (see following table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
Yes
No
N/A
56
Doctorate in Educational and Child Psychology
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are
considered good, consider any cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
B3 Counterbalancing of change agent (participants who receive intervention from a single
therapist/teacher etc were counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence - 2
57
Doctorate in Educational and Child Psychology
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
0=No Evidence
C Appropriate Statistical Analysis
Analysis 1 -
Generalised Linear Model = RAST
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Analysis 2 - WISC for WMI (Working Memory Index)
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Analysis 3 - Connors Rating Scale
Appropriate unit of analysis
Familywise/expermenter wise error rate controlled when applicable
Sufficiently large N
Yes - Minimum of 22 participants needed based on an effect size of 1.3
Overall Rating of Analysis:
3
2
1
0
58
Doctorate in Educational and Child Psychology
Appendix 5
Weighting of Studies
A: Methodological Quality
The Kratochwill (2003) coding protocol was used to weight designs; studies were
weighted on measures; comparison groups and analysis.
1. Measures
Weighting
High
Medium
Low
Description
Used 2 measures which have reliability of .85 or above (reported in text).
Data collected using multiple methods and multiple sources.
Validity is reported.
Used measures that produce reliable scores of .70 for the population under
study
Data collected using either multiple methods AND/OR from multiple sources.
A case for validity does not need to be presented.
Uses measures that produce reliable scores of 0.50 for the population under
study
2. Comparison Group
Weighting
High
Medium
Low
Description
Uses at least 1 type of ‘active’ comparison group
Initial group equivalency is reported
Evidence of counterbalancing
Equal mortality rate between the intervention and comparison group (of under
20% attrition)
Uses at least a ‘no intervention group’ comparison group.
Equivalent mortality with low attrition (under 20%).
If equivalent mortality is not demonstrated, conducted an intent-to-intervene
analysis.
Uses a comparison group.
Demonstrated 1 of the following: 1. Counterbalancing of change agents, 2.
Group equivalency, 3. Equivalent mortality with low attrition (under 20%).
If equivalent mortality is not demonstrated, conducted an intent-to-intervene
analysis.
59
Doctorate in Educational and Child Psychology
3. Analysis
Weighting
High
Medium
Low
Description
1. Conducted an appropriate statistical analysis
2. Corrected for family wise / experimenter error (if appropriate)
3. Used a sufficiently large sample size
4. Provided enough information for the effect size to be calculated
Demonstrated two of the following:
1. Conducted an appropriate statistical analysis
2. Corrected for family wise / experimenter error (if appropriate)
3. Used a sufficiently large sample size
4. Provided enough information for the effect size to be calculated
Demonstrated one of the following:
1. Conducted an appropriate statistical analysis
3. Used a sufficiently large sample size
4. Provided enough information for the effect size to be calculated
Overall Methodological Quality
The following scores were assigned to calculate the overall methodological quality of
studies:
High weightings = 3
Medium weightings = 2
Low weightings = 1
Studies which did not meet criteria = 0
Scores were then averaged.
Overall Methodological
Quality
Average Scores
High
Medium
Low
At least 2.5
Between 1.5 and 2.4
Less than 1.4
60
Doctorate in Educational and Child Psychology
Studies
Measures
Comparison
Group
Analysis
Green, T. C., Long, D. L.,
Green, D., Iosit, A., Dixon, F. J.,
Miller, M. R., Fassbender, C &
Schweitzer. (2012).
Medium
Score = 2
Medium
Score = 2
Medium
Score = 2
Medium
2
Medium
Score = 2
Low
Score = 1
Low
Score = 1
Low
1.33
Van der Oord, S., Poinsioen, A.
J. G. B., Guerts, H. M., Ten
Brink, E. L. & Prins, P. J. M.
(2012).
Medium
Score = 2
Medium
Score = 2
Medium
Score = 2
Medium
2
Chacko, A., Bedard, A. C.,
Marks, D. J., Feirsen, N.,
Uderman, J. Z., Chimiklis, A.,
…& Ramon, M. (2013).
Medium
Score = 2
High
Score = 3
Medium
Score = 2
Medium
2.33
Holmes, J., Gathercole, S. E.,
Place, M., Dunning, D. L., Hilton,
K. A. & Elliot, J. G. (2010)
Medium
Score = 2
Medium
Score = 2
Medium
Score = 2
Medium
2
Klingberg, T., Forssberg, H. &
Westerberg, H. (2002)
Overall Quality
of Methodology
(B) Relevance of Methodology
This weighting reviews the appropriateness of the evidence for answering the review
question.
Weighting
High
Medium
Low
Description
Uses random assignment of participants.
Uses an active comparison group.
Takes pre and post measures for both the intervention
and control group.
Uses a comparison group.
Takes pre and post measures for both the intervention
and control group.
Takes pre and post measures.
61
Doctorate in Educational and Child Psychology
(C) Relevance of Evidence to Review Question
This weighting is a review-specific judgement about the relevance of the focus of the
evidence for the review question.
Weighting
High
Medium
Low
Description
Uses Working Memory Training exclusively
Measures working memory improvement pre and post
intervention
Complies with special population (ADHD)
Medication use is acknowledged and controlled for.
Uses Working Memory Training exclusively.
Some measures of working memory improvement pre and
post intervention are available.
Complies with special population (ADHD).
Some measures of working memory improvement pre and
post intervention are available.
Complies with special population (ADHD).
Working Memory Training is included as an intervention with
other interventions.
(D) Overall Weight of Evidence
This is an overall assessment of the extent to which the study contributes evidence
to answer the review question. It is calculated by assigning the following scores:
High weightings = 3
Medium weightings = 2
Low weightings = 1
Studies which did not meet criteria = 0
The scores were then averaged and the following weighting of evidence was
awarded for each score:
High weighting = More than 2.5
Medium weighting = Between 1.5 and 2.4
Low weighting = Less than 1.4
62
Doctorate in Educational and Child Psychology
Studies
Green, T. C., Long, D. L.,
Green, D., Iosit, A., Dixon,
F. J., Miller, M. R.,
Fassbender, C &
Schweitzer. (2012).
Klingberg, T., Forssberg,
H. & Westerberg, H.
(2002)
Van der Oord, S.,
Poinsioen, A. J. G. B.,
Guerts, H. M., Ten Brink,
E. L. & Prins, P. J. M.
(2012).
Chacko, A., Bedard, A.
C., Marks, D. J., Feirsen,
N., Uderman, J. Z.,
Chimiklis, A.,… & Ramon,
M. (2013).
Holmes, J., Gathercole, S.
E., Place, M., Dunning, D.
L., Hilton, K. A. & Elliot, J.
G. (2010)
(A)
Quality of
Methodology
(B)
Relevance of
Methodology
(C)
Relevance of
evidence to
the review
question
(D)
Overall Weight of
Evidence
Medium
Score = 2
High
Score = 3
High
Score = 3
Medium
2.33
Low
Score = 1
Low
Score = 1
Medium
Score = 2
Low
1.37
Medium
Score = 2
High
Score = 3
High
Score = 3
High
2.66
Medium
Score = 2
High
Score = 3
Medium
Score = 2
Medium
2.33
Medium
Score = 2
Medium
Score = 2
High
Score = 3
Medium
2.33
63