Doctorate in Educational and Child Psychology
Isabella McDonald
Case Study 1: An Evidence-Based Practice Review Report
Theme: Interventions for children with Special Educational Needs
Humans vs. Robots: What is the effectiveness of a robot versus a
human-led intervention on the social interaction skills of children with
ASD?
Summary
Autism spectrum disorders (ASD) are psychological conditions
associated with impairments in social interaction as a primary characteristic
(American Psychiatric Association, 2000). Children with ASD have been shown
to have an affinity with computer-based technologies (Hart, 2005). The use of
robots in intervention and therapy for children with ASD is in its preliminary and
exploratory stage, with many studies exploring the technical possibilities of using
robots as a tool for child interaction.
This literature review aims to evaluate the effectiveness of robot
interventions compared to human-led interventions. The studies in this review
aimed to improve a target social behaviour such as shared attention, selfinitiated interaction and imitative behaviours. Five studies met the inclusion
criteria for children with ASD or related disorders such as Asperger’s and
Pervasive Developmental Disorder Not Otherwise Specified (PDD-NOS). The
age range was between two and 14 years. Findings show variability in children’s
reactions to a robot, with some studies showing a larger effect size when a robot
is used compared to a human.
1
Doctorate in Educational and Child Psychology
Isabella McDonald
Introduction
Robot intervention
As technology develops ever faster and further, there has been an
increasing interest in the application of robots in diagnosis, research and
therapy of children with autism spectrum disorders (ASD) and related difficulties.
As a result, a number of robots have been designed with children with autism in
mind with different capabilities, functions and specifications (Cabibihan, Javed,
Ang & Aljunied, 2013). The application of robots in therapy has received
considerable media attention, most recently in the UK (Criado, 2014). However,
efficacy and effectiveness research on this topic is in its infancy (Diehl, Schmitt,
Villano, & Crowell, 2012).
Much research has studied the feasibility and functioning of humanoid or
non-humanoid robots for use in research and therapy (Rick & Colton, 2010)
which becomes clear when searching databases for studies, but as Diehl et al.,
(2012) states, not as much emphasis has been placed on the clinical application
of (social) robots for children with autism. In the Diehl et al. (2012) review of the
clinical use of robots, the application of interactive robots is categorised in four
ways: responses to robots or robot-like characteristics, eliciting behaviour,
modelling, teaching, or practising skills and providing feedback or
encouragement.
Psychological Basis: Robots and ASD
Socially interactive robots are used to communicate, express emotion
and maintain social relationships (Fong, Nourbakhsh & Dautenhahn 2003),
2
Doctorate in Educational and Child Psychology
Isabella McDonald
2003). These are areas of social and communication that children with ASD find
challenging (Figure 1).
Figure 1. Triad of impairments in Autism Spectrum Disorder (Adapted
from Cabibihan et al., 2013).
Social and communication impairments associated with autism are
deficits in imitating others (Williams, Whiten & Singh, 2004), self-initiations
(Koegel, Camarata, Valdez-Menchaca & Koegel 1998) and reading others’
expressions (Celani, Battacchi, & Arcidiacono, 1999). These social skills require
a degree good theory of mind and empathising with others – an ability to
3
Doctorate in Educational and Child Psychology
Isabella McDonald
imagine other’s thoughts, feelings, and intentions and to try to predict their
behaviours (Baron-Cohen, 1995). The mind-blindedness theory proposes that
people with autism lack this ability.
Further to this, Baron-Cohen (2009) uses the cognitive theory, the
systematising-empathising theory to explain the social and communication
deficits that children with autism display. This theory recognises the discrepancy
between an area of difficulty, the ability to empathise with others, and an area of
strength, systematising, found in people with autism (Baron-Cohen, 2002).
Baron-Cohen (2009) suggests that this gap determines whether a person is
likely to develop autism.
Systematising is a drive to analyse a system which is guided by rules.
Baron-Cohen (2009) suggests that by doing so people with autism use this
superior skill to predict how a system may behave. Evidence for this theory is
derived which have used a scale, the Systematising Quotient (SQ) which
evaluates a “drive to systematise”. For example, people with autism scored
higher on the scale than the general population (Baron-Cohen, Richler, Bisarya,
Gurunathan, & Wheelwright, 2003).
The systematising-empathising theory may be used to explain the
psychological theory behind the clinical application of robots. It has been
suggested that children with autism have a greater affinity to robots and
mechanical components than humans (Hart, 2005). There is even some
evidence that children with ASD prefer robots to humans (Dautenhahn & Werry,
2004). Robots can be understood as a system which children with autism may
be motivated to engage with. As Klin, Lin, Gorrindo, Ramsay and Jones (2009)
4
Doctorate in Educational and Child Psychology
Isabella McDonald
outlines, a strength associated with ASD is an understanding of objects.
Children with ASD are intrinsically interested in treatment carried out by
technology (robotic or electronic elements) (Robins, Dautenhahn & Dubowski,
2006). These factors may all prove beneficial in the clinical application of robots
for practising and promoting social engagement skills.
It must be mentioned that the autistic spectrum covers a wide range of
difficulties and this review includes broad samples within the target population.
Not every child with ASD, Asperger’s Syndrome or PDD-NOS will show all or
many of these impairments to the same degree (Happé & Ronald, 2008; Wall,
2009). Each child will exhibit a certain set of the symptoms of varying degrees
and intensities. Autism is defined as a spectrum disorder and typically
developing children may also demonstrate some of the impairments as shown in
Figure 1 (Constantino and Todd, 2003).The reviewed studies use a robot and a
human-led intervention to promote the following areas of social behaviour:
imitation, eye contact or gaze, emotional expression, joint attention and selfinitiated behaviour. All of which play significant roles in social development.
A key role of an educational psychologist is supporting the education of
children with special educational needs such as those with autism. The skills of
communicating and interacting with others are heavily relied upon in the
mainstream school environment. The application of robots may be a method in
which to improve the social and communication skills of these children in
schools. Currently, the use of robots in the classroom is being trialled in a
primary school in (Griffiths, 2014). An example of the robot’s use in school is
cited in the article. It tells the story of a boy named Stephen with Asperger’s
Syndrome who works with the robot, Nao, after lunch. Stephen recites a story
5
Doctorate in Educational and Child Psychology
Isabella McDonald
he has written to the robot and the robot will react to the story at moments of
interest. Teachers report that the children will often follow instructions given by
the robot when they may be reluctant to respond to an adult. The trial is being
implemented by the University of Birmingham’s Autism Centre for Education and
Research (Acer) and educational psychologists have been following the
project’s progress. Should this trial and other similar projects be successful,
there may be a future role for psychologists in supporting the programming of
the robots to suit the skills and development of the children, or in training and
research.
Review Question
In light of the above, the review question reads: What is the effectiveness of a
robot versus a human-led intervention on the social interaction skills of children
with ASD?
Critical Review of the Evidence base
Literature Search
A literature search was carried out in January 2014 using the databases
ERIC, Medline, PsychInfo and Web of Science. In order to be included in the
review, the article had to be published in a peer-review journal between 2008
and 2013. An initial search using the terms “robot*” in the journal abstract and
autism or ASD or Asperger* or PDD-NOS in the journal title returned 139
results, of which many were duplicates. Many studies focussed on effective
robot design, rather than the effect of the robot intervention itself. Diehl et al.
(2012) also reported that much of the published research is in journals that
6
Doctorate in Educational and Child Psychology
Isabella McDonald
focus on robotics (e.g., Autonomous Robots, Robotica) rather than in prominent
ASD journals or clinically focused journals. The search terms were therefore
narrowed down, as below, and restricted to the past five years (2008 – 2013) in
order to review the most recent technologies, which returned a total of 37
results. The abstracts of these 37 were inspected and 19 (excluding duplicates)
were rejected based on the rationale for exclusion and inclusion. For a full
account of rationale for exclusion of these studies and a flowchart documenting
this process, please see Appendix 1.
Table 1. Search terms
Difficulty
Intervention
Measure
in Title
in Abstract
In Abstract
Autism
Robot*
Socia*
ASD
Asperger*
PDD-NOS
Including and excluding studies using criteria
The criteria shown in Table 2 were used as a reference to obtain the
included five papers.
Table 2. Inclusion and exclusion criteria.
Criterion
1. Publication type
Inclusion
The study is
published in a peer reviewed
journal to ensure some
quality control, as these have
been assessed using
7
Exclusion
The study is not
published in a peer-reviewed
journal. This excludes
conference papers and
thesis research.
Doctorate in Educational and Child Psychology
Isabella McDonald
similarly stringent criteria.
2. Publication year
The year of
publication is between a five
year period, 2008 and 2013.
This is to capture the most
recent technological
advances.
The year of
publication is outside of this
timeframe.
3. Language of
The study is
published in English. This
criterion has been drawn due
to the lack of resources
available for translation.
The study is
published in a language
other than English.
4. Methodology
The study has a
human – child interaction
condition and a robot – child
interaction condition.
The study does not
have a both a human – child
interaction condition and a
robot - child interaction
condition.
5. Intervention
The study includes
an intervention targeted at
improving participants’ social
engagement skills.
The study does not
have an intervention targeted
improving participants’ social
engagement skills.
6. Participant need
The study must
confirm a previous diagnosis
of ASD (DSM-IV A.P.
Association, 2000), for
participants described as
autistic. This diagnosis may
be given as in the Diagnostic
Manual, ADOS (Autism
Diagnostic Observation
Schedule) or confirmed by a
medical professional.
The study does not
confirm a diagnosis of ASD
for participants described as
autistic.
7. Participant age
The study includes
children and young people
between the ages of 0 and
25. This is the age group with
which educational
psychologists work.
The study samples
participants outside of this
age range.
publication
8
Doctorate in Educational and Child Psychology
Isabella McDonald
Table 3. Studies meeting the inclusion criteria and summary
Study
Sample
Intervention
Design
Measures
Outcomes
Duquette et al.
(2008)
N=4
low
functioning
autism, 4 – 5 years,
pre-verbal and nonverbal.
Encouraged interaction
with a robot or human
Single case design
Observation.
Shared attention, Shared
conventions, Absence of
sharing,
Other
phenomenon.
Huskens
al., (2013)
et
N=6
ASD according to
DSM-IV TR criteria.
8 – 14 years, IQ 80
and above
ABA-based
intervention to promote
self-initiated question
asking
Single case design
Kim
et
(2012)
al.
N = 24
4 -12 years, high
functioning ASD, IQ
at least 70
Group design with
randomised
conditions
Tapus et al.
(2012)
N=4
2 – 6 years
Moderate
autism,
some with delayed
cognitive
and
language
development
Social behaviours in
three conditions: Pleo
dinosaur robot, human
or
touchscreen
computer.
Encouraged imitation
of a Nao robot or
human trainer
Observation.
Statement
Syntactically
correct
question
that implies an action
Syntactically
correct
question
that does not imply an
action
Incorrect response
Observation
Utterances
towards
mediator or confederate
The results indicate that non-verbal children
demonstrate less interest and participation than
pre-verbal children. Children exposed to the robotic
mediator showed reduced repetitive plays with
inanimate objects of interest (their favourite toy),
and no repetitive or stereotyped behaviour toward
the robot.
For both experimental groups, the number of selfinitiated questions increased significantly between
the first baseline and the first intervention,
indicating that the interventions conducted by a
robot and a human trainer were both effective in
promoting self-initiated questions in children with
ASD.
Single case design
Observation
Free initiations, initiations
with prompt, eye gaze,
gaze shifting, smile/laugher
Results are mixed and show variability in reaction to
robot.
Vanderborght
et al. (2012)
N=4
4 - 8 years
,elements
of
severe autism
Social story telling by a
robot (Probo) or a
human
Single case design
Observation
Level of prompting needed
to
perform
a
social
behaviour
Compared to the Social Story intervention,
the Social Story with RAT intervention had a
stronger effect on decreasing the level of prompting
the therapist needed to offer to each child in order
for the child to perform the target behaviour for all
the participants.
to
9
Children with ASD spoke more while interacting
with a social robot than with another adult or a
novel, touchscreen computer game.
Doctorate in Educational and Child Psychology
Isabella McDonald
Comparison of Selected Studies
Evaluation process
The methodological qualities of the studies were examined and coded
using the UCL Educational Psychology Literature Review Coding Protocol,
adapted from APA Task Force on Evidence Based Interventions in School
Psychology (Kratochwill and Shernoff, 2003). The Gough (2007) framework as
outlined in Appendix 3 was used to assess the applicability of each study to the
research question. Full evaluations of the studies reviewed can be obtained in
Appendix 3.
Table 4. Weight of evidence.
Weight of Evidence Rating
Methodologi
Methodolo
cal Quality
gical Relevance
(A)
(B)
Study
Duquette
et al. (2008)
Huskens
et al. (2013)
Kim et al.
(2012)
Tapus et
al.
(2012)
Vanderbo
rght et al. (2012)
Review
Relevance
(C)
Overall
(D)
Low
1.3
Medium
1.6
Medium
2
High
2.6
Low
1
High
3
High
3
Low
1
Medium
2
High
3
Low
1
Medium
2
Low
1.4
High
2.5
Medium
2
Medium
1.9
Medium
2.3
Medium
2
High
3
Medium
2.4
Participants
The age of participants in all studies ranged from two to 14 years of age.
Children with ASD cover a broad spectrum of strengths and difficulties. For the
purpose of the review, it is important to mention the degree of the participants’
difficulties. In all five studies, children were diagnosed according to the
10
Doctorate in Educational and Child Psychology
Isabella McDonald
Diagnostic and Statistical Manual of Disorders IV criteria of autism (APA, 2000)
or by the ADOS-G (Autism Diagnostic Observation Schedule-Generic), as
specified in the inclusion criteria. One study, Kim et al. (2012), included children
with high functioning autism as the only research sample. In contrast, Duquette
et al. (2008) only included a sample of non-verbal and pre-verbal children with
low functioning autism. These low functioning samples must be borne in mind
when interpreting the small effect sizes associated with the robot intervention
conditions in this study. Vanderborght et al. (2012) sampled children with
moderate autism and found large effect sized across the board.
Two studies (Huskens et al., 2013 and Kim et al., 2012) controlled for IQ,
a full-scale IQ above 80 and equal to or above 70, respectively. Others reported
the ranges of IQ in their sample, but did not report IQ in the inclusion criteria for
participation.
Kim et al. (2012) is the only study to report the cultural background of the
24 participants, with the majority of white origin. The study was given a rating of
low in Weight of Evidence C as social behaviour deficits were not described.
Ascertaining the social behaviour deficitis may however be more difficult in a
larger sample size in group study such as that by Kim et al. (2012). It could be
argued that deficits in social behaviour deficit is implied by the confirmed
diagnosis using the ADOS and delivered by an experienced clinician. Weight of
evidence C specifies that the study must give clear descriptors’ of the children’s
diagnosis and social skills or deficits. The four studies with a single case design,
excluding Kim et al. (2012), were more appropriate for this purpose as they gave
background information on the difficulties of each child participant.
11
Doctorate in Educational and Child Psychology
Isabella McDonald
Duquette et al. (2008) sampled pre-verbal and non-verbal children who
show a deficit in imitation, also a severe delay in receptive and expressive
language and have a sensori-motor interest that interferes with the development
of their communication skills. They were all assessed using the imitation tool
PEP-R (Psycho Educational Profile Review) (Schopler, Reichler, Bashford,
Lansing & Marcus, 1990). Tapus et al. (2012) studied four children as part of a
single case study design, three of whom have cognitive deficits and all of whom
have severely delayed language development and a diagnosis of autism from
mild to severe. The study, which uses an imitation task to investigate whether
children with autism show more social engagement when interacting with a Nao
robot compared to a human partner, details the children’s ability to socially
interact and imitate others. For the purpose of the review question, this study
has a highly relevant sample of children with autism to target a specific social
deficit using both a robot and a human trainer. Huskens et al. (2013) had a
sample of six male participants whose diagnoses were confirmed using a Social
Communication Questionnaire (SCQ). This partially accounted for a Medium
score for Weigh of Evidence C. The rating was brought down by the lack of
documentation of the relationship between the child and the researcher.
It is also important to consider the relationship between the children
receiving the intervention and the person or robot delivering the intervention.
This is especially the case when studying children with ASD who may not feel
as comfortable around unfamiliar adults as typically developing children.
Vanderborght et al. (2013) had a highly relevant and gender balanced sample of
two boys and two girls who attended a therapy centre. All four children between
the ages of four and nine had a diagnosis of moderate autism. The study clearly
12
Doctorate in Educational and Child Psychology
Isabella McDonald
states that the children were familiar with the room in which the experiment took
place. Additionally, the therapist who performed the interaction was a permanent
staff member of the autism centre and had given regular therapy sessions to the
children involved. Huskens et al. (2013) indicates that a room that is familiar to
the children as it is part of their day treatment centre. It is not clear whether the
children are therefore known to the researcher, or the children have had prior
exposure to the robot, Nao. Included in the design of the intervention conditions,
Tapus et al. (2012) incorporated a familiarisation stage with the robot. This
lasted five to ten minutes depending on the child’s reaction to the robot. The
child was encouraged to interact with the robot by saying hello and asking its
name. The experimenter also modelled what the robot was able to do. These
phases show an understanding of the needs of a child with autism who may find
this unfamiliar situation and technology uncomfortable. Once the child was at
ease with the robot, the intervention phase began.
Robots
Each robot in the reviewed studies has its unique appearance and
interaction methods. The design and functionalities of the robot have a
significant influence on its effectiveness in eliciting social behaviour. Four
studies used a humanoid robot which has more human-like characteristics. Two
of these (Huskens et al., 2013 and Tapus et al., 2013) used the same humanoid
robot, Nao, developed by Aldebaran Robotics. Kim et al. (2012) used a robot
described as an affectively expressively toy dinosaur robot named Pleo.
As shown in Table 5, all robots had differing technical capabilities. It is
possible that these capabilities had an effect on the outcomes of the studies and
13
Doctorate in Educational and Child Psychology
Isabella McDonald
the extent to which the robots were effective in socially engaging the children.
For example, Tito, the humanoid robot used in Duquette et al.’s study lacks
reciprocity, whereas Nao does not.
Table 5. Robots used to deliver the intervention.
Study
Robot (with picture)
Duquette
et al. (2008)
Tito
Huskens
et al. (2013)
Nao
Kim et al.
Pleo
Robot functions and characteristics

60 cm tall

plainly coloured (red,
yellow, blue)

Has feet and legs but uses
wheels to move

Moves arms, head, and
mouth

Expresses a few simple
emotions

Camera and microphone

Generates vocal
messages

Different body parts can be
illuminated

Vocabulary of 25 words

neutral-coloured humanoid
robot

57cm tall

25 mechanical degrees of
freedom

digital cameras,
microphones, speakers, touch sensors

has a ‘‘simple’’ face to
prevent confusion

female pre-recorded voice
was used
 21 inches long, 6 inches wide, and
(2012)
8
inches tall
 Green and brown
 customised for the purpose of the
study
 Movement are synchronised with
pseudo-verbal vocalisations, to express
14
Doctorate in Educational and Child Psychology
Isabella McDonald
interest, disinterest, happiness,
disappointment, agreement and
disagreement
Tapus et
Nao
As above
al.
(2012)
Vanderb
orght et al.
(2012)

Probo
expresses attention and emotions
via
gaze and facial expressions

human-like social cues

soft, “huggable” material

green
Intervention
Treatment integrity was assessed in one study, on 33% of the sessions
for each child with a mean of 99% (Huskens et al., 2013). The integrity by which
the treatment or intervention was carried out in the remaining four studies could
not be directly assessed, however Vanderborght et al. (2012) and Tapus et al.
(2012) in much lengthier reports give very clear guidelines on how, where and
15
Doctorate in Educational and Child Psychology
Isabella McDonald
with whom the interventions were carried out. These descriptions make the
studies replicable to a high degree.
Measures
All studies used observation as a method of data collection, but slightly
varied in approach and coding. Experimental scenarios were videotaped and
coded by at least two observers. In Tapus et al. (2012), this was by three trained
clinical psychologists. Inter-coder reliability ratings were highest in Duquette et
al. (2008), however this was based on data from pre-experimental sessions.
Huskens et al. (2013) report an inter-rater reliability coefficient only for treatment
integrity and on 33% of the observations. No inter-observer coefficient is stated
for observations of social behaviour.
All studies varied in their measures of social engagement, interaction or
behaviour. Two studies, Huskens et al. (2013) and Vanderborght et al. (2012)
used a single measure of self-initiated questions (child behaviour) and level of
prompting (researcher behaviour) respectively. These were appropriate
measures in order to answer the research questions. Tapus et al. (2012) was
weighted highly in Weight of Evidence A and had a total of five measures of
social engagement; free initiations, interactions with prompt, eye gaze, gaze
shifting and smile or laughter.
Huskens et al. (2013) operationalised the variable of self-initiated
questions including syntactic correctness in the definition of the variable. A
limitation of the measurement of utterances in Kim et al.’s (2012) sample is that
speech did not have to be syntactically correct. Both studies controlled for IQ
however. Kim et al. (2012) required a slightly lower threshold of or above 70,
16
Doctorate in Educational and Child Psychology
Isabella McDonald
compared to 80 or above. These criteria may have wielded impact over the
findings and effect sizes.
In general, the measures used were well documented. The expressions
or reactions chosen by Duquette et al. (2008) were set according to the work of
Camaioni and Aureli (2002) and observations made during the pre-experimental
sessions.
Research Design
The majority of studies employed a single case design, Huskens et al.
(2013), Tapus et al. (2012) and Vanderborght et al. (2012). Duquette et al.
(2008) applied a mixed method of a combined crossover multiple baseline
design across participants with random assignment to experimental groups and
Kim et al. (2012) employed the only randomised group design methodology with
interactional conditions.
Kim et al. (2012) randomised the three conditions; robot, human and
computer game phases which were completed by all participants. Kim et al.
(2012) received a high weighting for quality of the comparison group in Weight
of Evidence A, methodological quality. Twenty-four participants was deemed a
large enough sample using a GPower power analysis calculation with a
medium effect size to yield significant results. The study received a high
weighting for its methodological relevance to the review questions as it use
random assignment of all participants to the intervention phases, used active
comparison intervention phase and established group equivalence.
17
Doctorate in Educational and Child Psychology
Isabella McDonald
Additional interviews and play sessions were interwoven into the design
of Kim et al.’s (2012) group study. These six minute interviews were semistructured and included before, between and after intervention phases. It is
reported that these were aimed to give children a break in between the
intervention phases and were recorded similar to these, but the dialogue or
conclusions of which are not recounted. This may have been an opportunity for
the research to gather some information on the children’s reactions and feelings
towards the Pleo robot.
A clear positive of the Vanderborght et al. (2012) design is its ecological
validity. The researchers report that they were able to take into account an
individual need of the child – a particular social behaviour – and achieve
improvement in that behaviour. These needs are also relevant outside of the
experimental scenario and therefore have meaning to the child.
For a single case design to be effectively applied, it is necessary to
establish a high quality baseline. Using the coding protocol, insufficient
information on the baselines established in Duquette et al. (2008) was given
which added to a lower rating for Weight of Evidence A. Huskens et al. (2013),
Tapus et al. (2012) and Vanderborght et al. (2012) were all awarded two out of
three for quality of baseline measures. All of which were displayed clearly in the
reports with multiple baselines before both the robot and human intervention
phases.
Findings
Table 6. Effect sizes
18
Doctorate in Educational and Child Psychology
Study
Population
Isabella McDonald
Significa
nt results
Conditio
n
Characteristic
ge
Duquett
e et al. (2008)
oE
R
s
obot
Low functioning
-5
autism. 2 pre-verbal, 2
years
non-verbal
W
Shared
H
uman
Small*
L
attention
ow
Shared
Small*
conventions
Absence
Small*
of sharing
Husken
s et al. (2013)
IQ of 80 or
-14
above
N
Selfinitiated questions
o effect
H
Large
igh
years
Kim et
al. (2012)**
High
-12
functioning ASD, IQ at
years
least 70
Utterance
s (total)
S
mall
Directed
toward
S
mall
M
edium
N
S
o effect
mall
confederate
Directed
toward partner
Tapus et
2 – 6 years
al. (2012)
M
edium
M
edium
Total
M
initiations
Moderate
Child 1
19
edium
L
N
Doctorate in Educational and Child Psychology
Isabella McDonald
autism, delayed
arge
o effect
cognitive and language
development
Elements of
N
Child 2
o effect
autism, delays in
N
o effect
language development
Moderate
Child 3
autism, delayed
N
S
o effect
mall
cognitive and language
development
Severe autism,
N
Child 4
o effect
non-verbal, delayed
S
mall
cognitive development
Vanderb
Number
orght et al.
of prompts for
(2012)
target social
M
edium
behaviour
Moderate
Child 1
autism, female
L
arge
L
arge
years
Moderate
Child 2
autism, female
L
arge
L
arge
years
Moderate
Child 3
autism, male
arge
years
20
N
L
o effect
Doctorate in Educational and Child Psychology
Moderate
Isabella McDonald
Child 4
autism, male
N
L
arge
o effect
years
*only robot vs. human reported
** compared to computer game condition taken as baseline for this
purpose
The findings are mixed and show a high variability in the children’s
reactions to the robot and human interactive phases. It is important to recognise
that the children across the reviewed studies have a broad spectrum of
difficulties. Tapus et al. (2012) recognises that the children with low functioning
autism may have found the Nao robot more appealing. However, this was not
the case in the study by Duquette et al. (2008) for which a small effect size of
the robot versus the human intervention was found. This study only sampled
four low functioning participants who are either non-verbal or pre-verbal. In the
study by Tapus et al. (2012), particularly for Child 1 who has moderate autism,
the robot intervention was more effective in engaging her than the human
partner. Table 6 shows a large effect size for Child 1 in the Nao robot
intervention and only a small or no effect for the remaining three participants.
However, it is important to mention that Tapus et al. (2012) reports a
possible habituation effect. All children showed interest in the robot at the
beginning of the session, but this tended to tail off after some time. Positive
emotions were greater in the first few minutes of interaction.
21
Doctorate in Educational and Child Psychology
Isabella McDonald
The findings in Vanderborght et al. (2012) appear conclusive that the
robot, Probo, was effective in acting as a mediator in delivering a Social Story
intervention. Table 6 shows large effect sizes for all four children who received
robot assisted therapy which was measured by levels of prompting to perform a
social behaviour. In one robot scenario, no prompting was needed for the child
to perform the behaviour showing self-initiation.
Kim et al. (2012) report in their findings that children with ASD spoke
more when interacting with the Pleo robot, rather than with a human partner or
touchscreen computer game. The effect sizes in Table 6 however only show a
small difference (small versus no effect) for the speech directed towards a
confederate. This is in line with the research design to support more speech with
the confederate rather than the interaction partner.
Similar to the findings of Tapus et al. (2012), the children in the study by
Kim et al. (2012) appear to show more curiosity and excitement toward the
robot.
In contrast, Table 6 shows a large effect size in human condition and no
effect in the robot condition in the study by Huskens et al. (2013). Fewer selfinitiated questions were recorded for children interacting with a robot, than with
a human. The results of the study by Duquette et al. (2008) were reported as
mixed. As this study was a mixed methods design, only an interaction between
the robot and human conditions was calculable from the results presented in the
paper. These differences may be explained by the sample. Duquette et al.’s
2008 study focussed only on children with low functioning autism, whereas
Husken et al.’s 2013 sample had a full-scale IQ of at least 80.
22
Doctorate in Educational and Child Psychology
Isabella McDonald
Conclusion and Recommendations
The aim of this review was to assess the effectiveness of the application
of robots, in comparison to humans, on the social interaction skills of children
with ASD. The quantitative results in the five reviewed studies are mixed and
show a high variability in the children’s reactions to the robot. Bearing in mind
the functions of the robot, degree of children’s difficulties and the social
behaviours that the intervention aims to target, the application of robots to
promote the social engagement of children can be effective. The development
of technologies for use with children with ASD remains in its infancy. At present,
the extent to which robots can be used in therapy with children with ASD is
limited also due to the resource implications associated with advanced
technologies and limitation in the functions of robots. The results however show
promise for the future application of robots in promoting social behaviour. In
doing so, therapists and psychologists working with children with autism should
consider the degree of children’s difficulties. Considerations for future research
are outlined below.
Future research
All of the studies reviewed were conducted over a short period of time.
Future research using a robot conducted over many therapy sessions with more
opportunity to change would be interesting to evaluate. A more longitudinal
study in this area would help to ascertain whether the robot is more effective
than a human-led intervention.
A setting which was not included in the studies was the classroom.
School-based therapies using robots would be an interesting area of further
23
Doctorate in Educational and Child Psychology
Isabella McDonald
exploration and one that has gained some recent media attention. This is a
setting which is relevant to educational psychology and professionals in the area
may look towards implementing a study in a special school setting, for example.
Regarding the social behaviours involved, in the conclusions drawn from
Tapus et al. (2012), it is suggested that by involving real objects in an imitation
task, imitative behaviour would be facilitated and therefore more meaningful to
the child. This suggestion is reinforced by research by Nadel (2004) and Pierno,
Mari, Lusher and Castiello (2008) (2008) which found that children with low
functioning autism can make spontaneous initiations (Nadel, 2004) and imitate
(Pierno et al., 2008) when actions are goal directed and involve objects.
The reviewed studies included an age range from two to 14 years.
Perhaps children and young people older than this age bracket may lose
interest in robots, but further research could investigate the older age bracket. In
addition, the studies assessed age in terms of chronological age. Different
results may be found if children are assessed according to developmental age.
A wide range of difficulties were presented in these studies. It may be of interest
to assess whether children on the lower, or indeed the higher end of the autistic
spectrum react mostly effectively and enjoyably with a robot.
It is clear that as technology improves and researchers hone the
functionalities of robots to appeal to children with ASD, more opportunities and
possibilities to employ social robots in therapy will develop.
24
Doctorate in Educational and Child Psychology
Isabella McDonald
References
American Psychiatric Association. (2000). Diagnostic And Statistical
Manual Of Mental Disorders DSM-IV-TR Fourth Edition (Text Revision).
Baron-Cohen, S. (1995). Mindblindness: An Essay on Autism and Theory
of Mind. Boston: MIT Press/Bradford Books.
Baron-Cohen, S. (2002). The extreme male brain theory of autism.
Trends in Cognitive Sciences, 6, 248–254.
Baron‐Cohen, S. (2009). Autism: The Empathizing–Systemizing (E‐S)
Theory. Annals of the New York Academy of Sciences, 1156(1), 68-80.
Baron-Cohen, S., Richler, J., Bisarya, D., Gurunathan, N., & Wheelwright,
S. (2003). The systemizing quotient: an investigation of adults with Asperger
syndrome or high–functioning autism, and normal sex differences. Philosophical
Transactions of the Royal Society of London. Series B: Biological Sciences,
358(1430), 361-374.
Cabibihan, J. J., Javed, H., Ang Jr, M., & Aljunied, S. M. (2013). Why
robots? A survey on the roles and benefits of social robots in the therapy of
children with autism. International Journal of Social Robotics, 5(4), 593-618.
Camaioni, L., & Aureli, T. (2002). Trajectoires développementales et
individuelles de la transition vers la communication symbolique. Enfance, 3.
25
Doctorate in Educational and Child Psychology
Isabella McDonald
Celani, G., Battacchi, M. W., & Arcidiacono, L. (1999). The understanding
of the emotional meaning of facial expressions in people with autism. Journal of
autism and developmental disorders, 29(1), 57-66.
Constantino, J. N., & Todd, R. D. (2003). Autistic traits in the general
population. A twin study, Archives of General Psychiatry, 60, 524-530.
Criado, E. (2014, February 6. Introducing ERWIN the robot with feelings.
The Independent. Retrieved from , http://www.independent.co.uk.
Dautenhahn, K., & Werry, I. (2004). Towards interactive robots in autism
therapy: Background motivation, and challenges. Pragmatics & Cognition, 12,
1–35.
Diehl, J. J., Schmitt, L. M., Villano, M., & Crowell, C. R. (2012). The
clinical use of robots for individuals with autism spectrum disorders: A critical
review. Research in Autism Spectrum Disorders, 6(1), 249-262.
Duquette, A., Michaud, F., & Mercier, H. (2008). Exploring the use of a
mobile robot as an imitation agent with children with low-functioning autism.
Autonomous Robots, 24(2), 147-157.
Fong T, Nourbakhsh I, & Dautenhahn K (2003). A survey of socially
interactive robots. Robotics and autonomous systems42(3–4):143–166
Gough, D. (2007) Weight of evidence: a framework for the appraisal of
the quality and relevance of evidence. Research Papers in Education, 22 (2).
pp. 213-228. ISSN 0267-1522.
26
Doctorate in Educational and Child Psychology
Isabella McDonald
Griffiths, A. (2014, February, 14) The robot teacher connecting with
autistic children, The Telegraph, Retrieved from, http://www.telegraph.co.uk.
Happé F, Ronald A (2008) The ‘fractionable autism triad’: a review of
evidence from behavioral, genetic, cognitive and neural research.
Neuropsychological Review, 18(4):287–304
Hart, M. (2005, October). Autism/excel study. In Proceedings of the 7th
international ACM SIGACCESS conference on Computers and accessibility,(
pp. 136-141). Huskens, B., Verschuur, R., Gillesen, J., Didden, R., & Barakova,
E. (2013). Promoting question-asking in school-aged children with autism
spectrum disorders: Effectiveness of a robot intervention compared to a humantrainer intervention. Developmental neurorehabilitation, (0), 1-12.
Kim, E. S., Berkovits, L. D., Bernier, E. P., Leyzberg, D., Shic, F., Paul,
R., & Scassellati, B. (2013). Social robots as embedded reinforcers of social
behavior in children with autism. Journal of Autism and Developmental
Disorders, 43(5), 1038-1049.
Klin, A., Lin, D. J., Gorrindo, P., Ramsay, G., & Jones, W. (2009). Twoyear-olds with autism orient to non-social contingencies rather than biological
motion. Nature, 459, 257–261.
Koegel, L. K., Camarata, S. M., Valdez-Menchaca, M., Koegel, R.L.
(1998). Setting generalization of question-asking by children with autism.
American Journal on Mental Retardation, 1998;102:346–357.
27
Doctorate in Educational and Child Psychology
Isabella McDonald
Kratochwill, T. R., & Shernoff, E. S. (2003). Evidence-based practice:
Promoting evidence-based interventions in school psychology. School
Psychology Quarterly, 18(4), 389.
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred
reporting items for systematic reviews and meta-analyses: the PRISMA
statement. Annals of internal medicine, 151(4), 264-269. Nadel, J. (2004). Do
children with autism understand imitation as intentional interaction?. Journal of
Cognitive & Behavioral Psychotherapies, 4(2).
Pierno, A. C., Mari, M., Lusher, D., & Castiello, U. (2008). Robotic
movement elicits visuomotor priming in children with autism. Neuropsychologia,
46(2), 448-454.
Robins, B., Dautenhahn, K., & Dubowski, J. (2006). Does appearance
matter in the interaction of children with autism with a humanoid robot?
Interaction Studies, 7, 509–512.
1. Schopler, E., Reichler, R. J., Bashford, A., Lansing, M., & Marcus, L.
(1990). Individualized assessment and treatment for autistic and
developmentally disabled children: Vol. 1, Psychoeducational profile-revised
(PEP-R). Austin, TX: Pro-Ed.
Tapus, A., Peca, A., Aly, A., Pop, C., Jisa, L., Pintea, S., & David, D. O. (2012).
Children with autism social engagement in interaction with Nao, an
imitative robot A series of single case experiments. Interaction studies,
13(3), 315-347.Vanderborght, B., Simut, R., Saldien, J., Pop, C., Rusu, A.
S., Pintea, S., ... & David, D. O. (2012). Using the social robot probo as a
28
Doctorate in Educational and Child Psychology
Isabella McDonald
social story telling agent for children with ASD. Interaction Studies, 13(3),
348-372.
Wall K (2009) Autism and early years practice. Sage, Thousand Oaks.
Williams, J. H., Whiten, A., & Singh, T. (2004). A systematic review of
action imitation in autistic spectrum disorder. Journal of autism and
developmental disorders, 34(3), 285-299.
29
Doctorate in Educational and Child Psychology
Isabella McDonald
Appendix 1
Identification
Flowchart 1
Additional records identified
through other sources
(n = 0 )
Studies identified through database
(x 4) searching
(n = 37 )
Screening
Records after duplicates removed
(n =24 )
Eligibility
Records screened
(n = 24 )
Records excluded
(n =19 )
Included
Full-text articles assessed
for eligibility
(n = 5 )
Studies included in
quantitative synthesis
(meta-analysis)
(n = 5 )
1
Adapted From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009).
30
Doctorate in Educational and Child Psychology
Isabella McDonald
Appendix 2
Included and Excluded Studies
Articles from PsychINFO, ERIC, Medline, Web of Science
Study
Rational for
Exclusion
Bekele, E. T., Lahiri, U., Swanson, A. R.,
(5) Does not have
Crittendon, J. A., Warren, Z. E., & Sarkar, N. (2013).
a measure of social
A step towards developing adaptive robot-mediated
engagement.
intervention architecture (ARIA) for children with
autism. Neural Systems and Rehabilitation
Engineering, IEEE Transactions on, 21(2), 289-299.
Boccanfuso, L., & O’Kane, J. M. (2011).
(5) Does not have
CHARLIE: An adaptive robot design with hand and
a measure of social
face tracking for use in autism therapy. International
engagement.
Journal of Social Robotics, 3(4), 337-347.
Cabibihan, J. J., Javed, H., Ang Jr, M., &
Aljunied, S. M. (2013). Why robots? A survey on the
(robot design)
(1)
Revi
ew of research
roles and benefits of social robots in the therapy of
children with autism. International Journal of Social
Robotics, 5(4), 593-618.
Costa, S., Resende, J., Soares, F. O.,
(1)
Typ
Ferreira, M. J., Santos, C. P., & Moreira, F. (2009,
e of publication –
September). Applications of simple robots to
conference paper
31
Doctorate in Educational and Child Psychology
Isabella McDonald
encourage social receptiveness of adolescents with
autism. In Engineering in Medicine and Biology
Society, 2009. EMBC 2009. Annual International
Conference of the IEEE (pp. 5072-5075).
Dautenhahn, K., & Werry, I. (2004). Towards
interactive robots in autism therapy: Background,
(1)
Disc
ussion article
motivation and challenges. Pragmatics & Cognition,
12(1), 1-35.
(2) Published
before 2008
Dickstein-Fischer, L., Alexander, E., Yan, X.,
Su, H., Harrington, K., & Fischer, G. S. (2011,
August). An affordable compact humanoid robot for
(1)
Discussion/
review paper
Autism Spectrum Disorder interventions in children.
In Engineering in Medicine and Biology Society,
EMBC, 2011 Annual International Conference of the
IEEE (pp. 5319-5322).
Francois, D., Powell, S., & Dautenhahn, K.
(4) Does not
(2009). A long-term study of children with autism
address research
playing with a robotic pet: Taking inspirations from
question – no robot –
non-directive play therapy to encourage children's
child and human –child
proactivity and initiative-taking. Interaction Studies,
conditions
10(3), 324-373.
Giannopulu, I., & Pradel, G. (2010).
Multimodal interactions in free game play of children
32
(4) No human –
child interaction
Doctorate in Educational and Child Psychology
with autism and a mobile toy robot.
Isabella McDonald
condition
NeuroRehabilitation, 27(4), 305-311.
Jordan, K., King, M., Hellersteth, S., Wirén, A.,
(1)
& Mulligan, H. (2013). Feasibility of using a
study for
humanoid robot for enhancing attention and social
experimental
skills in adolescents with autism spectrum disorder.
setup
Pilot
International Journal of Rehabilitation Research.
Kozima, H., Nakagawa, C., & Yasuda, Y.
0
(1)
Typ
(2007). Children–robot interaction: a pilot study in
e of publication –
autism therapy. Progress in Brain Research, 164,
pilot study.
385-400.
(3) Published
before 2008.
Mazzei, D., Lazzeri, N., Billeci, L., Igliozzi, R.,
1
(1)
Publ
Mancini, A., Ahluwalia, A., ... & De Rossi, D. (2011,
ication type -
August). Development and evaluation of a social
Conference paper
robot platform for therapy in autism. In Engineering in
Medicine and Biology Society, EMBC, 2011 Annual
International Conference of the IEEE (pp. 45154518).
Nadel, J. (2006). Does imitation matter to
33
(1)
Revi
Doctorate in Educational and Child Psychology
2
children with autism. Imitation and the social mind,
Isabella McDonald
ew chapter
118-134.
Pierno, A. C., Mari, M., Lusher, D., & Castiello,
3
U. (2008). Robotic movement elicits visuomotor
does not target
priming in children with autism. Neuropsychologia,
improving social
46(2), 448-454.
engagement skills.
Robins, B., Dickerson, P., Stribling, P., &
4
(5) Intervention –
Dautenhahn, K. (2004). Robot-mediated joint
(4) No humanchild condition.
attention in children with autism: A case study in
robot-human interaction. Interaction studies, 5(2),
161-198.
(2) Published
before 2008
Robins, B., Dautenhahn, K., & Dubowski, J.
5
(4) No human-
(2006). Does appearance matter in the interaction of
child interaction
children with autism with a humanoid robot?
condition.
Interaction Studies, 7(3), 509-542.
(2) Published
before 2008.
Scassellati, B., Admoni, H., & Mataric, M.
6
(2012). Robots for use in autism research. Annual
a measure of social
Review of Biomedical Engineering, 14, 275-294.
engagement.
Wainer, J., Dautenhahn, K., Robins, B., &
7
(5) Does not have
Amirabdollahian, F. (2010, December). Collaborating
(1)
Conf
erence paper
with kaspar: Using an autonomous humanoid robot
to foster cooperative dyadic play among children with
34
Aim - Pilot study
Doctorate in Educational and Child Psychology
Isabella McDonald
autism. In Humanoid Robots (Humanoids), 2010 10th investigating
IEEE-RAS International Conference on (pp. 631-
experimental set up
638).
Wass, S. V., & Porayska-Pomsta, K. (2013).
8
(5)
The uses of cognitive training technologies in the
ew of robot
treatment of autism spectrum disorders. Autism,
design
Revi
1362361313499827.
Welch, K. C., Lahiri, U., Warren, Z., & Sarkar,
9
N. (2010). An approach to the design of socially
acceptable robots for children with autism spectrum
disorders. International Journal of Social Robotics,
2(4), 391-403.
35
(5)
Aim
– robot design
Doctorate in Educational and Child Psychology
Isabella McDonald
Included Studies
Duquette, A., Michaud, F., & Mercier, H. (2008). Exploring the use of a
mobile robot as an imitation agent with children with low-functioning autism.
Autonomous Robots, 24(2), 147-157.
Huskens, B., Verschuur, R., Gillesen, J., Didden, R., & Barakova, E.
(2013). Promoting question-asking in school-aged children with autism spectrum
disorders: Effectiveness of a robot intervention compared to a human-trainer
intervention. Developmental neurorehabilitation, (0), 1-12.
Kim, E. S., Berkovits, L. D., Bernier, E. P., Leyzberg, D., Shic, F., Paul,
R., & Scassellati, B. (2013). Social robots as embedded reinforcers of social
behavior in children with autism. Journal of Autism and Developmental
Disorders, 43(5), 1038-1049.
Tapus, A., Peca, A., Aly, A., Pop, C., Jisa, L., Pintea, S., & David, D. O.
(2012). Children with autism social engagement in interaction with Nao, an
imitative robot A series of single case experiments. Interaction studies, 13(3),
315-347.
Vanderborght, B., Simut, R., Saldien, J., Pop, C., Rusu, A. S., Pintea, S.,
& David, D. O. (2012). Using the social robot probo as a social story telling
agent for children with ASD. Interaction Studies, 13(3), 348-372.
36
Doctorate in Educational and Child Psychology
Isabella McDonald
Appendix 3
Gough’s (2007) Systematic Review Weight of Evidence (WoE) Criteria
Weight of Evidence A; Methodological quality
These are generally accepted criteria for evaluating evidence by those who use and produce it.
Weight of Evidence B; Methodological relevance
This is a review-specific judgement about the appropriateness of the design and measures to answer the
review question.
Weight of Evidence C; Relevance of evidence
This is a review-specific judgement about the relevance of evidence to the review question.
Weight of Evidence D; Overall weighting
The three sets of judgments (A, B, C) combine to make an overall assessment.
In order to receiver a rating of high, the study must demonstrate three of the following; In order to receiver
a rating of medium, the study must demonstrate two of the following; In order to receiver a rating of low,
the study must demonstrate one or fewer of the following:
A
Quality of measures
 Two or more
measures of social
engagement are
used
 A case for validity
is made
 More than one
method of data
collection is used
Quality of analysis
 Inter-rater or
inter-observer
reliability or
agreement is 0.7 or
above
 Appropriate unit
of data analysis
 Effect sizes
reported, or enough
information given to
be calculated
37
Quality of baseline/
comparison group
Single case design
 Length of baseline phase
– at least three data points
are used
 Level of baseline phase –
consistently warrants an
intervention
 Overlap of scores –
extreme scores in baseline
do not overlap with treatment
phase
Group design
 Sufficiently large N is
studied to detect medium
effect size using GPower
application (at power 0.80 for
0.50)
 An active comparison
group is used
 Demonstrates group
equivalence
Doctorate in Educational and Child Psychology
Isabella McDonald
B
Relevance of methodology

Uses random assignment of participants to intervention group or phase

Uses active comparison intervention group or phase

Establishes group equivalence
C
Relevance to review question
 Documents relationship between and previous exposure to
researcher/confederates/ robot and participants
 Clear descriptors of participants’ diagnosis and social deficits/skills
 Intervention must target a particular social skill or deficit which has been
appropriately identified
D
Calculated as an average of the weight of evidence A, B and C.
Overall weighting
High At least 2.5
Medium Between 1.5 and 2.4
Low Less than 2.4
Unclassified 0
Study
Kim et al
(2012)
Huskens
(2013)
Duquette
(2008)
Vanderborght
(2012)
WoE Methodological Quality
Quality of
measures
Quality of
analysis
Quality of baseline/
comparison group
Overall
methodological
quality (A)
Low
1
Low
1
Medium
2
Low
1
High
3
High
3
Medium
2
High
3
High
3
Low
1
Unclassified
0
High
3
Medium
2.3
Medium
1.6
Low
1.3
Medium
2.3
38
Doctorate in Educational and Child Psychology
Tapus (2012)
Medium
2
Study
Duquette et al.
(2008)
Huskens et al.
(2013)
Kim et al.
(2012)
Tapus et al.
(2012)
Vanderborght
et al. (2012)
Isabella McDonald
High
3
High
3
High
2.6
Weight of Evidence Rating
Methodological
Methodological
Quality
Relevance
(A)
(B)
Review Relevance
(C)
Overall
(D)
Low
1.3
Medium
1.6
Medium
2.3
High
2.6
Medium
2.3
Medium
2
High
3
Low
1
Medium
2
High
3
Low
1.4
High
2.5
Medium
2
Medium
1.9
Medium
2.1
Low
1
High
3
High
3
Low
1
Low
1
Effect size calculations
Type of effect size
Percent NonOverlapping Data
points (PNDs)
Small
50-69%
Medium
70-89%
Large
90-100%
Cohen’s d
0.20
0.50
0.80
39
Doctorate in Educational and Child Psychology
Isabella McDonald
Appendix 4
Adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology Association, Kratochwill,
T.R. (2003)]
Coding Protocol: Single-Participant Design
Domain:
School- and community-based intervention programs for social and behavioral problems
Academic intervention programs
Family and parent intervention programs
School-wide and classroom-based programs
Comprehensive and coordinated school health services
Name of Coder(s): ____________________________
Date: ____8th February 2014____
M/D/Y
Full Study Reference in APA format:
Duquette, A., Michaud, F., & Mercier, H. (2008). Exploring the use of a mobile robot as an imitation agent
with children with low-functioning autism. Autonomous Robots, 24(2), 147-157
Intervention Name (description from study): __________Robot intervention______________________________
__________________________________________________________________________________________
__________________________________________________________________________________________
Study ID Number (Unique Identifier): __________________________________________________________
Type of Publication: (Check one)
Book/Monograph
40
Doctorate in Educational and Child Psychology
Isabella McDonald
Journal article
Book chapter
Other (specify):
I. General Characteristics
A. General Design Characteristics (Classify studies according to the type of design)
A1. Type of Single-Participant Design (select A1.1, A1.2, A1.3, A1.4, or A1.5)
A1.1
Within-series design (select A1.1.1 or A1.1.2)
A1.1.1
A1.1.2
A1.2
Between-series design (select A1.2.1 or A1.2.2)
A1.2.1
A1.2.2
A1.3
A1.4
Simple phase change
Complex phase change
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select A1.3.1., A1.3.2, A1.3.3, or A1.3.4)
A1.3.1
A1.3.2
A1.3.3
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
A1.3.4
Multiple probe design
Mixed design (select A1.4.1 or A1.4.2)
A1.4.1
Combined single-participant and group design (see group manual), A1.4.2
Combined single-participant design (if combined single-participant
design, check A1.4.2.1, A1.4.2.2, or A1.4.2.3)
A1.4.2.1
A1.4.2.2
Within-series design (select i or ii)
i.
Simple phase change
ii.
Complex phase change
Between-series design (select i or ii)
i.
ii.
Comparing two interventions
Comparing interventions with no interventions
41
Doctorate in Educational and Child Psychology
A1.4.2.3
A1.5
Isabella McDonald
Combined-series design (select i, ii, iii, or iv)
i.
ii.
iii.
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
iv.
Multiple probe design
Other (specify):
A2. General Study Characteristics and Cultural Factors
A2.1
Social Significance of Goals
A2.2
A2.3
Social Appropriateness of Procedures
Social Importance of Effects
A3. Rating for Theoretical Basis (select 0, 1, 2, or 3): 3
2
1
0
A4. Rating for Documenting the Relationship Between the Implementers and Participants (select 0, 1, 2, or 3):
3
2
1
0
3 2
1
0
A6. Rating for Researcher Perspective (select 0, 1, 2, or 3): 3
2
1
0
A7. Rating for Moderator Variables (select 0, 1, 2, or 3):
3
2
1
0
A8. Rating for Rival Interpretations (select 0, 1, 2, or 3):
3
2
1
0
A5. Rating for Ecological Validity (select 0, 1, 2, or 3):
B. Other Design Characteristics (when randomization is used)
B1. Unit of assignment to conditions/groups (select one of the following)
B1.1
B1.2
B1.3
B1.4
Individual
Classroom
School
Other (specify):
B1.5
N/A (randomization not used)
B2. Type of assignment to conditions/groups (select one of the following)
B2.1
Random after matching, stratification, blocking
B2.2
B2.3
B2.4
B2.5
Random, simple (includes systematic sampling)
Nonrandom, post hoc matching
Nonrandom, other
Other (specify):
42
Doctorate in Educational and Child Psychology
B2.6
B2.7
Isabella McDonald
Unknown/insufficient information provided
N/A (randomization not used)
B3. Overall confidence of judgment on how participants were assigned to conditions/groups (select one of the following)
B3.1
B3.2
B3.3
B3.4
B3.5
B3.6
B3.7
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A (randomization not used)
Unknown/unable to code
B4. Equivalence of conditions/groups tested at pretest (select one of the following)
B4.1
Yes
B4.2
B4.3
B4.4
No
Unknown/insufficient information provided
N/A (randomization not used)
B5. Total size of sample (start of the study): _9___
N
B6. Intervention sample size _2___
N/A (randomization not used)
N
B7. Control sample size
_2___
N/A (randomization not used)
N
C. Type of Program (select one)
C1.
C2.
C3.
Universal prevention program
Selective prevention program
Indicated prevention program
C4.
Intervention/Treatment
C5.
Unknown
D. Stage of the Program (select one)
D1.
D2.
D3.
D4.
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
43
Doctorate in Educational and Child Psychology
Isabella McDonald
E. Concurrent or Historical Intervention Exposure (select one)
E1.
E2.
E3.
Current exposure
Prior exposure
Unknown
II. Key Features for Coding Studies and Rating Level of Evidence
(3=Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence)
A. Measurement: Issues of Reliability and Validity (answer A1. through A4.)
A1. Use of outcome measures that produce reliable scores (select one of the following)
A1.1
Yes
A1.2
A1.3
No
Unknown/unable to code
A2. Multi-method (select one of the following)
A2.1
Yes
A2.2
A2.3
A2.4
No
N/A
Unknown/unable to code
A3. Multi-source (select one of the following)
A3.1
A3.2
A3.3
A3.4
Yes
No
N/A
Unknown/unable to code
A4. Validity of measures reported (select one of the following)
A4.1
A4.2
A4.3
Yes
No
Unknown/unable to code
Rating for Measurement (select 0, 1, 2, or 3):
3
2
1
0
B. Quality of Baseline. Rate quality of baseline: (a) for each participant (when there is more than one participant), and
(b) for each phase (when the study includes more than one phase). These procedures should be followed for each primary outcome
44
Doctorate in Educational and Child Psychology
Isabella McDonald
under investigation.
Participant 1 (answer B1. through B5.) Robot and human intervention phases
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 1: (select 0, 1, 2, or 3):
3
2
1
0
__________________________________________________________________________________________________
Participant 2 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
45
Doctorate in Educational and Child Psychology
Isabella McDonald
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 2: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 3 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
46
Doctorate in Educational and Child Psychology
Isabella McDonald
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 3: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 4 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
47
Doctorate in Educational and Child Psychology
Isabella McDonald
Rating of quality of baseline for participant 4: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Average Quality of Baseline Rating Across Participants:
 of X =
N
X = individual quality of baseline ratings for each participant
N = number of participants in the study
Overall Rating for Quality of Baseline: (select 0, 1, 2, or 3):
3
2
1
0
(Round up or down to the nearest whole number when providing a mean rating for the study. For example, 2.0 to 2.4 rated as 2; 2.5 to
2.9 rated as 3).
C. Measures Support Primary and Secondary Outcomes (list primary outcomes first in alphabetical order for each
participant followed by secondary outcomes. Use extra tables if necessary).
Outcome #1: ____________________________________________
Primary vs. Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who changed
What changed
Participant 1:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Participant 2:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Participant 3:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Participant 4:
48
Measurement used to
determine change
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Describe out-come
variables
Treatme
Mediator's
imitative play
patterns (all)
Shared
conventions
Shared attention
Absence of
sharing
Doctorate in Educational and Child Psychology
Primary
Secondary
Unknown
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Isabella McDonald
Behavior
Attitude
Knowledge
Other
Unknown
49
Parent Report
Teacher Report
Observation
Test
Other
Unknown
O
Other
Phenommenon
Doctorate in Educational and Child Psychology
Isabella McDonald
Coding Protocol: Single-Participant Design
Domain:
School- and community-based intervention programs for social and behavioral problems
Academic intervention programs
Family and parent intervention programs
School-wide and classroom-based programs
Comprehensive and coordinated school health services
Name of Coder(s): ____________________________
Date: ____8th February 2014____
M/D/Y
Full Study Reference in APA format:
Huskens, B., Verschuur, R., Gillesen, J., Didden, R., & Barakova, E. (2013).
Promoting question-asking in school-aged children with autism spectrum disorders:
Effectiveness of a robot intervention compared to a human-trainer intervention.
Developmental neurorehabilitation, (0), 1-12.
Intervention Name (description from study): __________ABA intervention delivered by robot or human
__________________________________________________________________________________________
__________________________________________________________________________________________
Study ID Number (Unique Identifier): __________________________________________________________
Type of Publication: (Check one)
Book/Monograph
Journal article
Book chapter
Other (specify):
50
Doctorate in Educational and Child Psychology
Isabella McDonald
I. General Characteristics
A. General Design Characteristics (Classify studies according to the type of design)
A1. Type of Single-Participant Design (select A1.1, A1.2, A1.3, A1.4, or A1.5)
A1.1
Within-series design (select A1.1.1 or A1.1.2)
A1.1.1
A1.1.2
A1.2
Between-series design (select A1.2.1 or A1.2.2)
A1.2.1
A1.2.2
A1.3
A1.4
Simple phase change
Complex phase change
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select A1.3.1., A1.3.2, A1.3.3, or A1.3.4)
A1.3.1
A1.3.2
A1.3.3
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
A1.3.4
Multiple probe design
Mixed design (select A1.4.1 or A1.4.2)
A1.4.1
A1.4.2
Combined single-participant and group design (see group manual),
Combined single-participant design (if combined single-participant
design, check A1.4.2.1, A1.4.2.2, or A1.4.2.3)
A1.4.2.1
A1.4.2.2
Within-series design (select i or ii)
iii.
Simple phase change
iv.
Complex phase change
Between-series design (select i or ii)
iii.
iv.
A1.4.2.3
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select i, ii, iii, or iv)
v.
vi.
vii.
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
51
Doctorate in Educational and Child Psychology
viii.
A1.5
Isabella McDonald
Multiple probe design
Other (specify):
A2. General Study Characteristics and Cultural Factors
A2.1
Social Significance of Goals
A2.2
A2.3
Social Appropriateness of Procedures
Social Importance of Effects
A3. Rating for Theoretical Basis (select 0, 1, 2, or 3): 3
2
1
0
A4. Rating for Documenting the Relationship Between the Implementers and Participants (select 0, 1, 2, or 3):
3
2
1
0
3 2
1
0
A6. Rating for Researcher Perspective (select 0, 1, 2, or 3): 3
2
1
0
A7. Rating for Moderator Variables (select 0, 1, 2, or 3):
3
2
1
0
A8. Rating for Rival Interpretations (select 0, 1, 2, or 3):
3
2
1
0
A5. Rating for Ecological Validity (select 0, 1, 2, or 3):
B. Other Design Characteristics (when randomization is used)
B1. Unit of assignment to conditions/groups (select one of the following)
B1.1
B1.2
B1.3
B1.4
Individual
Classroom
School
Other (specify):
B1.5
N/A (randomization not used)
B2. Type of assignment to conditions/groups (select one of the following)
B2.1
Random after matching, stratification, blocking
B2.2
B2.3
B2.4
B2.5
B2.6
B2.7
Random, simple (includes systematic sampling)
Nonrandom, post hoc matching
Nonrandom, other
Other (specify):
Unknown/insufficient information provided
N/A (randomization not used)
B3. Overall confidence of judgment on how participants were assigned to conditions/groups (select
one of the following)
52
Doctorate in Educational and Child Psychology
B3.1
B3.2
B3.3
B3.4
B3.5
B3.6
B3.7
Isabella McDonald
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A (randomization not used)
Unknown/unable to code
B4. Equivalence of conditions/groups tested at pretest (select one of the following)
B4.1
Yes
B4.2
B4.3
B4.4
No
Unknown/insufficient information provided
N/A (randomization not used)
B5. Total size of sample (start of the study): _6___
N
B6. Intervention sample size ____
N/A (randomization not
used)
N
B7. Control sample size
N/A (randomization not
used)
____
N
C. Type of Program (select one)
C1.
C2.
C3.
Universal prevention program
Selective prevention program
Indicated prevention program
C4.
Intervention/Treatment
C5.
Unknown
D. Stage of the Program (select one)
D1.
D2.
D3.
D4.
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure (select one)
E1.
E2.
E3.
Current exposure
Prior exposure
Unknown
53
Doctorate in Educational and Child Psychology
Isabella McDonald
54
Doctorate in Educational and Child Psychology
Isabella McDonald
II. Key Features for Coding Studies and Rating Level of Evidence
(3=Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence)
A. Measurement: Issues of Reliability and Validity (answer A1. through A4.)
A1. Use of outcome measures that produce reliable scores (select one of the following)
A1.1
Yes
A1.2
A1.3
No
Unknown/unable to code
A2. Multi-method (select one of the following)
A2.1
Yes
A2.2
A2.3
A2.4
No
N/A
Unknown/unable to code
A3. Multi-source (select one of the following)
A3.1
A3.2
A3.3
A3.4
Yes
No
N/A
Unknown/unable to code
A4. Validity of measures reported (select one of the following)
A4.1
A4.2
A4.3
Yes
No
Unknown/unable to code
Rating for Measurement (select 0, 1, 2, or 3):
3
2
1
0
B. Quality of Baseline. Rate quality of baseline: (a) for each participant (when there is more than one participant), and
(b) for each phase (when the study includes more than one phase). These procedures should be followed for each primary outcome
under investigation.
Participant 1 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
55
Doctorate in Educational and Child Psychology
B1.1
B1.2
B1.3
Isabella McDonald
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 1: (select 0, 1, 2, or 3):
3
2
1
0
__________________________________________________________________________________________________
Participant 2 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
56
Doctorate in Educational and Child Psychology
Isabella McDonald
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 2: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 3 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
Yes
57
Doctorate in Educational and Child Psychology
B4.2
B4.3
Isabella McDonald
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 3: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 4 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 4: (select 0, 1, 2, or 3):
3
2
1
0
_______________________________________________________________________________
58
Doctorate in Educational and Child Psychology
Isabella McDonald
Average Quality of Baseline Rating Across Participants:
 of X =
N
X = individual quality of baseline ratings for each participant
N = number of participants in the study
Overall Rating for Quality of Baseline: (select 0, 1, 2, or 3):
2
3
1
0
(Round up or down to the nearest whole number when providing a mean rating for the study. For example, 2.0 to 2.4 rated as 2; 2.5 to
2.9 rated as 3).
C. Measures Support Primary and Secondary Outcomes (list primary outcomes first in alphabetical order for each
participant followed by secondary outcomes. Use extra tables if necessary).
Outcome #1: ____Self-initiated question asking___________________________
Primary vs. Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who changed
What changed
Measurement used to
determine change
Participant 1:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Participant 2:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Participant 3:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Participant 4:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
59
Describe out-come
variables
Treatme
Robot
/human
interaction
(All participants)
Statement
Syntactically correct
question
that implies an action
Reinforcement by
trainer (action)
Reinforcement by robot
(action)
Syntactically correct
question
that does not imply an
action
Verbal reinforcement
by trainer
Incorrect response
Open-question-prompt
Waiting-prompt
Tell-prompt
Fill-in-prompt
Doctorate in Educational and Child Psychology
Isabella McDonald
Coding Protocol: Single-Participant Design
Domain:
School- and community-based intervention programs for social and behavioral problems
Academic intervention programs
Family and parent intervention programs
School-wide and classroom-based programs
Comprehensive and coordinated school health services
Name of Coder(s): ____________________________
Date: ____8th February 2014____
M/D/Y
Full Study Reference in APA format:
Tapus, A., Peca, A., Aly, A., Pop, C., Jisa, L., Pintea, S., David, D. O. (2012). Children with
autism social engagement in interaction with Nao, an imitative robot A series of single case
experiments. Interaction studies, 13(3), 315-347.
Intervention Name (description from study): __________Promoting imitative behaviour with Nao
__________________________________________________________________________________________
__________________________________________________________________________________________
Study ID Number (Unique Identifier): __________________________________________________________
Type of Publication: (Check one)
Book/Monograph
Journal article
Book chapter
Other (specify):
60
Doctorate in Educational and Child Psychology
Isabella McDonald
I. General Characteristics
A. General Design Characteristics (Classify studies according to the type of design)
A1. Type of Single-Participant Design (select A1.1, A1.2, A1.3, A1.4, or A1.5)
A1.1
Within-series design (select A1.1.1 or A1.1.2)
A1.1.1
A1.1.2
A1.2
Between-series design (select A1.2.1 or A1.2.2)
A1.2.1
A1.2.2
A1.3
A1.4
Simple phase change
Complex phase change
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select A1.3.1., A1.3.2, A1.3.3, or A1.3.4)
A1.3.1
A1.3.2
A1.3.3
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
A1.3.4
Multiple probe design
Mixed design (select A1.4.1 or A1.4.2)
A1.4.1
A1.4.2
Combined single-participant and group design (see group manual),
Combined single-participant design (if combined single-participant
design, check A1.4.2.1, A1.4.2.2, or A1.4.2.3)
A1.4.2.1
A1.4.2.2
Within-series design (select i or ii)
v.
Simple phase change
vi.
Complex phase change
Between-series design (select i or ii)
v.
vi.
A1.4.2.3
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select i, ii, iii, or iv)
ix.
x.
xi.
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
61
Doctorate in Educational and Child Psychology
xii.
A1.5
Isabella McDonald
Multiple probe design
Other (specify):
A2. General Study Characteristics and Cultural Factors
A2.1
Social Significance of Goals
A2.2
A2.3
Social Appropriateness of Procedures
Social Importance of Effects
A3. Rating for Theoretical Basis (select 0, 1, 2, or 3): 3
2
1
0
A4. Rating for Documenting the Relationship Between the Implementers and Participants (select 0, 1, 2, or 3):
3
2
1
0
3 2
1
0
A6. Rating for Researcher Perspective (select 0, 1, 2, or 3): 3
2
1
0
A7. Rating for Moderator Variables (select 0, 1, 2, or 3):
3
2
1
0
A8. Rating for Rival Interpretations (select 0, 1, 2, or 3):
3
2
1
0
A5. Rating for Ecological Validity (select 0, 1, 2, or 3):
B. Other Design Characteristics (when randomization is used)
B1. Unit of assignment to conditions/groups (select one of the following)
B1.1
B1.2
B1.3
B1.4
Individual
Classroom
School
Other (specify):
B1.5
N/A (randomization not used)
B2. Type of assignment to conditions/groups (select one of the following)
B2.1
Random after matching, stratification, blocking
B2.2
B2.3
B2.4
B2.5
B2.6
B2.7
Random, simple (includes systematic sampling)
Nonrandom, post hoc matching
Nonrandom, other
Other (specify):
Unknown/insufficient information provided
N/A (randomization not used)
B3. Overall confidence of judgment on how participants were assigned to conditions/groups (select
one of the following)
62
Doctorate in Educational and Child Psychology
B3.1
B3.2
B3.3
B3.4
B3.5
B3.6
B3.7
Isabella McDonald
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A (randomization not used)
Unknown/unable to code
B4. Equivalence of conditions/groups tested at pretest (select one of the following)
B4.1
Yes
B4.2
B4.3
B4.4
No
Unknown/insufficient information provided
N/A (randomization not used)
B5. Total size of sample (start of the study): ___
N
B6. Intervention sample size ____
N/A (randomization not
used)
N
B7. Control sample size
N/A (randomization not
used)
____
N
C. Type of Program (select one)
C1.
C2.
C3.
Universal prevention program
Selective prevention program
Indicated prevention program
C4.
Intervention/Treatment
C5.
Unknown
D. Stage of the Program (select one)
D1.
D2.
D3.
D4.
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure (select one)
E1.
E2.
E3.
Current exposure
Prior exposure
Unknown
63
Doctorate in Educational and Child Psychology
Isabella McDonald
64
Doctorate in Educational and Child Psychology
Isabella McDonald
II. Key Features for Coding Studies and Rating Level of Evidence
(3=Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence)
A. Measurement: Issues of Reliability and Validity (answer A1. through A4.)
A1. Use of outcome measures that produce reliable scores (select one of the following)
A1.1
Yes
A1.2
A1.3
No
Unknown/unable to code
A2. Multi-method (select one of the following)
A2.1
Yes
A2.2
A2.3
A2.4
No
N/A
Unknown/unable to code
A3. Multi-source (select one of the following)
A3.1
A3.2
A3.3
A3.4
Yes
No
N/A
Unknown/unable to code
A4. Validity of measures reported (select one of the following)
A4.1
A4.2
A4.3
Yes
No
Unknown/unable to code
Rating for Measurement (select 0, 1, 2, or 3):
3
1
2
0
B. Quality of Baseline. Rate quality of baseline: (a) for each participant (when there is more than one participant), and
(b) for each phase (when the study includes more than one phase). These procedures should be followed for each
primary outcome under investigation.
Participant 1 (answer B1. through B5.)
Robot intervention phase
B1. Length: At least 3 data points during baseline (select one of the following)
65
Doctorate in Educational and Child Psychology
B1.1
B1.2
B1.3
Isabella McDonald
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 1: (select 0, 1, 2, or 3):
3
2
1
0
__________________________________________________________________________________________________
Participant 2 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
66
Doctorate in Educational and Child Psychology
B2.1
B2.2
B2.3
Isabella McDonald
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 2: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 3 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
67
Doctorate in Educational and Child Psychology
Isabella McDonald
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 3: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 4 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
68
Doctorate in Educational and Child Psychology
B5.1
B5.2
B5.3
Isabella McDonald
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 4: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Average Quality of Baseline Rating Across Participants:
 of X =
N
X = individual quality of baseline ratings for each participant
N = number of participants in the study
Overall Rating for Quality of Baseline: (select 0, 1, 2, or 3):
3
2
1
0
(Round up or down to the nearest whole number when providing a mean rating for the study. For example, 2.0 to 2.4
rated as 2; 2.5 to 2.9 rated as 3).
B. Quality of Baseline. Rate quality of baseline: (a) for each participant (when there is more than one participant), and
(b) for each phase (when the study includes more than one phase). These procedures should be followed for each
primary outcome under investigation.
Participant 1 (answer B1. through B5.)
Human intervention phase
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
69
Doctorate in Educational and Child Psychology
Isabella McDonald
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 1: (select 0, 1, 2, or 3):
3
2
1
0
__________________________________________________________________________________________________
Participant 2 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
70
Doctorate in Educational and Child Psychology
B3.1
B3.2
B3.3
Isabella McDonald
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 2: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 3 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
71
Doctorate in Educational and Child Psychology
Isabella McDonald
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 3: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 4 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase (select one of the
following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 4: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Average Quality of Baseline Rating Across Participants:
72
Doctorate in Educational and Child Psychology
Isabella McDonald
 of X =
N
X = individual quality of baseline ratings for each participant
N = number of participants in the study
Overall Rating for Quality of Baseline: (select 0, 1, 2, or 3):
2
3
1
0
(Round up or down to the nearest whole number when providing a mean rating for the study. For example, 2.0 to 2.4 rated as 2; 2.5 to
2.9 rated as 3)C. Measures Support Primary and Secondary Outcomes (list primary outcomes first in alphabetical order for each
participant followed by secondary outcomes. Use extra tables if necessary).
Outcome #1: ____Imitative behaviour_____________________
Primary vs. Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who changed
What changed
Measurement used to
determine change
Participant 1:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Participant 2:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Participant 3:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Participant 4:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
73
Describe out-come
variables
Treatme
Robot
/human
Interaction
(all)
Total initiations in
response to a
human
or robot
trainer
Total initiations in
response to a human
or robot
trainer
Total initiations in
response to a human
or robot
trainer
Total initiations in
response to a human
or robot
trainer
Doctorate in Educational and Child Psychology
Isabella McDonald
Coding Protocol: Single-Participant Design
Domain:
School- and community-based intervention programs for social and behavioral problems
Academic intervention programs
Family and parent intervention programs
School-wide and classroom-based programs
Comprehensive and coordinated school health services
Name of Coder(s): ____________________________
Date: ____8th February 2014____
M/D/Y
Full Study Reference in APA format: Vanderborght, B., Simut, R., Saldien, J., Pop, C., Rusu, A. S.,
Pintea, S., ... & David, D. O. (2012). Using the social robot probo as a social story telling
agent for children with ASD. Interaction Studies, 13(3), 348-372.
Intervention Name (description from study): __________Robot and human intervention using a Social Story
________________________________________________________________________________
Study ID Number (Unique Identifier): __________________________________________________________
Type of Publication: (Check one)
Book/Monograph
Journal article
Book chapter
Other (specify):
74
Doctorate in Educational and Child Psychology
Isabella McDonald
I. General Characteristics
A. General Design Characteristics (Classify studies according to the type of design)
A1. Type of Single-Participant Design (select A1.1, A1.2, A1.3, A1.4, or A1.5)
A1.1
Within-series design (select A1.1.1 or A1.1.2)
A1.1.1
A1.1.2
A1.2
Between-series design (select A1.2.1 or A1.2.2)
A1.2.1
A1.2.2
A1.3
A1.4
Simple phase change
Complex phase change
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select A1.3.1., A1.3.2, A1.3.3, or A1.3.4)
A1.3.1
A1.3.2
A1.3.3
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
A1.3.4
Multiple probe design
Mixed design (select A1.4.1 or A1.4.2)
A1.4.1
A1.4.2
Combined single-participant and group design (see group manual),
Combined single-participant design (if combined single-participant
design, check A1.4.2.1, A1.4.2.2, or A1.4.2.3)
A1.4.2.1
A1.4.2.2
Within-series design (select i or ii)
vii.
Simple phase change
viii.
Complex phase change
Between-series design (select i or ii)
vii.
viii.
A1.4.2.3
Comparing two interventions
Comparing interventions with no interventions
Combined-series design (select i, ii, iii, or iv)
xiii.
xiv.
xv.
Multiple baseline across participants
Multiple baseline across behaviors
Multiple baseline across settings
75
Doctorate in Educational and Child Psychology
xvi.
A1.5
Isabella McDonald
Multiple probe design
Other (specify):
A2. General Study Characteristics and Cultural Factors
A2.1
Social Significance of Goals
A2.2
A2.3
Social Appropriateness of Procedures
Social Importance of Effects
A3. Rating for Theoretical Basis (select 0, 1, 2, or 3): 3
2
1
0
A4. Rating for Documenting the Relationship Between the Implementers and Participants (select 0, 1, 2, or 3):
3
2
1
0
3 2
1
0
A6. Rating for Researcher Perspective (select 0, 1, 2, or 3): 3
2
1
0
A7. Rating for Moderator Variables (select 0, 1, 2, or 3):
3
2
1
0
A8. Rating for Rival Interpretations (select 0, 1, 2, or 3):
3
2
1
0
A5. Rating for Ecological Validity (select 0, 1, 2, or 3):
B. Other Design Characteristics (when randomization is used)
C. Type of Program (select one)
C1.
C2.
C3.
Universal prevention program
Selective prevention program
Indicated prevention program
C4.
Intervention/Treatment
C5.
Unknown
D. Stage of the Program (select one)
D1.
D2.
D3.
D4.
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure (select one)
76
Doctorate in Educational and Child Psychology
E1.
E2.
E3.
Isabella McDonald
Current exposure
Prior exposure
Unknown
II. Key Features for Coding Studies and Rating Level of Evidence
(3=Strong Evidence 2=Promising Evidence
1=Weak Evidence
0=No Evidence)
A. Measurement: Issues of Reliability and Validity (answer A1. through A4.)
A1. Use of outcome measures that produce reliable scores (select one of the following)
A1.1
Yes
A1.2
A1.3
No
Unknown/unable to code
A2. Multi-method (select one of the following)
A2.1
Yes
A2.2
A2.3
A2.4
No
N/A
Unknown/unable to code
A3. Multi-source (select one of the following)
A3.1
A3.2
A3.3
A3.4
Yes
No
N/A
Unknown/unable to code
A4. Validity of measures reported (select one of the following)
A4.1
A4.2
A4.3
Yes
No
Unknown/unable to code
Rating for Measurement (select 0, 1, 2, or 3):
3
1
2
0
B. Quality of Baseline. Rate quality of baseline: (a) for each participant (when there is more than one participant), and
(b) for each phase (when the study includes more than one phase). These procedures should be followed for each
77
Doctorate in Educational and Child Psychology
Isabella McDonald
primary outcome under investigation.
Participant 1 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 1: (select 0, 1, 2, or 3):
3
2
1
0
__________________________________________________________________________________________________
Participant 2 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
Yes
78
Doctorate in Educational and Child Psychology
B1.2
B1.3
Isabella McDonald
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 2: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 3 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
79
Doctorate in Educational and Child Psychology
Isabella McDonald
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
B4.2
B4.3
Yes
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 3: (select 0, 1, 2, or 3):
3
2
1
0
________________________________________________________________________________
Participant 4 (answer B1. through B5.)
B1. Length: At least 3 data points during baseline (select one of the following)
B1.1
B1.2
B1.3
Yes
No
Unknown/insufficient information provided
B2. Stability: Variability in scores does not eliminate the detection of treatment effects (select one of the
following)
B2.1
B2.2
B2.3
Yes
No
Unknown/insufficient information provided
B3. Overlap: Extreme scores during baseline do not overlap with most scores during intervention phase
(select one of the following)
B3.1
B3.2
B3.3
Yes
No
Unknown/insufficient information provided
B4. Level: Behavior is serious enough during baseline to warrant an intervention (select one of the following)
B4.1
Yes
80
Doctorate in Educational and Child Psychology
B4.2
B4.3
Isabella McDonald
No
Unknown/insufficient information provided
B5. Trend: Behavior is not systematically increasing or decreasing in the desired direction of intervention
effects during baseline.
B5.1
B5.2
B5.3
Yes
No
Unknown/insufficient information provided
Rating of quality of baseline for participant 4: (select 0, 1, 2, or 3):
2
3
1
0
________________________________________________________________________________
Average Quality of Baseline Rating Across Participants:
 of X =
N
X = individual quality of baseline ratings for each participant
N = number of participants in the study
Overall Rating for Quality of Baseline: (select 0, 1, 2, or 3):
3
2
1
0
(Round up or down to the nearest whole number when providing a mean rating for the study. For example, 2.0 to 2.4
rated as 2; 2.5 to 2.9 rated as 3).
C. Measures Support Primary and Secondary Outcomes (list primary outcomes first in alphabetical
order for each participant followed by secondary outcomes. Use extra tables if necessary).
Outcome #1: ____ Number of prompts needed for child to display a social behaviour - robot and human conditions
Primary vs. Secondary
Primary
Secondary
Unknown
Primary
Secondary
Unknown
Who changed
Measurement used to
determine change
What changed
Describe out-come
variables
Participant 1:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Number of
prompts
needed
to show a target
social skill
(Georgia - saying
thank you)
Participant 2:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Number of prompts
needed to show a
target social skill
(Antonia - sharing toys)
81
Treatme
Robot
/human
Interaction
(all)
Doctorate in Educational and Child Psychology
Isabella McDonald
Unknown
Primary
Secondary
Unknown
Participant 3:
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Participant 4:
Primary
Secondary
Unknown
Child
Teacher
Parent/sign. adult
Ecology
Unknown
Behavior
Attitude
Knowledge
Other
Unknown
Self Report
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Self Report
Behavior
Attitude
Knowledge
Other
Unknown
Parent Report
Teacher Report
Observation
Test
Other
Unknown
Number of prompts
needed to show a
target social skill
(Mihnea - saying hello)
Number of prompts
needed to show a
target social skill
(Nicu - sharing toys)
[adapted from Task Force on Evidence-Based Interventions in School Psychology, American Psychology Association,
Kratochwill, T.R. (2003)]
Coding Protocol
Name of Coder:______________________
Date:______04/02/2014____________
Full Study Reference in proper format:_____ Kim, E. S., Berkovits, L. D., Bernier, E. P., Leyzberg, D., Shic, F., Paul, R., &
Scassellati, B. (2013). Social robots as embedded reinforcers of social behavior in children with autism. Journal of
Autism and Developmental Disorders, 43(5), 1038-1049.
Intervention Name (description of study):_____Social robots as embedded reinforcers of social behaviour
Study ID Number:__________________
Type of Publication:
Book/Monograph
Journal Article
Book Chapter
Other (specify):
1.General Characteristics
82
Doctorate in Educational and Child Psychology
Isabella McDonald
A. General Design Characteristics
A1. Random assignment designs (if random assignment design, select one of the following)
Completely randomized design
Randomized block design (between participants, e.g., matched classrooms)
Randomized block design (within participants)
Randomized hierarchical design (nested treatments
A2. Nonrandomized designs (if non-random assignment design, select one of the following)
Nonrandomized design
Nonrandomized block design (between participants)
Nonrandomized block design (within participants)
Nonrandomized hierarchical design
Optional coding for Quasi-experimental designs
A3. Overall confidence of judgment on how participants were assigned (select on of the following)
Very low (little basis)
Low (guess)
Moderate (weak inference)
High (strong inference)
Very high (explicitly stated)
N/A
Unknown/unable to code
B Participants
Total size of sample (start of study): __30____ - Two did not meet the IQ criteria.
Intervention group sample size:___28____
Control group sample size:____28____
83
Doctorate in Educational and Child Psychology
Isabella McDonald
C. Type of Program
Universal prevention program
Selective prevention program
Targeted prevention program
Intervention/Treatment
Unknown
D. Stage of Program
Model/demonstration programs
Early stage programs
Established/institutionalized programs
Unknown
E. Concurrent or Historical Intervention Exposure
Current exposure
Prior exposure
Unknown – Ongoing studies into language development, but not similar to this intervention.
Section 2
Key Features for Coding Studies and Rating Level of Evidence/Support
A Measurement (Estimating the quality of the measures used to establish effects)
(Rating Scale: 3= Strong Evidence, 2=Promising Evidence, 1=Weak Evidence, 0=No Evidence)
A1 The use of the outcome measures produce reliable scores for the majority of the primary outcomes (see following
table for a detailed breakdown on the outcomes)
Yes
No
Unknown/unable to code
A2 Multi-method (at least two assessment methods used)
Yes
No
N/A
Unknown/unable to code
A3 Multi-source (at least two sources used self-reports, teachers etc.)
84
Doctorate in Educational and Child Psychology
Isabella McDonald
Yes
No
N/A
Unknown/unable to code
A4 Validity of measures reported (well-known or standardized or norm-referenced are considered good, consider any
cultural considerations)
Yes validated with specific target group
In part, validated for general population only
No
Unknown/unable to code
Overall Rating of Measurement:
3
2
1
0
B Comparison Group
B1 Type of Comparison group
Typical intervention
Attention placebo
Intervention element placebo
Alternative intervention
Pharmacotherapy
No intervention
Wait list/delayed intervention
Minimal contact
Unable to identify type of comparison
B2 Overall rating of judgment of type of comparison group
Very low
Low
Moderate
High
Very high
Unable to identify comparison group
85
Doctorate in Educational and Child Psychology
Isabella McDonald
B3 Counterbalancing of change agent (participants who receive intervention from a single therapist/teacher etc were
counter-balanced across intervention)
By change agent
Statistical (analyse includes a test for intervention)
Other
Not reported/None
B4 Group equivalence established
Random assignment
Posthoc matched set
Statistical matching
Post hoc test for group equivalence
B5 Equivalent mortality
Low attrition (less than 20 % for post)
Low attrition (less than 30% for follow-up)
Intent to intervene analysis carried out?
Findings_____________
Overall Level of Evidence ____3
3= Strong Evidence
2=Promising Evidence
1=Weak Evidence
86
0=No Evidence
Doctorate in Educational and Child Psychology
Isabella McDonald
C Primary/Secondary Outcomes – Complete the Outcome Tables before completing this section
C1 Evidence of appropriate statistical analysis for Primary Outcomes
Appropriate unit of analysis
Familywise/experimenter wise error rate controlled when applicable
Sufficiently large N
C2 Percentage of Primary Outcomes that are significant
Proportion of significant primary outcomes out of the total primary outcome measures for each key construct.
at least 75%
50-74%
25%-49%
less than 25%
C3 Evidence of appropriate statistical analysis for Secondary Outcomes
Appropriate unit of analysis
Familywise/experimenter wise error rate controlled when applicable
Sufficiently large N
C4 Percentage of Secondary Outcomes that are significant
Proportion of significant primary outcomes out of the total secondary outcome measures for each key
construct.
at least 75%
50-74%
25%-49%
less than 25%
Overall Rating of Analysis:
3
2
1
0
87
Doctorate in Educational and Child Psychology
Significant
Outcomes
Outcome 1
Utterances
in robot
interaction
phase
Primary
vs
Secondar
y
Primary
Secondar
y
Unknown
Who
Changed
Child
Teacher
Parent/Sig.
A
Ecology
What
Changed
Behaviour
Attitude
Knowledg
e
Source
Treatment
Informatio
n
Outcome
Measure
Used
Reliabilit
y
Effect
Size
Self
Report
Promoting
social
interaction
with a robot
using
scripted
cues or
affirmations
Coded
observatio
n from
video
recording
Interrater
reliability
(K=0.88)
d=
1.06
Promoting
social
interaction
with a robot
using
scripted
cues or
affirmations
Coded
observatio
n from
video
recording
Interrater
reliability
(K=0.88)
d=1.5
2
Promoting
social
interaction
with a robot
using
scripted
cues or
affirmations
Coded
observatio
n from
video
recording
Interrater
reliability
(K=0.88)
d=0.6
5
Parent
Report
Teacher
Report
Other
Other
Unknown
Isabella McDonald
Unknown
Observatio
n
Test
Other
Unknown
Outcome 2
Utterances
directed
towards a
partner
Primary
Secondar
y
Unknown
Child
Teacher
Parent/Sig.
A
Ecology
Behaviour
Attitude
Knowledg
e
Parent
Report
Teacher
Report
Other
Other
Unknown
Self
Report
Unknown
Observatio
n
Test
Other
Unknown
Outcome 3
Utterances
directed
towards a
confederat
e
Primary
Secondar
y
Unknown
Child
Teacher
Parent/Sig.
A
Ecology
Behaviour
Attitude
Knowledg
e
Parent
Report
Teacher
Report
Other
Other
Unknown
Self
Report
Unknown
Observatio
n
Test
Other
Unknown
88