Facial Expression Wonderland (FEW) — A Novel Design
Prototype of Information and Computer Technology (ICT)
for Children with Autism Spectrum Disorder (ASD)
Rung-Yu Tseng
Ellen Yi-Luen Do
College of Architecture,
Georgia Institute of Technology
Atlanta, GA 30303, USA
+1-626-673-8985
College of Architecture & School of Interactive
Computing, Georgia Institute of Technology
Atlanta, GA 30332, USA
+1-404-385-5041
rytseng@gatech.edu
ellendo@gatech.edu
Autism Spectrum Disorder (ASD) is one of the Pervasive
Developmental Disorders (PDD). Significant deficits in the
children with ASD include lack of social communication skills
and cognitive dysfunction. There is currently no cure for autism;
however, with appropriate aid, autistic children could learn
progressively and maintain a good quality of life with their
disorder. Therefore, the market demands Information and
Computer Technology (ICT) applications to facilitate children
dealing with ASD in their lifetime. Although many ICT designs
have been developed and used already in early intervention and
treatment, they are not common in long-term training for the
children with ASD. The proposed work here presents a novel
design prototype, Facial Expression Wonderland (FEW), to train
the children with ASD based on the progressive levels of training
under a given background context. This prototype is designed to
improve the ability of the ASD children in facial expression
recognition. This work also discusses how ICT can facilitate the
life further for these youngsters with ASD.
recognition and Theory of Mind (ToM), language delay, and rigid
behaviors [3]. The result from previous behavioral experiment has
shown that ASD children have no statistically significant
difference in sorting upside down faces and objects [16]. Through
the eye movement monitoring system, researchers also found that
ASD children viewed non-feature areas more than core features
when seeing faces [13]. The deficits in social communication
limit the life quality of the people with ASD due to the
unawareness of social cue. The shortfall in facial expression
recognition is particularly a limiting factor among those social
skills to interact with people. There exist some solutions to
maintain their life in equal quality, nevertheless. Information and
Computer Technology (ICT) is able to accomplish many tasks
autonomously and efficiently. The paper of Philipp Michel
initially directed and launched studies in applying ICT to ASD
from a technical point of view [8]. This provides an overview of
the assistive ICT for the children with ASD in different levels of
technology and evidence that ICT assists people with disability,
especially in ASD.
ABSTRACT
ICT has been used for a long time to diagnose and assess many
kinds of disorders, including ASD; for instance, the Wisconsin
Card Sorting Test to assess executive dysfunction [5]. Table 1
lists the advantages of ICT designs over its traditional counterpart.
Clearly, ICT designs have significantly greater impact on people
requiring specific needs. Since the ASD people have monotropic
interest system and they prefer an environment with restricted
stimuli, stable, clear-cut boundary, controllable and text-free
conditions [10], the computers are suitable for people with ASD.
According to the specific user preference, the advantages of ICT
can meet the particular use for the children with ASD, such as the
multi-prompts that allow ASD children to pay attention except the
visual instruction.
Categories and Subject Descriptors
H5.2. Information Interfaces and Presentation (e.g., HCI): User
Interfaces: Training, help, and documentation
General Terms
Design, Human Factors
Keywords
Information and Computer Technology, Autism Spectrum
Disorder
1. INTRODUCTION
Several studies have employed ICT to facilitate or train ASD
children. In the Ph.D. dissertation by Miranda, J.C., avatars were
used to provide training in the rehabilitation of ASD children [9].
The proposal focuses on developing the algorithm of face
simulation and agrees that animated facial expressions are more
flexible in the training session. Another facial expression
recognition toolkit is ‘Let’s Face it!’, the psychological battery to
teach ASD children facial recognition skills with human faces
[15]. This battery has already been used in several neuroimaging
studies as stimuli. The aforementioned studies suggest that ICT is
an effective approach to help children with ASD. However, none
Autism Spectrum Disorder (ASD) is typically characterized by
the lack of social skills, cognition deficits in facial expression
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
IHI’10, November 11–12, 2010, Arlington, Virginia, USA.
Copyright 2010 ACM 978-1-4503-0030-8/10/11...$10.00.
464
of these examples provides a fully training program with different
levels under a background context. The goal of the proposed work
is to present a more thorough and complete ICT prototype design
to facilitate children with ASD using the novel computer game,
Facial Expression Wonderland (FEW).
Table 1. The advantages of ICT designs
z
z
z
z
z
z
z
z
Higher accuracy
Multi-prompts choices
High endurance to perform long-period session
Long term archive of data
Easy to compare data among cross-groups
Easy to process complicated sessions
Does not require social interaction
Save time and money in administration
2. PROTOTYPE
‘FEW’ is a novel design prototype of a training program for the
children with ASD in facial expression recognition. The acronym
FEW comes from ‘Facial Expression Wonderland’ with
expectation to enhance necessary cognitive abilities in ASD
children. The goal of FEW is to train children with ASD in facial
expression recognition via playing game since children are
generally to enjoy and concentrate more in gaming. Figure 1
pictures the elements contributed to FEW and highlights the
feature of it. FEW is designed to familiarize ASD children using
given facial expressions under background context. The animated
characters in this game are expected to draw more attention and
motivate ASD children to participate the game. In addition, the
training program takes the advantage of voice prompt and
instructions rather than textual message to guide them through the
entire game. FEW has different training levels to drill ASD
children efficiently. Once the ASD children pass a level, they
proceed to the next level. Every new level increases the difficulty
in addition to the previous level and requires more skills to get to
the next level. Consequently, the proposed training program
provides ASD children problem-solving skills, flexibility and
independence.
Figure 1. The elements consist of the prototype ‘Facial
Expression Wonderland’, FEW.
The voice instructions of FEW are different in each level and
children could repeatedly listen to it by clicking the sound button.
Table 2 lists the content of the voice instructions in FEW. Since
children with ASD have deficits in social interaction, the voice
instruction guides in a clear way to keep children involved in the
game. In addition, voice prompt cues children to look at the facial
expression carefully as a reminder to keep children concentrate on
the game itself.
Table 2. Content of the voice instructions within FEW
FEW is a computer game based on the film ‘Alice In
Wonderland’ by Walt Disney [1]. The target users of FEW are
children with ASD from preschoolers to 4th grade students.
During the first several times, ASD children should play the game
with a well-trained supervisor, a therapist or parents. After
becoming familiar with FEW, ASD children would be able to
play the game independently. Moreover, FEW is a daily training
program, which requires ASD children to participate everyday in
order to improve skills necessary to function in their environment.
To avoid habituation, the computer records the time that a player
spends on each level in every trial and controls the run time
within one hour per day. Three different training levels are
embedded in the game and all participants will start the game
from first level. Each training session contains six tasks in first
level, one task in second and third levels, respectively. Then
children move on to the next level when passing the previous
level by over 90 percent of the accuracy rate. Once failed in lower
level, children restart the game from the same level till the level
is cleared.
465
Onset
Content of Voice Instruction
Beginning of
the game
Welcome, please choose the character you
would like to use in the game
Beginning of
the game
Please choose the story you would like to
experience this time
Level 1, 2, 3
Please look at his/her face carefully
Level 1
Please choose the facial expression different
from/the same with the one on his/her face
Level 2
Please click on the picture of food made
you have the same facial expression with
him/her
Level 3
Please click the correspondent facial
expression from the pool below and put
them into the cell
either the same or different facial expressions. The buttons for
“voice instruction” and “return to last page” appear on the top of
the screen during the whole training session (refer to Figure 2).
Children are allowed to repeatedly listen to instructions and freely
return to the previous step.
2.2 The Second Level in FEW
The second training level is implemented based on ASD
children’s preference in daily life. The item of preference here
targets a variety of food. Generally, food is used in almost every
behavioral study as a reward, and researchers usually use food in
the experiment to encourage and get children participants
involved [2]. However, in case that the children have specific
taste in food, a list of food items is prepared for parents or
caregivers in order to use proper food items for each participant
accordingly.
Figure 2. The first level of FEW trains the basic ability for
facial expression recognition.
2.1 The First Level in FEW
Prior to starting the game, ASD children choose a character from
the story and a background story associated with the character.
There are four background stories available and each corresponds
to a different game set. After making the decision, the game will
begin immediately. The first level is designed to train basic
abilities for facial expression recognition; this level contains two
sub-levels. Figure 2 shows the serial scenes from one of the firstlevel training modes. The top picture shows the target of this task,
a facial expression of Cheshire cat, followed below by a pool of
various expressions in the middle. The only possible answer is
shuffled in the pool. When the bottom picture appears, the
participants need to follow voice instructions to choose their
answer. The objectives in this level are to recognize the SAME
and to distinguish the DIFFERENT facial expressions,
respectively. FEW tests if these two abilities are the same degree
of malfunction in ASD children. During the six tasks in this level,
the software randomly presents three tasks to detect the same
expressions and three to distinguish the different expressions.
Voice instruction in addition guides ASD children to choose
Figure 3. The second level of FEW chooses the food in a five
by five checkerboard depend on the given facial expression.
In this level, the screen displays a five by five checkerboard with
pictures of different kinds of food in the cells as shown in Figure
3. ASD children then are asked to put check marks on the pictures
within the checkerboard that would correspond to a given facial
expression. For instance, if there is a ‘disgusting’ facial
expression, then the participants need to pick the food that would
466
make them feel the same way. In the example of Figure 3, the
facial expression implies that the checked food should keep the
ASD children happy. To pass this level, children also are
expected to apply the ability into their real life in addition to
differentiate various facial expressions which is already welltrained from the previous level. That is, children have to connect
their preference in food to make the choice to complete this level.
This level trains ASD children gradually to understand how to
represent mental status of others and this is exactly what the
researchers called ‘Theory of Mind’. The clip provides children a
background to realize why the characters in the picture have
different moods in response to the changing situation.
3. DISCUSSION
FEW, ineluctably, might still have several limitations that need to
be addressed when applying to the children in reality. One is the
appropriateness of the selection of facial expression and
background context. The facial expression in Psychology usually
uses a set of module: Facial Action Coding System (FACS)
developed by Dr. Paul Ekman [12]. The cartoon expression in
FEW might encounter problem to define the baseline of the
emotion in testing. In addition, the real emotions of people are
sometimes confusing even to the ordinary observers. Therefore,
there should be a proper selection of the emotions to be used in
FEW. The increase in difficulty between each level could also be
an issue unless it is verified with real experiment. Another
question would be that if the ASD children could make correct
eye-gaze on the picture to detect the change of facial expression.
This could be solved by adding the device ‘Eye Tracker’ during
the game.
2.3 The Third Level in FEW
Before entering the last level of FEW, it requires children to
watch a two to three minutes animation clip based on the
background context chosen in the beginning of the game. After
experiencing the story with character, children are ready to start
the third level. This level is more complicated than the previous
parts. The objective of this level is to recognize the shift of facial
expressions through changing the story line. As seen in Figure 4,
the screen serially displays two pictures from the clip that
children just watched. These two pictures are related to each other
so that the ASD children have to detect the changes of the
character’s facial expression in the pictures. Next, the children
need to pair up the appropriate facial expression from the pool of
pictures below to fit the facial expression in the picture.
Despite the concerns mentioned above, FEW demonstrates a
novel approach to train ASD children via computer. ‘Play’ is the
most basic component in children’s life and FEW is just the
playable training toolkit for children. This feature of different
levels in FEW is suitable for ASD children who are individually
different from one another since the disorder is a wide spectrum.
By the background context in FEW, this program would be
advanced the ecological validity to mimic part of real interaction
and reflect the training effect into daily life. Also, the second
level of FEW connects training session with the participants using
personalized food list; individual difference is, again, adjusted
here. This pilot ICT design specifically for children with ASD
opens a window to both the designers and caregivers to facilitate
the life of ASD children, requiring a great amount of efforts and
development to deliver to those children. In the future, FEW is
anticipated to improve ASD children’s abilities in facial
expression recognition and ToM.
4. CONCLUSION AND FUTURE WORK
The prototype ‘FEW’ presented in this paper is designed to mend
the deficits of ASD children in facial expression recognition and
ToM. It employed animation as background context to set up the
frame of the game. The nonverbal elements motivate children
with ASD to stay involved in the game [11]. This novel design
prototype is an initial idea to expand the applications of ICT
designs for the ASD children in long-term training. The real effect
of FEW is not yet estimated and is to be validated in the future.
While researchers and designers are trying to create more ICT
facilities for the children with ASD, it is important to pay
Figure 4. The last level of FEW pairs the appropriate facial
expression under different contexts.
To complete this training level, ASD children have to possess the
abilities of facial expression recognition and ToM. For instance,
Alice fell into the hole of tree with a scared look but when she
saw a white rabbit ran in front her, she forgot the fear and
suddenly became curious about what the rabbit was going to do.
467
[3] Diagnostic and Statistical Manual of Mental Disorders :
DSM-IV (4th ed.). American Psychiatric Association,
Washington, DC, USA, 1994.
attention to the issues of increasing overloaded difficulty during
the use by the ASD children. Additionally, since ASD includes a
wide spectrum of symptoms, researchers and designers should
consider the individual life style of the children with ASD into
their design. It might still be difficult to create a design to apply to
all children with ASD. As a designer of the proposed training
prototype, some degree of flexibility can be further implemented
such as making the designs more personalized. Meanwhile,
designers and researchers need to have more knowledge about the
ASD children’s life and the disorder itself. There should be an
awareness that they are working with the real people of high
specialty and this would help create the facilities more applicable
for the use of the children with ASD [7].
[4] Hardy, C., Ogden, J., Newman, J. and Cooper, S. Autism and
ICT. David Fulton Publisher Ltd, London, UK, 2002.
[5] Harris, M.E. Wisconsin Card Sorting Test: Computer
Version 1.0. Psychological Assessment Resource, Odessa,
FL, USA, 1990.
[6] Ijichi, S. and Ijichi, N. Computerized Lifelong Mentoring
Support Using Robot for Autistic Individuals. Med
Hypotheses 68, (2007), 493-498.
[7] Kientz, J.A. and Abowd, G.D. When the Designer Becomes
the User: Designing a System for Therapists by Becoming a
Therapist. Ext. Abstracts CHI 2008, ACM Press (2008),
2071-2078.
This work manifests how ICT is able to help ASD children in
many aspects. However, ICT itself is just a tool; the important
thing is to use the tool effectively about the different symptoms of
ASD children. In an experiment of the robotic dog, AIBO, the
researchers suggest that the reason might be because AIBO is
similar to much of nature so ASD children interact more with
AIBO [14]. Does this imply that children with ASD have a desire
to interact with outer world as well? ICT should provide the
channel to connect children with ASD and the environment
outside. Dr. Ijichi came up with an interesting idea, ‘Jiminy
Cricket’, for ASD children with lifelong mentor [6]. Although it is
an initial idea, it addresses an important issue that there should be
some solutions to serve the children with ASD throughout their
life since ASD is currently not curable. It would be the
responsibility of all researchers and designers to find out how ICT
could do more for the future of children with ASD.
[8] Michel, P. The Use of Technology in the Study, Diagnosis,
and Treatment of Autism (2004).
http://www.cs.cmu.edu/~pmichel/publications/AutismTechn
ology.pdf
[9] Miranda, J.C. Interaction Between Virtual Characters and
Humans or Others Avatars in Rehabilitation Domain. Ph.D.
Dissertation Proposal, The Faculty of Engineering of the
University of Porto, Proto, Portugal, 2008. DOI=
http://paginas.fe.up.pt/~aas/pub/Aulas/PI/Avaliacao/Ano07_
08/JoseCarlosMiranda.pdf
[10] Murray, D.K.C. Autism and Information Technology:
Therapy with Computers. In S. Powell and R. Jordan (Ed.),
Autism and Learning (pp. 100-117). David Fulton Publishers
Ltd, London, UK, 1997.
[11] Ozonoff, S. Reliability and Validity of the Wisconsin Card
Sorting Test in Studies of Autism. Neuropsychlogy 9, 4
(1995), 491-500.
In the guidebook, Autism and ICT [4], the authors refer the ‘C’ as
‘Communication’ but not ‘Computer’. The methods mentioned in
the book are still expected to perform via computer, however.
This idea comes from the interviews with people around ASD
children, parents, caregivers, teachers and etc. All of these people
refer the word ‘computer’ as a positive feeling and pleasure. They
all agree that computers can greatly alter the world of the children
with ASD. In addition, they all believe children with ASD love
computer! The discussion of the relationship between ASD and
ICT then must not be a closure here. There will be more and more
ICT designed specifically for ASD children because ICT can truly
complete the real life for children with ASD.
[12] Paul Ekman. Facial Action Coding System (FACS). DOI=
http://www.face-andemotion.com/dataface/facs/description.jsp
[13] Pelphrey, K.A., Sasson, N.J., Reznick, J.S., Paul, G.,
Goldman, B.D. and Piven, J. Visual Scanning of Faces in
Autism. J Autism Dev Disord 32, 4 (2002), 249-261.
[14] Stanton, C.M., Kahn, P.H., Severson, R.L., Ruckert, J.H. and
Gill, B.T. Robotic Animals Might Aid in the Social
Development of Children with Autism. Proc. 3rd ACM/IEEE
Intl Conf. Human Robot Interaction, ACM Press (2008),
271-278.
[15] Tanaka, J.W., and Schultz, R.T. The Let’s Face It! Skills
Battery. Unpublished assessment, University of Victoria, BC,
Canada, 2008. DOI= http://www.oberlin.edu/newsinfo/02jul/autism_research.html
5. REFERENCES
[1] Alice in Wonderland (1951) on film, the Walt Disney
version. DOI=
http://disney.go.com/characters/#/characters/classics/alicein
wonderland/
[16] Tantam, D., Monaghan, L., Nicholson, H. and Stirling, J.
Autistic Children's Ability to Interpret Faces: A Research
Note. J Child Psychol Psyc 30, 4 (1989), 623-630.
[2] Birch, L.L., Zimmerman, S.I. and Hind, H. The Influence of
Social-Affective Context on the Formation of Children’s
Food Preferences. Child Dev 51, 3 (1980), 856-861.
468