Videogame to Support the Teaching of Reading to
Deaf Children using Gamification
Rancés Ramos-Ramirez 1, David Mauricio 1,2
u201312366@upc.edu.pe, dmauricios@unmsm.edu.pe
1 Faculty
2 AI
of Engineering, Universidad Peruana de Ciencias Aplicadas, Lima, Perú.
Group - UNMSM, Lima, Perú.
DOI: 10.17013/risti.n.pi-pf
Abstract: The present work shows the development of an educational
videogame of support for the education of deaf children whose objective is to
reinforce their literacy. In this videogame, the children interact using the
static gestures of the Peruvian sign language, in this way, they complete the
levels previously created by the teacher. In addition, gamification elements
that seek to encourage the participation and competitiveness of children are
used. The use of the videogame in 9 deaf children shows an increase of 22% in
the students' grades, and the SUS test, a high usability of 81/100.
Keywords: gamification; educational videogame; sign language; usability.
1. Introduction
The videogame industry has grown over the years to overcome the film and music
industry (Marchand & Henning-Thurau, 2013), thus, it has been incorporated in
different sectors such as work operations (Madrigal, 2012), marketing (Montola et
al, 2009) and in education (Jungeum, 2012).
Recent research has shown that videogames have a positive impact on children's
education and that these improve writing and reading depending on the thematic
of the videogame (Cano, Arteaga et al, 2015). For deaf children, sign language is
their primary language through which they can learn their mother tongue, which is
necessary to not have limitations in their language skills (Cano et al, 2015).
In some countries technological alternatives have been developed for the teaching
of languages for deaf-mutes, which consider augmented reality, 3d avatars that
simulate gestures, gesture recognition gloves, among others; which have shown
improvements in the teaching-learning process. However, they do not consider
gamification, a concept that creates motivation and interest in the learning process
(Bidarra et al, 2015) and that could improve the teaching-learning process even
more. In this work, a low-cost videogame is developed for teaching Spanish to deaf
children based on gamification, gesture recognition and rote learning.
The article is organized in 5 sections. In section 2, we make a brief review of the
literature on video for deaf children, usability in videogames and gamification. The
proposed game model is presented in section 3. In section 4, the implementation of
the solution is shown. The validation of the proposed videogame is shown in
section 5. Finally, the conclusions follow in section 6.
2. Literature Review
2.1. Videogames for Deaf Children
There are several investigations where videogames that support the education of
deaf children have been developed. In (Zikky et al, 2017), a videogame is proposed
for the teaching of the Indian sign language, where the movement of the hands is
used to play. In (Soares et al, 2015), a videogame is made using Kinect to teach
Portuguese sign language, where the goal is to promote the teaching of this
language to children from 6 to 10 years old. The game has as mechanics to make
the child perform the signs of a certain word that indicated the game while
progressing in the story. A videogame for the teaching of the sign language of
Macedonia, where there was a 3D avatar that made the gestures that the child had
to learn is presented in (Bouzid et al, 2016), where it is shown that the inclusion of
a 3D avatar improves the teaching quality. In (Bouzis et al, 2015), a tool was
created that allowed the teacher to create his own videogame without the need to
know about programming. However, the authors of this research found that this
method of teaching lacked certain important elements in videogames, such as the
visual part, which was not attractive to the players and there were no rewards or
competitiveness between these. A videogame of 2D platforms, which contains 3
game mechanics (3D simulation, memorization, exploration and discovery) for sign
teaching in Macedonia, is presented in (Ackovska & Kostoska, 2014), which shows
that the most successful mechanics is exploration and discovery, that is, the child
was finding packages that contained the signs to learn.
2.2. Usability in Videogames
The usability, according to ISO 9126, is the ability of a software to be understood
and liked by the user when it is used under certain specific conditions (Xu et al,
2014).
In (Bronner et al, 2015), it is said that usability can be measured by the
documentation, help, tutorials and the user interface provided by the application. A
methodology to ensure usability in videogames for deaf children, which considers 5
main factors: efficiency, satisfaction, learning ability, memory facility and low
percentage of errors, is given in (Cano et al, 2018). In this research it was found
that it is important to take the opinions of children and specialists into account,
since children have different difficulties when making gestures or when carrying
out certain activities, thereby reducing the cases in which children can not play
correctly. In (Frutos-Pascual & Zapirain, 2014), a crossword videogame is
developed for children with attention problems, which evaluates its usability using
the System Usability Scale (SUS), where they obtained a score of 87.11 out of 100. A
videogame was developed for the United States Navy, where the usability of this
one was tested using the SUS (Chung & Wu, 2017), obtaining a score of 57.95. In
(Gonzalez, 2016), a racing videogame was made that used low-cost technologies for
wind simulation. In this research the SUS was also used, where they obtained a
score of 85 out of 100. In (Pruna et al, 2017), a videogame was developed for the
rehabilitation of people with disabilities in their hands, which validated the
usability with the SUS, where a score of 79.5 was obtained.
As evidenced by the literature, the SUS is a mechanism widely used to measure
usability, so we will use it to measure the proposed videogame.
2.3. Gamification
The gamification in videogames consists of using game elements, such as points,
rewards, points ranking, game levels, among others, in order to increase the players
participation and motivate them to use the game (Figueroa, 2015). In this
investigation, gamification is used to learn a second language, which describes
various elements of gamification that can be used in a video game such as scores
obtained while playing, rewards for reaching certain objectives, score comparison
tables, avatars that represent the user , system of levels, progress bars, among
others. This work recommends the five-step model for the gamification of (Huang
& Soman, 2013): understanding the needs of the target audience, defining learning
objectives and game requirements, structuring the experience, identifying the
resources for gamification and applying the elements of gamification. In (Bidarra et
al, 2015), an educational videogame is proposed for the Portuguese sign language
where gamification is used, and a system of levels is used through which progress is
made based on the objects obtained by the player and a scoring system based on
the game time.
3. Videogame Model
It is proposed a game model that can be customized by the class teacher, so that he
can freely create levels of game that adjusts to the needs of the moment, that is, he
can create games according to the theme that the children require to learn. The
model has as components the actors, the basic module, the arcade module, the
crossword module, the teacher module and the gestures Peruvian sign language
processor. This model can be seen in figure 1.
Figure 1 – Proposed Game Model
In order for the model to work correctly, first, the teacher must upload the class
subject information corresponding to the basic module through the module of the
teacher. The child with hearing disability will learn with the videogame, interacting
with the basic module and the game modules. The basic module is intended to
reinforce the spelling in the sign language and the reading of words in Spanish, this
is done as follows. First, the video game shows the letter or word to spell, and then
the child makes the corresponding gesture. Second, the video game captures the
image of the gesture by using the webcam and then sends it to the gesture
processor. Third, the latter processes the image of the gesture and obtains the
spelled letter. Fourth, the spelled letter is shown on the interface, and if it is
correct, the child is rewarded. In a similar way, the child interacts with the game
modules to reinforce learning, but now in a game context.
3.1. Actors
Deaf children and teachers where the videogame is played are our actors. The
teacher is responsible for feeding the game based on the lessons he wants to take in
class. The children are the main players, who will have a personal account in which
they will keep all their scores and all the medals obtained throughout the use of the
videogame.
3.2. Basic Module
This module consists of reinforcing the gestures of sign language and learning, little
by little, how words are spelled, thus fulfilling the basic needs of deaf children. The
data of this module belongs to the information loaded by the teacher. Using the
challenge and sensory immersions, seen in (Kwon & Woo, 2013), the reward-time
gamification element was implemented, through which it was sought to slightly
pressure the child to fulfill its objective.
3.3. Arcade Module
The arcade module seeks to work the user's memory, giving words to spell until the
time runs out. The gamification elements used in this module are score, medals and
time limit. With these gamification elements it is sought that children are
competitive with each other; thus, they will want to overcome each other while
unconsciously working their memory.
3.4. Crossword Module
The crossword module, like the previous module, seeks to work the memory of the
child, making him remember the words seen in class sessions. The crosswords of
the game are thematic and cover the basic knowledge for children. As elements of
gamification, the reward for time was used, where the child starts with 5 medals,
which decrease over time.
3.5. Teacher Module
The teacher module is used to upload information, which will allow you to freely
design the game levels, both for the learning module and for the arcade module,
giving you the facility to create class sessions focused on specific topics.
The teacher is responsible for loading the information of the basic module by using
this module. It was taken into account that teachers are allowed to choose the topic
to teach and thus create learning phases, which contain letters and words that
children will have to spell during the basic game, in such a way that they can cover
different topics such as months of the year, the days of the week, among others.
3.6. Gesture Recognizer
This component will be responsible for processing the gesture made by the user,
which is received from the videogame. It was taken into account that the teacher,
through the use of the gesture dataset, which contains the static gestures of the
Peruvian sign language alphabet, obtains the result of the processed gesture, which
is returned in response to the videogame. The dataset of gestures, provided by
Berru (Berrú et al, 2017), only contains 24 letters of the alphabet, where the letters
J, Ñ and Z are omitted, since they are represented with dynamic gestures.
4. Videogame: EnSeñas
4.1. Logical Architecture
The logical architecture of the game, showed in figure 2, has two main components:
the videogame and the API that contains the gesture processor. Both components
work on the Windows 10 operating system. It was thought to work the gesture
recognizer as a separate server, in order to have this in the cloud and user it as a
service. The advantage of this is that, if the game starts to be used by several
institution, all the game instances could be connected to this server, allowing
improvements to the dataset or gesture recognizer, making all game instances be
benefited.
Figure 2 – Logical Architecture of Videogame EnSeñas
4.2. Technologies Used
For the development of the game, Unity v 2017.3.0f3 (64-bit) was used as a game
engine, together with C # as a programming language. In addition, we used
websocket-sharp to create a websocket client in Unity, allowing us to establish a
connection with the gesture recognizer. Also, we used Newtonsoft to format the
data type to JSON, which is a lightweight format to transport and store data.
For the gesture recognizer, Python v 3.7 was used, together with the Tornado web
framework v 5.1.1 in order to create a websocket server, allowing the gesture
recognizer to receive all the videogame requests and responding accordingly.
OpenCV was used in the gesture recognizer module to process the images received
from the websocket server.
4.3. Development Methodology
For the development of the project, the Scrum methodology was used. For this, a
product backlog was made with the tasks to be developed, prioritizing them by
importance and difficulty. There were 7 sprints, where the artifacts generated in
each one were validated with the association of deaf children who helped us during
the investigation.
4.4. Game Modules
User Login
When the game was designed, it was taken into account that it was necessary to
keep information of each player, their starts, their attempts, etc. and at the same
time, it was necessary to give total control to the teacher, who will need to create
game phases. Therefore, a user registration and a login module were created.
Teacher Module
In the teacher module, it is allowed to create phases or levels of game, which can be
edited and eliminated at any time. The phase creation screen varies depending on
which module the teacher wants to configure. If it is the basic module, the teacher
will be asked to enter letters and words, the latter formed based on the letters
added at the same level and at previously created levels. If it is the arcade module,
you will only be asked to enter words without restrictions. This phase creation
screen can be seen in figure 3.
Figure 3 – Basic Module Phase Creation Screen
Basic Module
The game interface is simple, in order not to distract the child while playing. In the
interface is shown the word or letter that the child must spell, the stars that the
child has and the time it has before losing of the stars. As time goes by, the green
bar will decrease until it is empty, where the child will lose his first star, then the
green bar will fill up again and continuously until the child runs out of stars. With
this, we want to measure how well and quickly the child was able to finish the phase
of the game.
Arcade Module
This game module seeks to reinforce the knowledge of the child by spelling words
constantly. The child has a time limit to spell the word that is asked, which
decreases every second and increases each time a word is finished. A scoring
system is managed, which is awarded based on the number of letters of the word
that the child spells. Also, stars will be awarded to the child when it reaches a
certain score, up to a maximum of 5 stars (see figure 4). It was sought that the
teacher can also customize this mode, in such a way that he can create thematic
levels of game or of different difficulties. That is why we also created a teacher
module to edit the arcade module.
Figure 4 – Arcade Module Play Screen
Crossword Module
In this game module there is a thematic crossword, where the child is presented
with several animals, whose images are shown in the initial game interface. These
images can be pressed by the child to begin the spelling of the respective animal. To
do this, the initial box of the animal inside the crossword puzzle is lit. In addition, a
panel is shown where the image captured by the web camera is captured, the figure
of the selected animal and the letter that is detected by the gesture recognizer. This
panel can be seen in figure 5. Once the child finished spelling the animal, the figure
disappears from the list. The player starts with 5 stars, which decrease as time
passes, this mechanics being similar to the basic module. With this, we seek to
know the time it takes the child to solve the crossword puzzle, measuring the effort
it takes to finish it. This module was created in order to see the impact it has on
children, so that, if positive, we can create a personalization of crosswords puzzles
in the future.
Figure 5 – Crossword Module Play Screen (Camera Panel)
Game Ranking
Each game mode has its own ranking, which is filled from the attempts made by
each player. Each attempt is recorded at the end of the game, thus, the game time,
the attempt number and the stars obtained are saved. This can be seen in the figure
8. With the implementation of this module, we seek to create competitiveness in
children, which is a way to motive them to continue playing, reinforcing their
knowledge. It is important to keep the record of each attempt per child, because it
is a measure of the improvement that has been throughout the game at that level.
5. Validation
Game sessions were conducted to test how effective the proposed videogame is in
the learning process, in addition, System Usability Scale was used to measure its
usability.
5.1. Experiment
The validations and tests of the videogame were made in the ENSEÑAS PERÚ
association, in a class with 9 children between the ages of 8 to 12 years, as we can
see in figure 6. The children in the class to be tested had difficulty learning the
names of the months of the year, of which 4 out of 9 children got a passing grade.
Figure 6 – Childs Playing EnSeñas Game During Experiment
5.2. Learning Effectiveness
A normal class session of 2 hours on the writing in Spanish of the months of the
year for the 9 deaf children was made. Then the knowledge test was taken to
measure what was learned from each student, as seen in table 1, which is calculated
from the correct writing of the months. Then there were two sessions of the use of
videogames in the classroom, in the first session there was a brief explanation of
the videogame and its operation, and each child was allowed to play 8 minutes
twice in the arcade mode. In the second session, each child was allowed to play 8
minutes in arcade mode and crossword mode. At the end of the second session of
the game, the knowledge test was again taken, where the notes shown in table 1
were obtained.
The use of the videogame helped the children to memorize the writing of some
months of the year. As can be seen in Table 1, there was an average increase of 2.37
points in the evaluation note taken after two sessions of videogame use.
5.3. Usability
The SUS is a 10 questions questionnaire where the user's experience of the game is
evaluated, and where he was asked to answer on a scale from totally disagree to
totally agree (1 - 5) (Lewis & Sauro, 2009).
The teacher was invited to know and use the videogame in the three videogame
modules and the teacher's module, in a period of 30 minutes.
Table 1 – Results of the Test of the Months of the Year
Person
First Evaluation
Second Evaluation
Child 1
11.62
14.94
Child 2
13.28
14.94
Child 3
20
20
Child 4
16.6
18
Child 5
9
3
Child 6
6.4
8.3
Child 7
11.62
14.94
Child 8
0
4.98
Child 9
18.26
20
Average
10.86
13.23
SD
7.37
5.96
Table 2 – Usability Results of SUS Evaluation
Person
SUS
(x-mean)
(x-mean)^2
Child 1
82.5
1.5
2.25
Child 2
77.5
-3.5
12.25
Child 3
82.5
1.5
2.25
Child 4
85
4
16
Child 5
77.5
-3.5
12.25
Child 6
75
-6
36
Child 7
82.5
1.5
2.25
Child 8
82.5
1.5
2.25
Child 9
77.5
-3.5
12.25
Teacher
87.5
6.5
42.25
Mean
81
SD
3.94
At the end of the knowledge test, the evaluation of usability was carried out through
the SUS, for this, the teacher asked the questions in both Spanish and sign
language to all participating children, who answered the questionnaire for each
question. The results of the SUS for the children and the teacher are shown in table
2.
An 81 of 100 points was obtained (M = 81, SD = 3.94), which, compared to past
research, it can be said that the game has a good usability.
6. Conclusions
In this work we introduce a videogame model that uses gamification and image
processing for the teaching of Spanish to deaf children, as well as its
implementation through a videogame that includes a teaching module with three
videogames, a teacher module to load teaching topics and a gesture recognizer.
The use of the videogame in 10 children shows an improvement in the learning
process, which is reflected in the 22% increase of the average grade. In addition, the
SUS usability test shows that the game has a usability of 81/100, above many
videogames of the literature.
Acknowledge
The authors thank the Universidad Peruana de Ciencias Aplicadas (UPC), for the
partial funding of the present investigation, the ENSEÑAS PERÚ association, and
Bryan Berrú, for providing us their Peruvian sign language gestures dataset.
References
Ackovska, N., & Kostoska, M. (2014). Sign language tutor — Rebuilding and
optimizing. 2014 37th International Convention on Information and
Communication Technology, Electronics and Microelectronics.
Berru-Novoa, B., Gonzalez-Valenzuela, R., & Shiguihara-Juarez, P. (2018).
Peruvian sign language recognition using low resolution cameras. 2018 IEEE
XXV International Conference on Electronics, Electrical Engineering and
Computing (INTERCON). doi:10.1109/intercon.2018.8526408
Bidarra, J., Escudeiro, P., Escudeiro, N., & Barbosa, F. (2015). Game Design of
Content: Assessing a Project for Learning Sign Language. Conference:
EDULEARN 2015 - 7th International Conference.
Bouzid, Y., Khenissi, M. A., & Jemni, M. (2015). Designing a game generator as an
educational technology for the deaf learners. 2015 5th International
Conference on Information & Communication Technology and Accessibility
(ICTA). doi:10.1109/icta.2015.7426914
Bouzid, Y., Khenissi, M. A., & Jemni, M. (2016). The Effect of Avatar Technology on
Sign Writing Vocabularies Acquisition for Deaf Learners. 2016 IEEE 16th
International Conference on Advanced Learning Technologies (ICALT).
doi:10.1109/icalt.2016.127
Bronner, S., Pinsker, R., & Adam Noah, J. (2015). Physiological and
psychophysiological responses in experienced players while playing different
dance exer-games. Computers in Human Behavior, 51, 34–41.
doi:10.1016/j.chb.2015.04.047
Cano, S., Arteaga, J. M., Collazos, C., & Amador, V. B. (2015). Aplicación Móvil para
el aprendizaje de la lectoescritura con Fitzgerald para niños con discapacidad
auditiva. Anais Dos Workshops Do IV Congresso Brasileiro De Informática
Na Educação (CBIE 2015). doi:10.5753/cbie.wcbie.2015.240
Cano, S., Arteaga, J. M., Collazos, C. A., & Amador, V. B. (2015). Model for Analysis
of Serious Games for Literacy in Deaf Children from a User Experience
Approach. Proceedings of the XVI International Conference on Human
Computer Interaction - Interacción 15.
Cano, S., Collazos, C. A., Aristizábal, L. F., Gonzalez, C. S., & Moreira, F. (2018).
Towards a methodology for user experience assessment of serious games with
children with cochlear implants. Telematics and Informatics, 35(4), 993-1004.
doi:10.1016/j.tele.2017.09.011
Chung, S.-M., & Wu, C.-T. (2017). Designing Music Games and Mobile Apps for
Early Music Learning. In Serious Games and Edutainment Applications (pp.
57–75). doi:10.1007/978-3-319-51645-5_3
Figueroa, F. (2015). Using Gamification to Enhance Second Language Learning.
Frutos-Pascual, M., & Zapirain, B. G. (2014). Guided crossword-puzzle games
aimed at children with Attentional Deficit: Preliminary results. 2014 Computer
Games: AI, Animation, Mobile, Multimedia, Educational and Serious Games
(CGAMES). doi:10.1109/cgames.2014.6934137
Gonzalez, J. (2016). Una Propuesta Metodológica para la Gamificación de Pruebas
Psicométricas. Un Caso Práctico. Congreso Ambiental Sobre Soluciones en
Ingenieria Ambiental, Ingenieria de Software y Salud Electrónica y Móvil.
Huang Hsin Yuan, W. and Soman, D. (2013). A Practitioner’s Guide to
Gamification of Education. Research Report Series: Behavioral Economics in
Action. University of Toronto –Rotman. School of Management.
Juan Madrigal, Sustainability Mobility CloudApps SuMo: The Way Appcelerator
Titanium is Helping Save the World, Think Mobile Blog
Jungeum Lee, Study on the successful case of gamification and its application to eBook, Policy Industrial Graduate School, Sookmyung Women's University,
2012.
Kwon, C., & Woo, T. (2013). A Research on Gamification Methodology for Korean
Language Education. Journal of Korea Game Society, 13(1), 61-74.
doi:10.7583/jkgs.2013.13.1.61
Lewis, J. R., & Sauro, J. (2009). The Factor Structure of the System Usability Scale.
Human Centered Design Lecture Notes in Computer Science, 94-103.
doi:10.1007/978-3-642-02806-9_12
Marchand, A., & Hennig-Thurau, T. (2013). Value Creation in the Video Game
Industry: Industry Economics, Consumer Benefits, and Research
Opportunities. Journal of Interactive Marketing, 27(3), 141–157.
doi:10.1016/j.intmar.2013.05.001
Montola, Markus, Stenros, Jaakko, Waern, Annika, Pervasive Games: Theory and
Design, Morgan Kaufmann Publishers, 2009. doi:10.1201/9780080889795
Pruna, E., S., A. A., Escobar, I., Pérez, S. A., Zumbana, P., Meythaler, A., & Álvarez,
F. A. (2017). 3D Virtual System Using a Haptic Device for Fine Motor
Rehabilitation. Advances in Intelligent Systems and Computing Recent
Advances in Information Systems and Technologies, 648-656.
doi:10.1007/978-3-319-56538-5_66
Soares, F., Esteves, J. S., Carvalho, V., Lopes, G., Barbosa, F., & Ribeiro, P. (2015).
Development of a serious game for Portuguese Sign Language. 2015 7th
International Congress on Ultra Modern Telecommunications and Control
Systems and Workshops (ICUMT). doi:10.1109/icumt.2015.7382432
Xu, J., Ding, X., Huang, K., & Chen, G. (2014). A Pilot Study of an Inspection
Framework for Automated Usability Guideline Reviews of Mobile Health
Applications. In Proceedings of the Wireless Health 2014 on National
Institutes of Health - WH ’14. doi:10.1145/2668883.2669585
Zikky, M., Hakkun, R. Y., Rizqi, A. F., Hamid, A., & Basuki, A. (2017). Development
of Educational Game for Recognizing Indonesian Sign Language (SIBI) and
Breaking Down the Communication Barrier with Deaf People. 2017 21st
International Computer Science and Engineering Conference (ICSEC).
doi:10.1109/icsec.2017.8443936