L a n ~ o a q eResources and Visual Communicatiorm in a Deaf-Centered
aultiniodal E-Learning Environment: Issues to be Addressed
'
Elena Antinoro ~izzuto',Claudia S. Bianchini
Daniele Capuano Gabriele Gianfreda '. ,
Paolo ~ o s s i n i '
' Istituto di Scieiize e Tecnologie della Cogiiizioiie. Corisiglio Nazionale delle Ricerclie. Via Nonieiitana. 56 - 001 61,
Roma. 'Uiiiversité Paris 8, Rue de la Liberté, 2 - 93526 St.Deiiis CEDEX 1 DIFILILE. Università di Perugia, Piazza
Morlacclii, I - 06 123. Perugia, 1taly. Università di Roma "Sapieiiza7'. Dipartiiiiento di Informatica. Pictorial
Computiiig Laboratory, Via Salaria 113, 00198 ~orna.%ipartimeiito di Scienze dell'Educazioiie e della Formazione,
U~liversitàdi Macerata, P.le Bertelli, 1. C.da ValIebona, 62 100 Macerata - Italy,
This paper examiiies some of tlie rnajor problems linked to tlie task of designiiig appropriate iiiultiliiigual e-learning
eiivironments for deaf leariiers (DL). Due to their liearing disability most DL experieiice drainatic difficulties in
acquiriiig appropriate literacy skills. E-leat-iliiig tools could in priiiciple be very useful for facilitatiiig access to
web-based lcnowledge and pronioting literacy developnient in DL. However, designing appropriate e-learning
enviroiimeilts for DL is a complex task especially because of tlie different liilguistic baclcground aiid experience DL
inay have, a~idof the inultimodal language resources that need to be provided and iiitegrated (e.g. lailguage produced iii
the visual-gestural or sigiled inodaiity, i11 writteii texts, closed captioiiiiig for vocal laiiguage iiiforinatioii). The purpose
of tliis paper is hvofold: (1) describe and discuss issues we believe iieed to be addressed, focusiiig oli tlie Iiinitatioiis
that appear to cliaracterize several e-learning platforiiis that have beeii proposed for DL; (3) preseiit and discuss
ongoing research aiined at overcoinirig these liinitatioils.
It is widely kiiowi~that al1 over tlie world deaf cliildreii
aiid. later, adults, experience dramatic difficulties in
achieving appropriate receptive and expressive skills iiot
only in ora1 or vocal laiiguage (VL) but also i11 written
laiiguage. Tlie vast rnajority of deaf leariiers (DL)
achieve literacy levels tliat are niarkedly below tliose
proper of their heariiig peers (see among otiiers Caselli.
Maragiia & Volterra, 2006; Garcia & Derycke. 2010;
Garcia & Perini, 2010). As a result, iii their scl-iool years
through adultliood, DL experieiice equalIy draniatic
difficulties i11 accessiiig tlie vast body of knowledge. aiid
tlie ricli learning eiiviroiimeiits made available by
advanced rnultiniedia technologies, inost iiotably
e-leariiiiig eiiviroiiinents. Appropriate written language
slcilis are i11 fact unquestioiiably a pre-requisite for
exploitiiig tlie possibilities arising fiom su ch mul tiniedia
a ~ i dmultiinodal leariiing eiivironments.
In Italy as al1 over tlie world' the situatjon of DL is
especially coinplex due to the very difrerent language
backgrouiid aiid experieiice deaf persons may Iiave
depeiiding upon the language tliey use as their priiiiary or
or L I . It is in fact
preferred iiieans of commuiiicatio~~,
necessary to distiiiguisli two groups: (1) those wlio use
Italiaii Sign Iangiiage (LIS), tlie visual-gestriral,
'
For reasoiis liiiked t0 iiie deinography of deafiiess aiid to ihe
compier: socioliiiguistic aiid cultura1 properties of sigiied
laiigiiages tlie obseivatioiis we make liere with respect to Ital~v
cm
easilx esteilded across natiolls aild c i i l ~ s e s .wjth the
1lecessai-y changes coiicemiiig tiie iiatioiial sigiied aiid
vocallwritten laiiguages.
face-to-face laiiguage of the ltalian deaf commuiiity
(LIS-LI); ( 2 )those wlio prefer to use spokeii aiid writteil
Italiaii (Italiaii-LI ). It is importaiit to stress that. oli the
whole, bofh g-l-ozips q f D L experience severe difficulties
in achieving appropriate literacy levels - though of
course 'exceptional leariiers' wlio overcome tliese
difficulties cali be found witliin each group.
With respect to sigiiers. tlie followiiig must be
iioted. Since the modei-11study of signed laiiguages (SL)
began with Stokoe's (1960) pioiieeriiig work o11
Aniericali Sigli language (ASL). world-wide research lias
led to describe. aiid to recognize as full-fledged liumai~
natura1 languages, a very large nuniber of national SL.
iiicluding LIS aiid al1 the other inajor Europea11 signed
languages. The use of SL for instructional purposes has
been esplicitly recoinmended by the European
Parliaiiieiit (see Resolutioil 17-6-1 988, arr. D).
Bilingual educati011 programs that offer sigiied and
orallwritteii language iiistruction to deaf students have
been developed in several coiintries, including ltaly
where they have been applied for the most to Elenientary
school cliildrei~.As reported by Caselli & al (2006). it is
uiiquestioi~abletliat the use of a SL. eveii if limited to its
usual, face-to-face- form. cali play a veiy iii~portaritrole
iii fostering DL's genera1 liiiguistic competence.
The iriclusion of SL witliin e-leariiing platfor~iis
designed for DL has come as a natura1 developinerit of
tlie advai~cemeiitthat have been made iii our kiiowledge
of SL alld of deaf sigliers. H
~ as ,ve poiiit
~
out~ili
~ e c t i o l l2~t0 4 below. nlail)l SeCeilt ai-id cuirent atrenlptS
to develop appropriate e-learning e~ivirolime~its
for DL
~
~
exliibit, aildlor implicate some ma-joo conceptual,
iiiethodoIogical ai~dpractical liil~itations.
Irl section 5 we preseilt and discuss ongoing
research aimed at overcoming tl~eseliinitations.
texts to sign. One exaiiiple is "Sigi~speak" (see also the
pro-ject "Dicta sign")
2 . S o m genera1 problems eormcerniwg
exhstiwg e-learning pl&iion.ms for DL
Irrespective of wliether real sigilers or signing avatars are
of inaiiy curreilt efforts
used, one additional limitatio~~
towards integratiilg SL materials into e-lear~iing
platforms concerns a failiire to recognize iinportant
differences between SL arid vocaliwritten languages. and
the probleins posed by the dramatically iiisufficient
refereilce tools, and overall liilguistic descriptioils, that
are cui-i-eiltlyavailable for SL.
It must first be recalled that al/ SL are languages
a -ir)ritfer7 tracfition. More importantly fsom a
~~'itliozit
research standpoii1t. and even thougl~almost 50 years
have passed since tlie i~lodernstudy of SL has begun.
researcl-iers still have not found a11 agreement on: (a)
what are the coiistituei-itelernents of SL; (b) what graphic
systenis cail be used for representing SL in written form
aiid. o11 this basis, develop appropriate i'eference tools
(e.g. dictionaries, grammars, usage-based corpora etc)
that are u~iquestio~lably Ilecessary for both the
coinlilui1ities of sigiiers, and the exploitatioil of SL for
educational aild instructional purposes (see Cuxac &
Antinoro Pizzuto, 2010; Garcia, 2006; 2010; Garcia Rr.
Deiyclte. 30 10).
It is not trivial to stress that, although oui.
ki1owledge of SL has considerably advanced, we still do
not have aily moilolingual dictioilasy or gramrnar, for
any of the SL that has been to date iilvestigated - not
eveil for ASL.
Iii this coiitext, one could expect that well-grounded
proposals ain-ied at exploitiilg SL for iilstsuctional
purposes would dedicate particular care in inal;iiig
explicit the models of SL elements and discourse they
adopt. This appears especially necessary becaiise. as
recalled hereafter: there are at present two ina-jor ciasses
of models for describing SL. 111agreemenl with Ciisac dC
Sallailcire (2007) we will refer to tllese il~odels as
"assiiniliatioi~ist"vs. "noil assin~iliationist": the first type
of models highligl~tprimarily the structural similarities
betkveeii SL a11d VL, while the secoild oiles underscore
that, in additi011 to important similarities there are
equally relevant differeilces between SL aild VL.
Within the Iiinits of the preseiit context, we
iliiistrate some of the criicial differeiices behveeii these
&vo types of inodels in relati011 to the problein of
defining what are the constituent elemei~tsof SL.
In substantial agreemeilt with early, very influential
descriptiolis of ASL provided by Stokoe ( 1960) and
subsequeiitly Iiiima & Bell~igi (1979), assiinilatioilist
models assume that SL coi~stit~ients
units are essel7tial[-i.
co117parnOleio T.2 1r~or.d~:
and are yr-i~ilnr.ifi,seqz~er7tiall-i~
o~gnrzked i11 ti117e. These models al-e still largel>7
prevailiilg in curreilt research o11 SL and have beeil foithe inost acritically adopted i11 educational applications
For- t l ~ epurposes of this paper, we liri-iit our attentioiis to
e-learniilg piatforrns designed for youiig or adult DL. Ai1
overview of severa1 such platforrns reveals the following
ma-jor iimitations. First, the griidelines for deveiopinp the
desired pjatforrns are often just "sketched7', aiid provide
fairly genera1 suggestions conceriiiiig, for exainple: - tlle
ii~clusioil of SL videos with SL tsai1siatioiis or
explanations of the written texts found in a specific
e-leariling platforin; - the deveIopinent of automatic tools
(Le. avatars) for tsaiislatiitg written texts into SL; -the use
of cooperati011tools such as video conferencing'. Secoild.
many existing or plaiined platforms appear to be
designed primarily for DL who kilow SL. but seen~lo
o
to use YL.
neglecr fl7e needs o f D L ~ v h pr-efer
011tile whole. there appears thus to be a general
ti-eild towards creating aild ii~cludingSL materials for
impleineiitiilg writteil text-based ei-iviroilinents. Tlze
are made more
contents eiicoded in written lar~g~iage
accessibie to (signing) DL via SL translations and
explanations. Otller esamples are tlle platforn~ created
within the project DEAL
for teacliiiig foreigil
vocal-written lailguages to DL, or the one designed by
Drigas SC. Koureinei~os(2005) for vocational and general
educational training.
A fairly large body of work has been dedicated to
the developinent of signing avatars to be added to tl-ie
users' interface. replacing SL materials preserited by rea1
sisners (see for exainple Efthimiou & Fotinea, 2007;
2008; I<arouzis, Caridaliis, Fotinea BL Efthirniou, 2007,
or also the recent Italian project "ATLAS"
Maily pro-jects for realiziilg signiilg avatars exhibit
however, ii-1 our view, a rather surprisiilg Iimiration: they
appear to have a uniclil-ecfional, FZ-cer?ier.edyer-spectii-e.
They start, for the inost, fiom VL written texts atid aiin
at producing avatars that cari tsailslate siich written texts
iilto individua1 signs and sigiied sequences. These pro-iect
tl~usigilore or underestimate the problem of tr.ar~slatirzg
Jho17~s i p to i~ocal/ii~ritte~7
rexts. There are o111y few
projects tliat explicitly aim at realizing sigiling avatars
firnctioning in both directions, i.e. frorn sign to speech
aiidlor also written tests, aiid from speech and writteil
'
9.
' See for ex.: Ir~divrclitois
iiho are Deaf or- I-inrii o f Hearrilg.
Center for Assistive Technologj and Ei~viroiiinei-~trtlAccess
(CATEA ).
T
L?
.
. : _ --.,
IAlS gziidelzr7es for D e ~ ~ e i o p i i ~..lccessibiIe
g
Lenr~~rng
Global
Leai-niilg
Coi~sortiurn.
..;I~~plicatlotzs. I M S
--,-,
,, i .'*
.
; ; Gel~eralgir~delir.res.for.
Inclztstve Onlrne Cliltz~ralContelir.
-Ca11adiai-i Netwosk for Ii~clusi~ie Cultura1 E ~ c h a i ~ g e .
t
>
%
-
-
L "
<,I
:1
3 z ' G
<
l".
t
,
.
L
p
-
*('i9
%i
I
-
i
I
'
3. SL eomwnhanèeatio~aand instiirretitonal
mateiiials: what modegs of Sk to adopt?
of differeilt types, including e-learning piatforms.
111 contrast, 11011 assiiniliationist models, based o11
extensive analyses of SL discoitrse, show tliat SL
constituent elen~eittscannot be easily assin~ilatedto VL
units. In addition to word-Iike eleinents. SL possess
complex, liiglily iconic sh-uctures (HIS) that are
sinlultaneously orgailized in a rnultilinear fashioii that
has no parallel i11VL. The differences between word-lilte
aiid non-word-like units are rnarlted by no12 inailual aild
manual ai-ticulators. rriost notably by rnodality-specific
eye-gaze patteriis: wheri producing word-like uilits. the
signer's gaze is directed towards the iilterlocutor,
whereas when producing HIS the signer's gaze is
typically directed away f o r ~ nthe iilterlocutor.
Figure 1 below provides just two illustrative
exainples of a word-iike unit ( l a ) and a noli-word-like
HIS (1b) that are coin~nonlyfound i11 SL discourse. The
examples are talteli froln LIS discourse but a wealth of
siinilar exaniples can be fouild in al1 SL (for relevant
disc~issions, see especially Cuxac, 2000: Cuxac 5(:
Ailtinoro Pizzuto, 2010; Pizzuto, Pietrandrea & Simone,
2007; Garcia & Derycke; 201 0).
sorts to be "attended to" and processed, and other
information stemmiiig from the interaction with other
fellow studeilts a~idlorwith a tutor (e.g. in excl~ailges
ta1;inp place in actual classroo~ilsor iil videocoiiferences).
Since deaf persons inust use their sight. and accordiiigly
orient their visual attention, to process bot1-i itinds of
iiiforinatioil, tile two tasks caiinot be carried out at the
sarne tinie: DL cannot simultaneo~~sly
look at teaching or
explanatory materials displayed o11 the coinputer screen
ar~dat Iiiiguistic, interaction-based iiiformation giveil on
the saine inaterials which they mttst aIways decode
primarily via visi011 (e.5. by lip-reading spolien
utterances, processi11g a message i11 SL, reading
subtitles).
Tliis is inuch u11lil;e what happens, in the sanie
cooperative leariiii~gsituatioii, for hearing learners wllo
cali simultaileously process co~nmunicative inessages
coilveyed through souiids and freely orient their visual
atteiltioii to other types of iilforinatioil coining fioin tlle
coinputer screeil. Devising ai1 appropriate e-learning
environment for DL thus requires accurate analyses of
the ways in which tliese learners use and distribute their
visual attenti011when performing differeilt leariiiiig taslts,
and how this cali influente the leanling process.
5.
Figure 1: Word-Iilte sign ( l a) and HIS ( 1b)
T11e point we wish to stress here is the followii~g:
rangiilg fioni 30%
HIS are very fieclueilt in SL disco~~rse,
to as much as 9090 (depeilding o11 discourse geisre) of the
constituent elernents that cali be identified and parsed in
SL discourse (Boutet. Sallaiidre & Fusellier-Souza, 20 10;
Cuxac & Ailtinoro Pizzuto, 2010; Sallandre, 2003; Di
Reiizo & al, 2009). It should thus be evident that SL
descriptioiis of aily sort, including modelisations via
sigiiing avatars, caiinot disregard as "inargiiial" these
structures that appear uilique of SL (see Cuxac & Dalle.
2007). E-Iearniilg materials based oli the assumption that
SL eleinents are for the most "ust Iike VL words" thus
exhibit severe Iimitations that need to be recognized.
critically discussed and, hopef~lli],amerided.
Ai1 appropriate e-Ieariiing ei~vironinentfor DL at Iarge,
i.e. for both sigilers arid non-sigiiers. inust talte in due
account a coilstraiiit that can be easily observed aild yet.
to our Iinowledge, has ilot beeil carefully investigated in
previorts research. Wheil working with a computer, the
~ i s ~ l aatteiition
'
Patteriis Proper of DL "lariiedi~differ
5oni those observable i11 hearixig Iearners. This is n-ue
especially ill situatioll of cooperative learning wllere the
studeilrs
silllu'taileous'y
iilformatioil concerilir-ig writteil materials of different
Towsar-ds deaf-centered multilingua1 and
multimoda1 e-iearrpiirag platforms
Figure 2 scliematically illustrates a mode1 for an
e-learning platform prototype (ELPP) prototype we are
currently developing wit1lii.i the fraiiie of a ilatioilal
project which pursues two major, iilterrelated objectives:
(1) irnproving muItilingua1 I inulti~nodal e-Iearning
enviro~iineiits for DL (High Scliool and Uiiiversity
students): (2) proinoting their Iiteracy skills ".
ITALIAN
i.lritt?n and
SPOKEn
t
..
4
DEAF
LEARldiQS
NEEDS
EEAF-CEIITERED
E-LEARFIING
ENVIRONMEIIT
PROTOTYPE
PLATFORM
+
+.----
( ~ 1 5and
~ 1
Itaiian-Li)
-
1-
-P
LCARNCRS
DEAF
VISUAL
ATTENTI014
PATTERNS li4
HCI
-- iL-----
l
L
v-
MULTINEDIA
TECHWOLOGIES
DEAF
WORLD VIEVd
Figure 3: A mode1 of deaf-centered e-learilinp platform
The ELLP model illusti-ated
6
i11
Figure 2 aims at
The prqject iil\~olvesfive research teains providing iiiter- arid
~alls-discip]iilayy co1ilpeteiices across tiic fie]ds ofi -CL
liriguistics; -special and bilingual educati011 for DL;
-11iultimedia tools
foi- DL
aild hearing learners:
-11uinali-coiilputei-interaction a i d visual leanling i11 e-Ieai-iling
envisonmenrs: -foreigii laiip~iage teachiiip rneliiodolopies in
both traditional aiid e-leariiing en~~iroilrneilts.
overcoiniiig tlie Iimitatioils proper of many e-leariiing
platforins (see discussion above) in different ways. Each
of tlie iiiajor L'co~iceptualcoiiipoiieiits" of tlie iiiodel is at
the salile tirne rilotivutrd Z7j*,and necessn~y~fòr
designing
a deufcer7ter'ed e-leurnir~gyla<forril. The platforii~ is
grouiided upoii the idea that research aiined at creating
useful products for deaf itsers needs to be developed,
fioni tlie very start, ivith deaf persons, not just,for, or or?
deaf people. Accordingly, and ratlier differeiitly fio111
what is reported for many past and ongoing pro-jects
directed to deaf persoiis, this idea guides our actuaI
'project inaiiagemeiit' practice. The project-leader team
includes six deaf colleagues who ynrricipule as
yroiugor~istsi11 tl~e
plalzrzilzg urzd ar'ticz~lationof tl7e erlrire
resec1rc11VI-oject, not only as "end users" or "eild
evaluators" of the language resources and didactic tools
to be produced or in1plemented. Al1 oiir deaf colleagues
are highly proficieiit iii LIS: three leariied to sign in
infaiicj~,witliiii deaf signiiig families, three ad different
ages, as it liappens to iiiost deaf sigiiers (see Cuxac &
Antinoro Pizzuto, 201 0); they possess different degrees
of lcnowledge of spolieii /writteii Italian wliich mirror in
part their educational backgrou~id
We give here just few practical exaiiiples of tlie
cruciaf iiivolvenient of our deaf colIeagues. The choice
of the "coiitents" we will focus on for developiilg tl-ie
ELPP~,and of tlie different forms in which such contents
will eveiitrially be presented to DL 011 the ELPP (e.g.
spoken aiid writteii texts, speech-to-text captions, SL
traiislations and explanations, zraphic illustrations), was
made following extensive discussions, arnoilg the deaf
aild the hearing meinbers of tlie pro-iect, of different,
alternative possibilities. Our deaf colleagues are
coiinibuting to the preparati011 of ad-hoc questionnaires
ai~d to a thorough examination aiid evaluation of
language taslis, materials, multirnedia techiiologies we
are using andlor are currently developing (iilcluding for
ex. the ELPP iiiterface). In short, the active involvement
of deaf colleagues ensmes that the eiid products of our
project be. on one hand, consistent with tlie "dec!fir~orId
i~ieiil",(see Figure 3 ) - i-e. a complex configuration of
experiential and conceptuaI knowledge that is strongly
grouiided in visi017 (see ainong others Lane, Hoffineister
& Bahail, 1996). and: oli the other haiid, effectively
respond to DL needs (see Figure 3).
Oile otlier irnportant elemeiit of the deaf-hearing
collaboratioii we are prornoting within the pro-ject is the
following: al1 the lieariiig members of the project-leader
teain possess a good or advaiiced kilowledge of LIS; four
of tlie five (hearing) young researchers of the otlies
research teains involved in the pro-ject are currentiy
atteiiding classes to iearn LIS. We are also seeliing
'.
Spohen/writteii 1aiiguai.e proficiency iii deaf liersons is liighl:
variable aild oilly partially Iiilked to t l ~ eeducatioilal leve1
achieved. Ous deaf colleagues iilclride one doctoral studeilt. one
college grad~iate. one Uiiiversitj studei~t.three higli school
graduates.
For space limits \ve can only inention 11ei-e tlle 'geiieral
coilteilts' o f tlie ELPP:we \vi11 focus on tlie histoi-)'. evolutioil
aild ilse of writiilg. and compare orol/sigiied vs. graphic/\~~itteil
Soril~sof coinmunicatioil.
furtlier collaboratioiis with deaf experts w l ~ ouse Italian
(rather tlian LIS) as their preferred language.
As iioted above, ~ n o s te-leariiiiig platforiiis for DL
or~ljl~fbr
sigr7ir7g DL. I11 coiitrast. as
appeas to be desig~~ed
shown in Figure 2. our ELPP aims at addr.essir7g the
needs o f botli sig11ir7g (LIS-LI) and 1lon sigr~irlg
(Ifalian-LI)DL. In fact, as also iioted above, botli such
groups of DL experielice dramatic diffjculties in literwq?
develoyr~~erzl.Our research aims at ascertainiiig tlie
specific comniunicative-linguistic needs of each group of
DL alid tlie extent to which tliese are, or are not
comparable. We expect tliat the
results of our
investigatioiis will provide: ( a ) novel, relevant
information oli the liiiguistic-cognitive profile of tlie two
grorips of DL, clariQiiig also wliether, andlor how
knowledge of LIS as L1 inay. or nlay not, interfere with
tile acquisition and Lise of spokeii/written ItaIiaii; ( b )
important indications oli how \ve may iieed to
differentiate tlie niultili~lgualaiid iiiultiinodal materials to
be created fos promoting literacy developnient in DL
with LIS-LI as compared to DL witl-i Italian-Ll. For
exainple, recalling what iioted in sectioii 3, it worild be
plausible to hypothesize that, for DL with LIS-L1, the
siinuItaneous1y
organized,
inultilinear
liiiguistic
sb-uctures that are higl-ily specific of tlieir SL, naniely
HIS, niay negatively interfere with the learniilg of more
sequeiltially organized liiiguistic structures that are
proper of writteil langiiage. It would be equally plausible
to hypothesize that tliese potential negative iiiterferences
sl~ouldbe abseiit in DL witli ItaIian-LI. However, tliese
Ilypotheses can be evaluated oniy by coinparing the
liiiguistic-cog~iitiveprofiles of tlle two groups of DL, as
we plan to do in our project.
A substantial novelty of tlie multilingua1 /
iiiultiiiiodal ELPP e-learniiig etiviroiiment we are
designing coiicerns the use. presentati011 (hence. by the
saine tolien. explicit modeling and representatioli) of tlie
two n~ajor types of Iai7gz~nge resozlrces tliat will be
employed for pedagogica1 purposes. iianiely: Italia11 aiid
LIS. What is nove1 in our niodel is that. as illustrated in
Figure 2. writteii texts \vi11 be provided not only in
11lritte17
Italia11 (the target language in which we aiin to
proinote DL literacy developinent), but also in ii~ritfen
LIS - a laitguage resource whicli: to our l~nowledge,has
never beeii experiniented in e-leariiing platforins for DL.
Spolcer~Iraliun aiid jucr-~o-fi~ce
LIS (the latter in tlie foi.111
of digital videos) will also be used (sec Figure 2).
For the iiistructioiial ~naterialsto be provided in
~i~r'itien
Italian, guided easificatioil procedures will be
used to facilitate DL's access to textual materials;
speech-to-text captioning tools will grant visual
accessibility to materials given in spoken Ituliar~;
Iing~iistic accessibility to the contents and forilis of
Italian-eiicoded instructional niaterials will be enhanced,
for DL witl~LIS-LI, via appropriate videos providing
tra~lslationsand explanations in (face-to-face) LIS. Due
to space limits, no further details are given here on tiiese
three types of Ianguage resources. which wili be
iiiipleinented driving o11 a co~~solidatedexperielice in
bilingual educati011 for DL (Caselli & al, 2006). and
more generally i11 language teaching inetIiodoIogies, as
detailed in our grant proposal. We describe briefly the
rationale. einpirical grounds, and major ainis of our
nove1 experimentatiori of written LIS .
As ~iotedi11 section 3. al1 SL are at present without
a written tradition. For DL with LIS-LI, the lack of a
writteit form of their own SL inay well be one of the
obstacles oli the road towards achievilig appropriate
literacy skills i11 a language - lilie Italia11- that 11ot only
does have a writteii tradition but is also typologically
very different fì-om their own (see especially our remarks
above on SL HIS). Recent research shows that Italian
signers can profitably use Sign Writing (SW). a grapliic
system proposed by Sutton (1999) for writiiig SL, for:
-transcribirig LIS face-to-face productioils; - creating, for
tlie first tiine in the history of this SL: texts conceived
directly in written LIS (SI47 llas beeii adapted for tliese
purposes to LISI. More importantly for the present
discussioil, tllis research shows tllat, relyiiig on
S W-encoded LIS texts, signers cail autonoinously
perforin ineailiiigful con~parisons between LIS and
spoken/written Italian, at al1 struckiral levels - lexical,
moi-pliological, syiitactic, textual, pragniatic.
011tllis basis, signers can foriiiulate metacognitive
and metalinguistic reflections on the strutture of LIS as
coinpared to spokeil/writtei~Italian, and more generally
ori the relations between "orality" or face-to-face vs.
written cornrnuilicatioil. i11 a way that has never been
possible. for thein. without relying on a written
representation of their SL (see ainong others Di Renzo &
al. 3006; 2009; Gianfreda & al, 2009; Pizzuto & al 3006;
Aiitinoro Pizzuto & al, 2008). Takiiig in due account the
crucial role t11at metacognitive and metalinguistic skilIs
notoriously play in the deveiopment of literacy sliills,
these research findings have motivated us to further
experiment written LIS, on our ELPP, as a potentially
very powerf~~l
pedagogica1 tool for proinoting literacy
abilities. S\V-encoded: writteil representatioils of LIS
have also proven te be extremely useful for advancing i11
tlle lii~guisticanalysis of tlle language (,41iti11oro Pizzuto
BI al. 2008), paving the way for more appropriate
modelisations whicli niay be i ~ s e d for botli genera1
descriptive purposes, and for iinplementing the use of
LIS as a linguistic resource on e-learning platforms.
We iloted in section 4 that DL 's viszral aitentioii,
patter-ns i17 HCI inay significantiy differ from tllose of
hearing leariiers. One other additional novelty of our
project conceriis the use of eye-tracking equipineilt for
ailalyzing DL's visual attentiori patterns, and compare
them with tl~oseof hearing learners', duriilg Iearniilg
taslts which dernaild tlie sim~iltai~eous
processing of
Ianguage resources along wit1-1 visual infonnation of
different sol-ts. Prelirniilary results of a pilot study we
have conducted indicate that. in processing rilultimodal /
niultilanguage materials, the gaze patterns of DL with
LI S-L 1 inarliedly di ffer fiotn those of hearing learners
(Capuano. Levialdi & Antinoro Pizzuto. subinitted). We
trust tllat the inore exteilsive investigations 011 this topic
we plan to develop witliin our project will provide us
much n eeded, nove1 information for a better
understaiiding of how visual ii~forinationneeds to be
spatially and ternporally structured in e-leaniing
eiivironments for DL, as compared to hearing users.
These analyses will also alIow to us ascertain whether
there are (or not) relevant differences between sigiiing VS.
non-signiiig deaf students, when these DL with different
language bacliground access and rise visually grounded
infor~nation,of both linguistic and non-lingiiistic type.
Finally, recalling the crucial importailce of vision in
the 'deaf world view', we think that web-based
ri~ztltiri~edin1echr.io1ogies and learning tools for a
deaf-centered ELPP may be sigilificantly improved
implelnenting a visually-based graphic interface.
Drawiiig on ongoing research 011 tlie topic (Capuano & al,
submitted), we aini at designing an iilterface that DL can
access and use easily a i ~ d'ii~tuitively' because textual
inforination (which is difficult for thenl) is significantly
reduced, or even entirely replaced by rnostly non-textual
( icoilic) iiiforniation. This entails the need of creating a
new, graphic way for browsing web pages, and
interacting with the ELPP.
For the natural, deaf-peculiar visual wlay of
grasping information to be exploited in our pIatfori11, we
are going to use a new interactioil paradigin based on t l ~ e
theories of embodied cogniiio\~arid sfor~~zelli~7g
(Lakoff
& Johiisoil, 1980; Johiison. 1987; I~naz8ci Benyon. 3007).
Wit1iii-i this paradigm, tlie learning process can be
n~etapl~orically
represented as a story that iacludes the
Liser as tile rnain character. Accordingly, the user 'lives'
the learning process by pliysically experiencing it - in
the vii-tua1 space of the ELPP - as a path with a starting
place, a sequence of severa1 Iear~iingsteps, and a final
goai. Such a metaphor seerns to be a very intuitive way
of representing the learniiig environn~ent.h4oreover, it
seelns to be ari adequate iiiteraction paradigin especially
for deaf users, since it exploits the visual cl-iailiiel as the
~nainsource of iilfornlation.
The preparation of this work was partially funded by -the
Italian
Ministry
of Educati011 aild Research
(MIUR-FIRB), Project E-learxirig, Deafness nnd TPi.itte17
Langunge / TrISEL -RBNE074TSL (3009-20 l?), see:
l : / / .i
. C I i ; -The Association Progelfi
Felicità, Pro-ject "Writiilg L1S and SignWriting" Rome,
(2005 -1. JVe thanlì our colieagues Stefano Levialdi,
Marilena De Marsico, Ailila Labella and Alessio Di
Renzo for helpfiil coinrnents.
7 . References
Antinoro Pizzuto, E., Chiari. I . & Rossini, P. (3008). The
represeiltation issue and its inultifaceted aspects in
constructing sign language corpora: questions.
answers, furtiier problems. P~.oceedir?gs qf the 3rd
Worlishop 071 1l7e Repr~esenic~tiorar~dPr~ocessingqf
Boutet. D., Sallandre, M.-A.. & Fusellier-Souza, I.
(3010). Gest~ialité huinaine et langues des signes:
entre continuuiil et variations. 111 B. Garcia et M.
Derycke (eds.), Sozfrds e1 lat7gzle n%s sigues. Aior.117e et
i-aiaiufions,Langage et Sociéié. no 13 1, mars 201 0,
55-74.
Cap~iano. D.. Levialdi, S. & Aiitinoro Pizziito, E.
(submitted). Embodied vis~iallearning and deafiless: a
concept paper.
Caselli, M.C., h~laragna. S. & Volterra, V. (2006).
Lingztaggio e sor.dità. Bologna: I1 blu lino,
Cusac. C. (2000). La Lai7gzte des Signes Fra~quise:/es
Tbies de I'icorqicité, Faits de Langues i1°15-16, Paris:
Op111ys.
Cuxac. C. & A~ltinoroPizzuto, E. (3010), Emergeilce,
norme et variati011 dans les langues des signes : vers
une redéfinitioil ~~otionilelle.In B. Garcia & M.
Derycke (eds.), Sozlr-ds e1 lcrngzie des signes. Norr71e e/
i~ar.iafioi7s,Lungage et Sociéié, n" 13 1 , inars 30 1 0.
37-53.
Cuxac, C., Dalle, P. (2007). Problérnatique des
chercheurs e11 traitement aiitomatique des laiigues des
sigiies. 111: Tr.aifenzeizf ,-luioiitutiqzle des Lai7gztes,
(-4T4L-4) - Traiien1e17taztlomafiqzte des langzres des
Vol.
48,
N.
3,
15-30
sigizes,
t"s:~-Lt&2-:Y;"-' L;),. ;Lr -. -.- l - iF\
-2-L
Cuxac. C. et Sallandre, M-A. (3007). Iconicity and
arbitrariness in French Sign Language: Izighiy iconic
sti-uctures, degenerated iconicity and diagraii~niatic
iconicity. in E. Pizzuto, P. Pietrandrea & R. Simone
( eds), T éi+bul and Sig~zed Langzlages, C0117pur.i11g
Stl.~tcfztres.Constrztcts nr7d Adethodologies, Mouton de
Gruyter, Berlin-New I'ork, 13-33.
Di Renzo, A., Gianfreda, G., Lamano, L., Lucioli' T.,
Igeilnacclii~B.. Rossini, P.. Bianchiiii C.S., Petitta G..
Antinoro Pizzuto- E., (2009). Representatioil Ai~alysis - Representatioil: nove1 approacl~esto the
study of face-to-face and written iiarratives in Italia11
Sign Language (LIS). Papes Presented at the CILS
Interr7aiional Conferente on Sig17 Langziages, Nainur,
Belgium. November 16-20,3009.
Di Renzo. A., Lamano, L-, Lucioli, T., Pennacchi. B.,
Porizo, L.. (3006). Italia11 Sign Lang~iage:Can we
write it and transcribe it witl-i Sign Writing ? 111ELRA
(eds.). LREC 2006, TiTlor.kshop Pr.occzedin,ar (W--15):
Second l$brlishop o17 fhe Repr-esenfation and
Processing of Sigt7 Langurges 1 1- 16.
Drigas, A.S. & I<ourerneilos, D. (2005). Al1 e-Leanling
Managen~eiltSystem foi- the Deaf people. Proceedings
qf JTrSEE+lSTr~ansacfionsor7 Adsnnces i17 Ei7gineer.ii7g
En'ztcation, VOI I, 2.20-24.
Efthimiou. E. & Fotinea. S. (2007). Ai1 Envirotlment for
Deaf Accessibilit). to Educational Content.
Proceedings of the Firasi Irifer*nationnlCo~?fei.enceon
i/?for.n1atio17 a17d Co117r1izinicatior7 Tecl717olog~~cz17u'
Accessibili~q,
Efthimiou, E. & Fotinea, S. (2008), Tools for Deaf
Accessibility t0 ai1 eGOV Environinent, in Lec1zli.e
Aiores in C0177pziter- Sciei7ce (LNC'S). Vol. 5105,
446-452.
Garcia. B. (2006). Tlie ~~~ethodological.
linguistic and
sen~iological bases for t l ~ eelaboration of a written
fori11 of LSF (French Sign Language). In ELRA (eds.),
LREC 2006 - JI.or-liLshopP~.oceedings ( /V-1.51, Second
TdurI1~170p017 tl7e Repl'esentafio~a17d Processing of
4
/sf3,b>
z
\.
.
Sigu Languages, 3 1-36.
Garcia. B . (20 10). Sott/.ds, szir.dilé, lungz~e(s)des signes ei
épisiémologie
des
scieilces
d z ~ languge.
Probléi17aiiqzies de la scr-ir)tzrr-isniioi7 e; n ~ o d é l i s n f i o ~ ~
des bas nii?eazcs en Langile des S i g ~ e sFrar7quise
(LSF). Mémoire d'Habilitation à Diriger les
Recherches, Université Paris 8-Saint-Denis.
Garcia, B. & Deryclie, M. 12010). Introductioil. In B.
Garcia & M. Deryclie (eds.), Sozlrds e f lnngzre des
signes. IVoriiie e1 1-nr-iatio17s.Lai7gage e1 Sociéti, no
131, nlars 2010, 5-17,
Garcia. B. et Perini. M. (2010). Normes en jeu et jeu des
nonnes dans les deux Iangues en préseiice chez les
sourds locuteurs de la Langue des Signes Francaise
(LSF). 111 B. Garcia et M. Deryclie (eds), Sozirds et
la17gz~cdes sig17es. A;or.li7e e1 ~ai*iafior7s,
Languge e f
Société, 11@131, mars 201 0. 75-94.
Gianfi-eda, G., Petitta. G., Biancl~ini,C.S., Di Renzo A.,
Rossini. P., Lucioli, T., Peiinacchi, B., Lamailo. L.
(3009). Dalla 1nodatit6 faccia-a-faccia a una liilgua
scritta emergente: nuove prospettive su trascrizione e
scrittura deila Lingua dei Segni Italiana (LIS). In C.
Consani, C. Furiassi, F. Guazzella & C. Pel-ta (eds.),
J t f i del 9" Cor~gressodell2ssociazione Italiana di
Lil2gziislica ,4ppiicnta - 01-alitiì Sc1.ittlrru. I17 nzemo~~ia
di Giorgio Rainzo~clo Cardoi7n. Perugia: Guerra
Edizioni. 413-437.
Iinaz, h4. & Benyon. D (2007). Desigr7ii1g ~rlithblends.
Tlie MIT Press.
ICarpouzis, K., Caridaltis, G.. Fotinea, S.-E.. & Efthimiou,
E. (2007), Educational resources and implenientation
of a Greek sign language synthesis arcl-iitecture.
Conrpzlters R- Edilcatiorz, 49, 54-74.
KIima E.S. et Beliugi, U. (1979). The S i p s ofLungzrngc.
Cambridge, M A : Harvard Uiiiversity Pi-ess.
.Tohnsoil. M. (1987). Tl7e ho&' i17 the nzir7d. University of
Cliicago Press.
Lakoff, G. & Joliiisoil, M. (1980). ibfetaphor. Tlé Live BJ:
University of Chicago Press.
Lane, H., Hoffnleister, R.J. & Bahan, B. (1996). A
journey into the Deaf-World. Sa11 Diego, CA:
DawnSign Press.
Pizzuto, E.. Pietrandrea, P., & Simot~e,R. Introdiiction.
In E. Pizzuto, P. Pietraildrea & R. Simolle (eds.)
(3007), J/érbal n17d Signed Langrrages - Conyarir7g
strnctirres, coi7s~rrcfsar7d n~ethodologies, Berliil /
New York: Mouton De Gruyter, 1-10.
Pizzuto, E. Rossini, P. aC Russo. T. 12006). Representing
signed languages i11 written form: questions that need
to be posed. 111ELRA (eds.). LREC 2006 - T.Ior-lsl7op
Proceedir~gs ((11-1.7): Second Worksl7op on 1ke
Representafion a17d Pr-ocessing of Sigr? Latlguages,
1-6.
Sallandre, M.-A. (3003). Les Z I M ~ T C Sdu discozliire efi
Langzre des Sig17es Fr.ar7qaise (LSF) - Tenfative de
de cntegorizatio~7duns le cadi-e d'ztne gra177171air.e
de l'iconiciié. Thèse de Doctorat en Sciences du
Langage, Paris, Université Paris 8.
Stolioe. W.C. (1960) Sign Lang~iageStrutture. in Sfztdies
in Lingztisfics - Occasiouial Paper- n. 8, 1960 (rev. ed.
Linstoli Press, Silver Spring. MD, 1978).
Sutton. V. (1999). Lessons in Sign JT?iting. Textbooli &
JTor-lcbook. La Jolla, CA: Deaf Actioil Comtnittee for
Sign Witing (2nd editioil, 1st edition 1995).
I'
Workshop on Suppor ng elearn
Language Resourcec and Seman
Valletta, Malta