Yet More Symbol Systems;
Relevant User Issues
Seminar on Speech and Language
Processing for Augmentative and
Alternative Communication
Spring 2010
Mahsa A.Yarmohammadi
Papers for Today
• Patel, R., Schooley, K. & Radhakrishnan, R. (2006).
Comparison of semantic versus syntactic message
formulation: A pilot study. Assistive Technology
Outcomes and Benefits, Volume 3, 96-110.
• Ottem, E., 2001. Use of pictographic-articulatory
symbols to promote alphabetic reading in a
language-impaired boy: Case study. Augmentative
and Alternative Communication 17 (March), 52-60.
• Venkatagiri, H. S., 2002. Clinical implications of an
augmentative and alternative communication
taxonomy. Augmentative and Alternative
Communication 18 (March), 45-57
Main topics of papers
• Patel et al. (2006)
▫ Compare message formulation using a semantic
frame-based approach versus the conventional
syntactic approach
• Ottem (2001)
▫ Use Blissymbols and a new type of pictographicarticulatory symbol to teach reading to a 9-yearold boy with specific language impairment
• Venkatagiri (2002)
▫ Offer a taxonomy of certain AAC system
components (signs) as a framework for clinical
decisions related to AAC
Patel et al. (2006)
• The use of picture symbols for communication
has gained increasing appeal
• Picture symbols paired with voice output serve
as a powerful means for expressing one’s ideas
• Most commercially available VOCAs use a
linear(syntactic) approach to message
construction
▫ “I” + “want” + “soda”
▫ At least basic knowledge of syntax is required
▫ No research support the notion that POS ordering
of vocabulary icons assists in language learning
Approach
• The use of semantic schema -associated with
the verb- may facilitate message construction by
▫
▫
▫
▫
making the rules of grammar more transparent
probably reducing keystrokes
easing the cognitive burden
improving access to vocabulary
• Compare message formulation using a semantic
frame-based approach (iconCHAT) versus the
conventional syntactic approach (Default) in
terms of:
▫ accuracy, speed, complexity, user preference
Method
• Eight developing children between 7 and 10
without speech and language impairments
• The child was shown 18 pictures and 4 threestep sequences
• The child used the iconCHAT and Default
prototypes to describe what was happening in
the scene he/she was shown
• Both prototypes had the same vocabulary size,
overall display size, icon size, graphic symbol
set, and the same type of synthetic voice output
• A survey was done to assess ease of use and
the satisfaction with both prototypes
iconCHAT
Default
Stimuli
Dependent measures
•
•
•
•
Construction time per linguistic unit
Time per button click
Button clicks per linguistic unit
Percentage accuracy
▫ semantic accurateness of the subject, verb, object
• Complexity
▫ the use of modifiers, one point for each modifier
• Use of quick reference panel
▫ one point for using the panel to select an agent or
pronoun, for iconCHAT only
• Reuse of prior message constructions
▫ one point for each reuse, for iconCHAT only
Results
• Some participants were faster and required fewer
keystrokes using iconCHAT, others performed better
using Default
Results
• All participants were 100% accurate in terms of
grammatical completeness using both prototypes
Results
• Sentences structured in both prototypes were
either in the SVO, SVMO, or SVMMO forms
Results
• The heterogeneity of participant strategies for
message formulation is evident in the results
Results
• No statistically significant differences between the
prototypes in terms of time per linguistic unit
Results
• Although there was a statistically significant difference in
time per button click between the two prototypes, the
difference was rather small
Results
• 5 sec/click in iconCHAT, 4 sec/click in Default
Results
• A statistically significant difference in the number of
button clicks per linguistic unit, iconCHAT requires
fewer clicks
Results
• Participants varied in the complexity of
utterances they generated
Results
• All participants used the quick reference panel to
a large extent
Results
• Only one child reused a prior message
construction
Survey results
• Various results across participants
• 2 participants thought that iconCHAT was easier
to use, 4 thought that Default was easier to use,
2 rated both equally
• 5 participants said that they used the semantic
schema as opposed to the serially construction
• 6 participants mentioned relying on the semantic
schema rather than the serial ordering
• 4 participants mentioned that they liked how
iconCHAT “showed what needed to be filled in”
Survey results
• Some positive feedbacks regarding Default:
▫ faster to learn
▫ liked the details (modifiers) along the bottom
▫ liked having all the categories on one page
• Very little negative feedback for either prototype
▫ 2 comments on the need for more vocabulary
▫ One participant noted that having to select the
verb first was odd in the beginning because it was
“not what I’m used to” but that “it was cool though,
I guess” because it “helped me.”
Conclusion
• Semantic-based approaches to message
formulation are at least as effective in facilitating
accurate and complete utterances for typically
developing children.
• The 2-dimensional semantic schema and the
serial ordering of icons may be beneficial to
children who use AAC given that their
constructions is limited to single word and has
atypical syntax
illustrates the relationships between icons
provides feedback regarding the output of the speech recognizer
Conclusion
• Semantic-based formulation may be especially
useful for those children who struggle with
grasping grammatical constructs
▫ semantic frame facilitate complete sentence
production through visual cues
Future work
• Future studies need to include children who use
AAC
• The stimuli should be expanded to include a
broader range of sentence complexity
• To assess the usefulness of the reuse feature,
multiple stimuli with the same predicate are
necessary
• Include a modifier panel along the bottom of the
iconCHAT interface
Ottem (2001)
• More than 80% of children with SLI experience problems
with reading (Tallal et al.,1996)
• Reading problems of children with SLI are probably due
to a deficit in phonological processing (Stanovich,
1998;Vellutino, 1979)
• To read using a grapheme-to-phoneme conversion
(alphabetic) system, phoneme awareness is required
• Some studies support the view that articulatory
awareness plays an important role in learning to read
(Heilman et al.,1996;Montgomery,1981;Alexander et
al.,1991)
▫ awareness of the movement of the position of one’s own
articulators is important in both phonological awareness
and the ability to grapheme-to-phoneme conversion
▫ The treatment of dyslexia should include speech therapy to
teach individuals to be aware of their articulatory gestures
Ottem (2001)
• Graphic Representational Symbols (GRSs) may
facilitate specific components of word
awareness, but their overall impact on beginning
reading may be minimal
▫ They don’t contain information about how the
basic units of the orthographic system are
pronounced by the articulators
• Can a GRS that contains visual information
about articularoty gestures facilitate
orthographic reading in a child with SLI?
Pictographic-articulatory system
• PAS was designed to portray the salient
articulatory features of letter sounds
• PAS was used to teach letter sounds for
alphabetic reading to a child with SLI
• Four phases of reading (Ehri, 1991):
▫ logographic phase: letters serve as visual cues
▫ transitional phase: letters provide alphabetic cues
to the sounds of spoken words
▫ alphabetic phase: recode spellings phonologically
into pronunciations
▫ orthographic phase: ability to read words without
phonologically recoding constituent letters
Pictographic-articulatory system
Pictographic-articulatory system
Case study
• Erik is a child with SLI
▫ problems with the structural aspect of language
▫ very short and ungrammatical utterances at the
age of 8 years
▫ poor word knowledge, unable to read any sight
words
▫ limited knowledge of the letters of alphabet,
unable to analyze or synthesize spoken or written
words
• Use of Blissymbols and the PAS with Erik over a
2-year period is studied
Assessments
• Erik was assessed using some standard tests
▫
▫
▫
▫
Wechsler Intelligence Scale for Children-Revised
Wechsler Preschool and Primary Scale of Intelligence
Reynell Developmental Language Scales
Illinois Test of Psycholinguistic Abilities
 Verbal Expression
 Manual Expression
 Sound Blending
• Erik was tested three times
▫ prior to the initiation of Blissymbols (8 years,2 months)
▫ prior to the introduction of the PAS (9 years,6 months)
▫ after termination of the use of the PAS (10 years, 3 months)
First assessment
• Initial scores on all of the verbal tests were quite low
Communication and Language
Intervention: Blissymbols
• Erik was taught Blissymbols in order to
▫ improve his knowledge of the meaning of the words
▫ increase his vocabulary
▫ improve his verbal communication skills
• Teaching one 30-minutes session per day, for about 7
months
• He acquired more than 200 Blissymbols (150 nouns, 50
verbs, pronouns, adjectives, prepositions)
• He could read sentences composed of up to 7
Blissymbols
Second assessment
• Both his verbal knowledge and his phonological
awareness were facilitated
Second assessment
• Erik’s ability to communicate was improved, he spoke
more often, spoke longer sentences and more variable
words
Reading Intervention: the PAS
• PAS was developed to proceed from
Blissymbols to orthographic reading
▫ Blissymbols in combination with letters didn’t
facilitate Erik’s ability to acquire beginning reading
skills
• Teaching the PAS three hours per day in order
to achieve four primary goals.
Reading Intervention: the PAS
• Promoting knowledge about the articulators
▫ teaching the articulators and their functions using
models, pictures, mirror work, and play
▫ What happens in your mouth when you make the
sound …?
• Promoting knowledge of the PAS symbols and their
relationship to orthographic letters
▫ link two symbols to form a word using the pictogram
cards
▫ read the words using PAS symbols
• Linking sounds and PAS symbols to words
▫ read two- and three-symbol words, then rearrange the
cards into new words
• Matching PAS symbols to letters
Matching PAS symbols to letters
• The letters of the alphabet were paired with the PAS
cards previously introduced
• A computer program for symbol-sound matching
was used to practice the link between the sounds
and the letters
• practice 30 min per
day for 8 months
Matching PAS symbols to letters
• Early in the process
▫ Erik understood the meanings of the constituent parts
of the PAS symbols
▫ He understood that each pictogram corresponded to a
letter sound
• After 7 months
▫ Erik knew 17 PAS symbols and their letters
▫ He was able to read simple stories comprised of
either PAS symbols or letters
▫ He reached the alphabetic phase of reading
Matching PAS symbols to letters
• After 8 months
▫ Erik refused to use the PAS any longer
▫ He could read without the aid of the symbols
▫ All of the remaining letters of the alphabet were taught
orthographically only
• Over the next year
▫ Erik gradually reached the orthographic phase of
reading
▫ He could fluently read quite complex instructions
▫ He still had difficulty interpreting the meaning of
abstract words, because of his delayed
conceptual development
Third assessment
• WISC-R and ITPA verbal expression results are the same
as the second assessment (or slightly declined in scaled
scores)
Third assessment
• The PAS didn’t appear to contribute to Erik’s
conceptual or general cognitive development
Third assessment
• His comprehension of spoken language and
phonological awareness had improved
Discussion
• Use of Blissymbols for 7 months didn’t facilitate
Erik’s acquisition of reading, however it
▫ increased his interest in communication and
improved his word knowledge
▫ promoted his understanding that words are basic
units of language, so increasing his print
awareness
▫ prepared him for the logographic phase of reading
• Use of the PAS facilitated Erik’s reading
acquisition by directing his attention to salient
parts of the sound system
Discussion
• Erik’s experience was sequential and hence
additive, so a different outcome is possible if PAS
use had proceed that of Blyssimbols
• Additional AAC research that examines the impact
of different types of GRSs on reading performance
is encouraged
Venkatagiri (2002)
• There are a huge number of AAC symbol/icon sets
and symbol/icon retrieval techniques
• This paper offers a taxonomy of AAC system
components
• Builds on and extends previous contributions to
AAC taxonomy by Lloyd (1985), Lloyd and Fuller
(1986), Lloyd and Blischak (1992), and Fuller, Lloyd,
and Schlosser (1992)
Current AAC taxonomies
• A widely used taxonomy: “aided” / ”unaided”
▫ useful for distinguishing between the physical
characteristics of AAC communication
▫ fails for differences in the complexity and scope
• A secondary-level distinction is “static” / “dynamic”
▫ confusion with respect to the use of these terms
 more movement and change for dynamic
communication, or
 durability of message elements in time
• the term “symbol” includes all types of message
elements involved in AAC communication, but
▫ fails to accommodate emotional states, comfort states,
physical & psychological needs, cognitive states
• AAC traditionally has a speech-centered definition,
however, it includes expressive communication in all
of its forms
AAC taxonomy
• Reasons for classifying AAC system components
▫ formulate a comprehensive and coherent theory of AAC
▫ provide guidelines for clinical decision making
 useful insight to the steps necessary to develop
appropriate AAC systems
• At minimum, an AAC system consists of a set of
signs and a retrieval method
Classification of AAC signs
Signs
• Communication takes place through signs
• Sign refers to all actions, events, and objects that
have a communicative value
▫ objects, tokens, marks, images, movements,
gestures, sound patterns, graphic patterns
• There are three main types of signs:
▫ indexes
▫ icons
▫ symbols
Indexes
• When the relationship between a sign and what it
represents is one of cause and effect and/or
physical proximity, then the sign is an index
▫ reflexive vocalizations
 cry, moan, grunt, sigh
▫ facial expressions
 happiness, sadness, anger, surprise
▫ body postures and movements
• Communication through indexical signs is
unintentional
• Communication partner is responsible for
recognizing the communicative value of indexes
Indexes
• It is useful to prepare
a detailed list of indexical
signs displayed by an
AAC user
Indexes
• Visceral indexes
▫ Emotional states
 fear, anger, happiness
▫ Comfort states
 fatigue, pain, feeling cold
• Visceral indexes are not amenable to shaping
through learning
• Skeletal indexes
▫ Physical and Psychological needs
 hunger, companionship, toileting
▫ Cognitive states
 alertness, excitement, anticipation
• Skeletal indexes may be modified through behavior
modification and social learning procedures
Icons
• When the sign bears a physical resemblance to
what it signifies, it is called an icon
▫ color, shape, size, texture, smell, sound, …
• Examples: some road signs (a drawing of a
falling rock), pictographic writing used by many
African tribes, pantomime, imitations of sounds
of animals
• The association of icons with their referents
requires some learning
Icons
• Transparent icons
▫ when a likeness of an object is used to represent that
object
 a picture of fire to represent fire
• Translucent icons
▫ icons that represent a property of the referent
 a picture of fire to signify hot rather than fire itself
• Translucent icons involve greater learning demands
than transparent ones
• Translucent icons must present salient physical or
functional features of the referent, otherwise they
are regarded as symbols
• Icons are well suited to represent everyday objects
and events, not abstract concepts (courage),
invisible objects (air), imaginary entities and events
(miracle)
Icons
• Transparent or translucent icons are either
▫ Gestural
 point to the nose to represent nose (transparent)
 a hand placed under the nose to signify smell (translucent)
▫ Auditory
 meow for the sound made by cats (transparent)
 meow for cat, noise of snoring for sleep (translucent)
▫ Material
 Pictographs
 a painting, photograph, etc, of object or
event(transparent) or for representing a salient physical
property of object (translucent)
 Miniatures
 a toy bed to represent bed (transparent)
 a small pillow to represent bed (translucent)
Symbols
• When the relationship between a sign and what it
represents is arbitrary and conventional, it is called
a symbol
• The relationship between a symbol and what it
signifies is entirely learned
• Examples: traffic signs such as two-way traffic,
religious symbols, political symbols (flags)
• Symbols are divided into verbal and nonverbal
categories
▫ verbal: symbols used within the framework of
natural language
▫ nonverbal: arbitrary and conventional signs that
are not part of a natural language
Primary symbols
• Primary verbal symbols:
▫ speech (vocal)
▫ gestural language: sign languages (e.g., American Sign
Language and signed English)
• Seamless and symbiotic relationship to cognition
• Cognition allows for producing meaningful spoken
and gestural communication, at the same time,
speech and gestural languages is used to organize
one’s cognitive domain
• Primary nonverbal symbols:
▫ speech (vocal): grunts, cries, sighs, and other vocal sounds
when used volitionally to represent arbitrary meanings
▫ gestural language: idiosyncratic and commonly understood
gestures
Secondary symbols
• Secondary verbal symbols: symbols used for writing
▫ Graphic: traditional alphabetic or other type of graphic writing
system
▫ Embedded-graphic: orthographic symbols in another modality
 Auditory
 Morse code
 Tactile-visual: three-dimensional cutouts of letters
 Visual-gestural
 finger spelling
• Secondary nonverbal symbols: artificial symbol sets
▫ Tactile: abstract 3-dimensional graphic symbols
recognized through tactile channels
▫ Auditory: arbitrary tones and noises used for
communication, have not been used in AAC so far
▫ Visual: abstract graphic symbols with assigned meaning
 many Blissymbols, color-coded geometric shapes,
Lexigrams
Flowchart of decision making
• The flowchart can be used to select indexes, icons,
and symbols for an AAC system
• AAC evaluation and system development should
occur simultaneously rather than consecutively
• It shows the order of integrating the different types
of signs into AAC system
▫ the rate of integration depends on the user’s capabilities,
motivations, and communication needs
1
2
3
4
5
6
7
Indexes
Primary nonverbal symbols (speech)
Primary nonverbal symbols (gestural)
Primary nonverbal symbols (speech)
Gestural symbols
Secondary verbal symbols
Material icons
Flowchart of decision making
• Indexes, volitional vocalizations, speech, gestures and
gestural languages are preferred to material icons
communication
• However there is a lack of research on indexical
communication
▫ it deals with vital aspects of well-being
▫ care givers are primarily responsible for indexical communication
▫ it is highly productive
• Disadvantages of material icons
▫
▫
▫
▫
problems of portability
availability
adaptability
scope
Conclusion
• Variables when selecting symbol/icon sets
▫ scope: what meaning may be expressed
▫ clarity: the degree of precision of meanings
▫ rate: the amount of information transferred/time
▫ learnability: the amount of time and effort to learn the skills
▫ cognitive and physical ease of message preparation
▫ availability: when communication can take place
▫ ease of comprehension for the partner
• Selection is consistent with the physical and cognitive
capabilities of the AAC user
• Importance of these factors varies depending the
needs and abilities of users and contexts
Conclusion
• Indexes (e.g., yawning and squirming) and primary nonverbal
symbols (e.g., grunts, idiosyncratic gestures) should be fully
exploited in an AAC system
▫ high rate, high availability, cognitive simplicity, ease of physical effort
• Gestural and auditory icons (e.g., commonly understood gestures)
should be important components of an AAC system
▫ highly learnable, early acquisition by developing children
• Primary verbal symbols (i.e., speech and gestural languages) can be
helpful in AAC systems
▫ high rate on scope, clarity, speed, comprehensibility, availability,
however
▫ they are more cognitively complex and require fine motor coordination
• Material icons (e.g., line drawings and miniatures) should be
integrated if the above signs fail to completely meet the client’s needs
• Secondary nonverbal symbols are problematic at best in AAC
systems
▫ poor in scope, rate, clarity, comprehensibility, availability, learnability