Technology Based Assessment of
Language and Literacy
Patti Price
- Intro
- Decoding
- Comprehension
- Opportunities/Discussion
PPRICE Speech & Language Technology
www.pprice.com
• Intro
• Decoding
• Comprehension
• Discussion
Evolution, primates to humans, millions of years ago
-60
-50
-40
-30
-20
-10
Primates
0
Gorillas
Humans to the leap
Humans
-7
-6
-5
-4
-3
-2
Stone
tools
Humans
Fire
-1
0
Shelter
Language: ~ 50,000 years ago
Spears
•Clothing
•Burial
Farming, 10,000 years ago
•Hunting
Bronze, 5500 years ago
•Art
Written language: ~5 – 6,000 years ago
• Intro
• Decoding
• Comprehension
• Discussion
Pre-language: Information access by
direct experience
Multimedia: The power of direct
experience, but not limited by time, place,
language, learning style, sensory
abilities…
Oral language: Information
access also from someone in
same time and place with same
language
Written language: Information access not
limited by time and place
Written and Spoken
Language are Different
•
•
•
•
• Intro
• Decoding
• Comprehension
• Discussion
These differences are important in learning to read.
How difficult it is to spell (many letter combos to one sound)
How difficult it is to read (many sounds for the same letter)
How closely your dialect matches written conventions
“Got to say this for you, you got guts. Guts
and no brains. But guts alone don’t mean
nothin’. “
“I gotta say this for you-- you have guts.
Guts and no brains. But guts alone mean
nothing.”
I’ve got to say this for you: You have
guts. You have guts and no brains,
but guts alone mean nothing.
Differences Between
Speech and Writing
Vocabulary
Distinct words
Sentence length
Sentence structure
Grammar
Prosody
Written
‘things’…
20,000+
10 – 30 words
Complex
Conservative
A . , ! ? ; : () 
• Intro
• Decoding
• Comprehension
• Discussion
Spoken
‘stuff’
2,000+
?
shallow
Liberal
Rich
Example
• Intro
• Decoding
• Comprehension
• Discussion
~130 words
(1) I’ve been talking – I’ve been multiplying matrices already, but certainly
time for me to discuss the rules for matrix multiplication.
(2) And the interesting part is the many ways you can do it, and they all give
~45 words
the same answer.
We’ve been
(3) So it’s – and they’re all important.
multiplying matrices.
(4) So matrix multiplication, and then, uh, come inverses.
Now let’s discuss the
(5) So we’re – uh, we – mentioned the inverse of a matrix, but there’s –
rules for matrix
that’s a big deal.
multiplication. Note
(6) Lots to do about inverses and how to find them.
that there are many
(7) Okay, so I’ll begin with how to multiply two matrices.
(8) First way, okay, so suppose I have a matrix A multiplying a matrix B and
– to do it and get
ways
giving me a result – well, I could call it C.
the same answer.
(9) A times B. Okay.
One way to multiply
(10) Uh, so, l- let me just review the rule for w- for this entry.
two matrices, A and
B, to get a new
Transcript from a linear algebra lecture (From Glass, Hazen, Lee and Wang,
matrix C is the
Analysis and Processing of Lecture Audio Data, 2004)
following rule:
Overview of TBALL
Technology Based Assessment of
Language & Literacy
•
•
•
•
• Intro
• Decoding
• Comprehension
• Discussion
Child-friendly testing system
Measure & analyze child reading & pronunciation
5-8 year olds
Native speakers of English & Spanish
Multidisciplinary collaboration:
o Engineering
o Education
o Computer science
o Linguistics
o Neuroscience
UCLA, USC, UCB
TBall Goals and Components
• Intro
• Decoding
• Comprehension
• Discussion
– Reliable and objective automatic reading
assessments
– Comprehensive instructional framework for
• Diagnosis
• Intervention
– Three main components:
• Children’s interface
• Teacher interface
• Assessment module
o Decoding (DAM)
o Comprehension (BARLA)
8
Decoding
– In alphabet-based languages, hints about
pronunciation
– If you know the language you are reading,
the hints may be enough
– If your first language is another language,
and/or if you know the LTS rules in another
language, there may be problems.
• Intro
• Decoding
• Comprehension
• Discussion
Comprehension
– MOSTLY depends on successful decoding
– Much more to learn than LTS rules…
9
Need to be balanced
Decoding
•
•
•
•
•
• Intro
• Decoding
• Comprehension
• Discussion
Necessary (but not sufficient) for reading comprehension
Necessary for reliable and efficient processing of text
Strongly related to speed of initial reading acquisition
A predictor of difficulties in comprehension development
Accounts for much variance in reading ability at all ages
English Letter to Sound Rules
Example: pronunciations of ‘o’
Pronunciations of the letter ‘o’ observed in the 100 most
frequent words of English
9 OW: go, home, most, no, only, over, so (know, own)
6 UW: into, to, who, do, (too, you)
6 AH: from, of, other, another, (some, come)
6 AO: for, or, on, often, (your, more)
5 OU: (about, hour; down, how, now)
3 UH: (good, goodbye would)
3 -: (people, work, world)
2 AA: got, not
1 W AH: one
• Intro
• Decoding
• Comprehension
• Discussion
Written Language, Complexity
Language
Letter
Sound
English
c
/s/, /k/
hot, car, walk, ah
/aa/
Spanish
a
/aa/
French
a
/aa/
allo, pot, eau
/ow/
Spelling
Sound
ache + past = ached
/ey k/ + /t/
aim + past = aimed
/ey m/ + /d/
paint + past = painted
/p ey n t/ + /ax d/
• Intro
• Decoding
• Comprehension
• Discussion
Decoding
Assessment
• Intro
• Decoding
• Comprehension
• Discussion
• Are the words read correctly?
• Decodability: closer to ‘one letter <-> one sound’ is better
In sample of 100 most frequent words in grade 1 material
– m, j, b, d… pretty straight forward (always /m jh b d/)
– Digraphs (2 letters -> 1 sound): -ng, -ck, (wh-)…
– 2 letters -> 2 sounds (or): th -> /dh, th/ (thy, thigh)
– 1 letter -> 2 sounds (and): x -> /k s/ (fox, box)
– G: get, thing, laugh, large
– E: because, get, been, they, great, often
– O: go, do, from, for, how, good, people, got, one
Decoding
Assessment
• Intro
• Decoding
• Comprehension
• Discussion
• Some words are more decodable than others
• High frequency words tend to be less decodable
• High frequency words tend to be ‘sight’ words (not
‘decoded’ but recognized instantly) (this is important)
• Decoding skills are also important
• If a word is well-known by a child, it may be treated as a
sight word and NOT decoded (tyrannosaurus)
• Therefore, unless we know which words are sight words
for a child, it’s hard to assess decoding skills
• Nonsense words must be decoded
Decoding Assessment Measure
Decoding Issues:
Dialect, Idiolect and Assessment
• Intro
• Decoding
• Comprehension
• Discussion
• If the child has trouble with the letter 'r' (common in English),
and says 'cow' when shown 'car' is it a misreading?
• If the child is from Boston and reads 'Carl' as 'Call' is it a
misreading? What if the teacher has never heard anyone from
Boston?
• if the child sees the word 'ask' and says 'aks'?
• if the child is a native speaker of another language, e.g.,
Spanish, and says 'seat' when presented the word 'sit', is it a
misreading?
• if the child is already learning to read, e.g., Spanish, and sees
the word 'sore' and pronounces it 'sore-ay', is it a misreading?
• SOLUTION:
– Map the child's system, whatever it is
– Augment the dictionary to diagnostic labels.
Dialect Modeling in ASR
• Intro
• Decoding
• Comprehension
• Discussion
Dialect is a system
The system is not well modeled in ASR
These are
being merged
When the point is the
Contrast system
Head sections adapted from Gunnar Fant’s Acoustic Theory of Speech Production
Decoding Issues:
Dialect, Idiolect and Assessment
Grape g r ey p
Grape g w ey p <r>
Grape g r aa p ey <Hispanic>
Bird b er d
Bird b uh d <r>
Bird b iy r d <Hispanic>…
Rocks w aa k s <r>…
Horse h ow s <r>…
Three th w iy…
Run w ah n…
• Intro
• Decoding
• Comprehension
• Discussion
Decoding Results
• Intro
• Decoding
• Comprehension
• Discussion
• nug: /n uw g/ - Sp. LTS, phonology, or short/long?
• jore, tay, bap: /hh ao r ey, t ay, b aa p/ - Sp. LTS
• yan, vag, hine: /zh aa n, b aa g, ay n/ - Sp. Phon.
• rin, bap, mot: /r iy n, b ey p, m ow t/ - short/long
• jore: /zh ow r ey/ - mixed Spanish phonology, LTS
• jick: /hh ih s k/ - Spanish LTS, misapplied English LTS
Comprehension Assessment
• Intro
• Decoding
• Comprehension
• Discussion
GRADE 2: …Kim has a dog. The dog's name is "Dot." Dot is a very
white dog, and he has a black dot on his leg. Sam, Ben, and Kim
like to play with Dot. They play ball with Dot all day long. Ben lost
the ball in the mud, and Dot went into the mud to get the ball.
Now, Dot is a brown dog from the mud! …
What does Dot look like?
Like a dog
A white dog
Please describe Dot’s physical
appearance?
A white dog with a black dot on his leg
White… and a dot … on his leg
Well he’s white and he has a brown dot on his leg
Like a rat
I don’t remember
A Dalmatian
Brown
Motivation/
Desiderata
• Intro
• Decoding
• Comprehension
• Discussion
• ‘Equivalent’ measures for reading or listening (to compare)
• Separate understanding of probe from understanding of text (probe as
picture adds interest and is not text or speech)
• Diagnose reading error patterns (by selection of alternate items)
• Easy to automate (compared to most comprehension tests)
• Items ascend in difficulty (several errors in a row stops the test)
• Many items (to allow retesting without repeating items)
• Equal number of plausible/implausible items (to mitigate guessing)
• PIAT-R (Peabody Individual Achievement Test – Revised) is best match,
but…
– Not automated
– Reading only
– They did not respond to our request
• Intro
• Decoding
• Comprehension
• Discussion
Accent adjustment
Prosody
Find words
Vocabulary
Syntax
Morphology
Print adjustment
Punctuation
Decoding (LTS)
In your first language, this might be easier
In a new language, this might be easier
BARLA Example
• Intro
• Decoding
• Comprehension
• Discussion
Hear OR Read:
1. (pot and box
reversed)
2.(dog in box)
3.(box on boy)
4.(correct)
The pot is on a box.
BARLA Example
• Intro
• Decoding
• Comprehension
• Discussion
Hear or read:
1.‘tire’ (rhymes)
2.‘fire’ (correct)
3.‘fur’ (vowel
differs)
4.‘fairy’(shares,
f, i, r)
Here is a fire.
Decoding
Opportunity
• Intro
• Decoding
• Comprehension
• Discussion
• We sampled the space with 25 items
– High Frequency Word: tell, nap
– Low Frequency Word: fell, zap
– Nonsense Word: zell, fap
• However, desirable would be
– Frequent retesting (perhaps in a game environment)
– Diagnostic of which decoding skills need more work
– Items need to be generated and coded for diagnosis
An Opportunity!
Content
Game
Letter sounds,
English:
a /ae/
b /b/
c /k/, /s/
…z /z/
By font, audio
files
p
Engine
b
p
d
Pedagogy
b
d
Levels:
Lower case
simple to complex
Upper case
Digraphs…
Move up if:
100% correct
Move down if:
3 wrong in a row
• Intro
• Decoding
• Comprehension
• Discussion
User
Model
Last played
Sounds mastered
Diagnosis
Preferences…
An Opportunity!
Content
Game
Vocabulary,
English:
…
bat
…
t
Engine
b
p
d
Pedagogy
b
a
Levels:
Lower case
simple to complex
Upper case
Digraphs…
Move up if:
100% correct
Move down if:
3 wrong in a row
• Intro
• Decoding
• Comprehension
• Discussion
User
Model
Last played
Sounds mastered
Diagnosis
Preferences…
A Comprehension
Opportunity!
• Intro
• Decoding
• Comprehension
• Discussion
Data collected for sentences analogous to PIAT-R
But not analyzed
Paradigm ready to hatch more uses
For more information about the project:
• http://nautilus.icsl.ucla.edu/tball/
For more information about the data:
iseli@ucla.edu
An Opportunity!
• Natural language processing and
reading augmentation
• Intro
• Decoding
• Comprehension
• Discussion
Simplify Vocabulary
• Intro
• Decoding
• Comprehension
• Discussion
• Simplify vocabulary
1. Select Text
2. Select
View
• Simplify
syntax
3. View
• Simplify both
By
By"augmenting
"augmentinghuman
humanintellect"
intellect"we
wemean
meanincreasing
increasingthe
thecapability
capabilityof
ofaaman
man
to
toapproach
approachaacomplex
complexproblem
problemsituation,
situation,to
togain
gaincomprehension
comprehensionto
tosuit
suithis
his
particular
particularneeds,
needs,and
andto
toderive
derivesolutions
solutionsto
toproblems.
problems.Increased
Increasedcapability
capabilityinin
this
thisrespect
respectisistaken
taken
to
tomean
mean
amixture
mixture
of
ofthe
thefollowing:
following:
more-rapid
more-rapid
By "adding
to apeople's
brains"
we
mean adding
to the way we
comprehension,
comprehension,
better
better
comprehension,
theadd
possibility
possibility
of
ofgaining
gainingaaccording
auseful
useful to
deal
withcomprehension,
difficult things,the
to
to understanding
degree
degreeof
ofcomprehension
comprehension
inaasituation
situation
that
thatsolutions
previously
previously
wastoo
toocomplex,
complex,
our needs, in
and
to
figure out
towas
problems.
speedier
speediersolutions,
solutions,better
bettersolutions,
solutions,and
andthe
thepossibility
possibilityof
offinding
findingsolutions
solutionsto
to
problems
problemsthat
thatbefore
beforeseemed
seemedinsoluble.
insoluble.And
Andby
by"complex
"complexsituations"
situations"we
we
include
includethe
theprofessional
professionalproblems
problemsof
ofdiplomats,
diplomats,executives,
executives,social
socialscientists,
scientists,
life
lifescientists,
scientists,physical
physicalscientists,
scientists,attorneys,
attorneys,designers--whether
designers--whetherthe
theproblem
problem
situation
situationexists
existsfor
fortwenty
twentyminutes
minutesor
ortwenty
twentyyears.
years.We
Wedo
donot
notspeak
speakof
of
isolated
isolatedclever
clevertricks
tricksthat
thathelp
helpininparticular
particularsituations.
situations.
• Intro
• Decoding
• Comprehension
• Discussion
Simplify Syntax
• Simplify vocabulary
1. Select Text
• Simplify syntax
2. Select View
• Simplify both
3. View
By "augmenting human intellect" we mean increasing the capability of a man
to approach a complex problem situation, to gain comprehension to suit his
particular needs, and to derive solutions to problems. Increased capability in
this respect is taken
to mean a mixture
the following:
more-rapid
By "augmenting
humanof
intellect"
we mean
increasing the
comprehension,capability
better comprehension,
possibility
of gaining
a useful
of a person to the
approach
a complex
problem
degree of comprehension
a situation
was too
complex,
situation. in
We
also meanthat
howpreviously
a person gains
comprehension
speedier solutions,
better
solutions, needs
and theand
possibility
of findingto
solutions to
to suit
his particular
derives solutions
problems that before
seemed insoluble. And by "complex situations" we
problems.
include the professional problems of diplomats, executives, social scientists,
life scientists, physical scientists, attorneys, designers--whether the problem
situation exists for twenty minutes or twenty years. We do not speak of
isolated clever tricks that help in particular situations.
Simplify Vocab & Syntax
• Simplify vocabulary
1. Select Text
• Simplify syntax
2. Select View
• Simplify both
• Intro
• Decoding
• Comprehension
• Discussion
3. View
By "augmenting human intellect" we mean increasing the capability of a man
to approach a complex problem situation, to gain comprehension to suit his
particular needs, and to derive solutions to problems. Increased capability in
this respect
is taken
mean abrains"
mixture
the following:
By "adding
to to
people's
weofmean
adding tomore-rapid
how we deal with
comprehension,
betterWe
comprehension,
the possibility
of gaining
a useful
difficult things.
also mean adding
to understanding
that
is suited
degreetoofour
comprehension
a situation
previously
was too
complex,
needs, and toinhow
we can that
figure
out solutions
to problems.
speedier solutions, better solutions, and the possibility of finding solutions to
problems that before seemed insoluble. And by "complex situations" we
include the professional problems of diplomats, executives, social scientists,
life scientists, physical scientists, attorneys, designers--whether the problem
situation exists for twenty minutes or twenty years. We do not speak of
isolated clever tricks that help in particular situations.
Simplify Vocabulary & Syntax:
Result
• Intro
• Decoding
• Comprehension
• Discussion
By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean
adding to understanding that is suited to our needs, and to how we can figure out solutions to
problems. By adding to what we can do, we mean a mix of several things. Faster and better
understanding are examples. We also mean making it possible to get some understanding when
before it was too difficult. And, we mean faster solutions, better solutions, and being able to find
solutions that didn't seem possible before. By "difficult situations" we mean the problems people
have in different kinds of jobs. The jobs can be in the government, running businesses, being
scientists, lawyers, or designers. The problems can be there for twenty minutes or twenty years.
We do not mean clever tricks that help in only some situations. We mean a way of life. In this new
way of life many things are useful. Useful things include hunches, cut-and-try, things we can't
touch, and our "feel for a things“. These things are more useful with great ideas, ways to talk about
the ideas and ways to write them, intelligent ways of doing things, and powerful technologies to
help us.
The number of people and what they create are growing fairly quickly. But our problems are getting
harder even more quickly. Also, the importance of finding solutions becomes greater because
things are changing more quickly than they used to change. Things that before happened only in
one place now are spreading to the whole world. This also makes it important to find solutions. If
we could make our brains better in this way, any intelligent person would run as fast as possible to
get it. To make them want to run, we need to show a way to do it. We also need to show what we
might get out of it.
• Intro
• Decoding
• Comprehension
• Discussion
Summary of Views and
Reading Ease
Original
Vocabu
-lary
Syntax
Both
239
275
249
315
7
7
15
22
Word/sentence
34.1
39.2
16.6
14.3
Characters/
word
5.6
4.7
5.6
4.4
Reading Ease
10.7%
42.3%
28.2%
71.7%
Flesch-Kincaid
Grade Level
12.0
12.0
12.0
6.8
Words
Sentences
Questions?
• Intro
• Decoding
• Comprehension
• Discussion