>> Merrie Morris: So thank you for joining us. For people joining online we have
enabled the live question asking features. So on resnet you will see a button and that will
send a question to me on a tablet and at appropriate intervals during the talk I will convey
your question to the speaker. So if you are watching online feel free to use that feature.
So good afternoon, I am really excited to be able to introduce you to Luz Rello who is a
post doc in the human-computer interaction institute at Carnegie Mellon. Luz also has
the distinction of being an Ashoka Fellow. She has a really interesting multi-disciplinary
background with degrees in linguistics, natural language processing and computer
science.
Her work is really interesting because it draws on all of those fields together to create
software to help people with dyslexia to be able to read more effectively. And for this
great work she has been recognized with many awards. So for example she was named
one of the MIT innovators under 35 in Spain and was also given a European Young
Researchers award. More than 100,000 people have downloaded and used the different
software that she has created around dyslexia, which I think for Luz perhaps more
rewarding than any of the formal awards, in fact she is even committed right now to
taking her work to reach an even broader audience through the foundation non-profit that
she has created called Change Dyslexia.
One of the goals that she is working towards over the next year or so is to begin to
establish a funding mechanism for that non-profit so that her software can reach an even
broader audience and can be made available freely to help the most people. So we are
really excited to have Luz here to speak with us today.
[Applause]
>> Luz Rello: Thank you. So actually I am the one who is excited to be able to share
with you the latest research and also research in the last year. So Merrie said everything
about me so I will go straight to the first –. Oh yes, so at CNU I worked with
[indiscernible] who is sitting there, that was the only detail. I will now go straight to the
talk. So first I am going to share with you a personal experience from my childhood. I
am sorry to share this, but I really want you to understand what dyslexia is so this is why
I am sharing this, okay.
So I was five years old, I was in kindergarten before entering school and this was the first
day that I thought there was something wrong about me, that I was not intelligent enough
or there was something going on. So all the students were sitting in a circle, the teacher
was in the middle and she was asking us to read one word that corresponds with one of
these pictures. So you know you are in the class, your turn is coming, so I wanted to
prepare myself to read this properly so no one laughed and I remembered perfectly how
the word was written, I remember perfectly how the picture was there, nothing was
wrong with that, but when I tried to read it is very difficult to explain, but something in
my mind crossed. I just couldn’t make it.
So very fast I was like, “Okay, okay my turn is coming. So there are 4 pictures, this girl
there is going to day the same word that I am going to say so I am going to pay a lot of
attention here and try to remember what she is going to say.” So she mentioned pata and
was like, “Okay I am remembering pata,” and when it came to my turn I said it and it was
right and the teacher passed to the next person. So it was like phew.
So this is what happens with dyslexia, you don’t know what’s going on, you try to keep
up with the rest, you have no idea you are doing it wrong because it’s a language
problem. You cannot compare to any write to any [indiscernible], because this how you
approach language and this is how you approach learning. The teacher also does not
notice. My parents also did not notice because you want to keep up with the rest. This is
why it is called a “hidden disability”, because it is very difficult to detect and to
recognize by everyone, even by people with dyslexia.
So an important fact: the 3 Pirates of the Caribbean have dyslexia. This is something that
children love and if you ever have the chance to speak with children tell them this
because this is very encouraging. What you can tell them in the nicest way is that
dyslexia is a cognitive disability, brains with dyslexia function differently. So if we go to
the left hemisphere where normally language is we see that in the case of dyslexia not all
the areas are activated when we read, but still there are still areas in the right hemisphere
that are activated when we read. So it is different, okay.
Three facts about dyslexia that we need to remember first are one that it is frequent, 10
percent of the population has dyslexia. According to the studies in the United States
there are even more, up to 17 percent. It is universal, we know it has a neurological
origin, but depending on the [indiscernible] of the languages these manifestations are
different. And third and more importantly is related to school failure, directly related to
school failure, 40 percent of the school failures today in Spain is due to dyslexia. This is
a statistic from the Ministry of Education. It is not related to general intelligence. So
there are children who work hard, give everything, they fail and they don’t know why.
So what’s going on in education is that we have like a funnel. So this funnel is given by
reading and writing, because what happens at school? At school if you want to learn
something you normally have to read it and if you want to show that you know something
you normally have to write it, this I how you are rated. So if you have problems with
reading and writing you have a problem at school and people with dyslexia, children, we
tried our best to enter into this funnel, but it is very difficult.
So our work is about how to widen this funnel, how to significantly improve the reading
of people with dyslexia. How to significantly improve the writing of people with
dyslexia and finally how to get rid of this tunnel by detecting dyslexia before children
fail, because nowadays the most frequent way to detect children with dyslexia is after
they start failing at school. So let’s start with the first part reading; how can we make the
text more accessible and the text more readable for people with dyslexia?
So let’s assume that the text has two parts: the text has a presentation point of view and
the text has a content point of view. So what can NLP do for making the content more
accessible? What different text presentation parameters do to make that test more
readable? Let’s measure this from using a reactive measure. So the way we measured
this was taking in different conditions from the presentation point of view, from the
content point of view. We will see this now and we make people read text under these
conditions. And we measure this reading using eye tracking. So these are the eye
movements of a person without dyslexia reading a text. So what we did is we took all
these fixations, we took the reading times and we found out, “Okay, which conditions
make this to be read significantly faster,” and for this we performed 17 experiments that
were in 2010 and 2014 in Spain.
So the conditions that we test in the experiments were from the text presentation point of
view. We made experiments testing these conditions from the content conditions. We
made experiments changing the frequency of the words, put them in different numerical
presentations, putting graphical [indiscernible] in the text to find out what was more
readable. We mapped these with the difficulties that people with dyslexia have and with
what NLP can do. So I am not going through now all the results. So you can ask me and
I can point you to the papers.
So out of all these experiments, what we did we had a set of presentations that make the
text more accessible for people with dyslexia that make them read significantly faster,
and this set of recommendations, and this NLP applications which basically was lexical
simplification. So [indiscernible] has been already integrated for tools; one reader for
Andriod, one reader for iOS. So this is how we show more simple synonyms on demand
and a server where you just type the word and the web page is put in a more accessible
way for people with dyslexia and a plug-in for Chrome. And with this plug-in if it can
show you a synonym or incase we don’t find the synonym it shows the definitions.
But it would be great to find this, not in a super specified tool for people with dyslexia,
but in a tool that everybody uses like Microsoft tools. That will make it more accessible
for everybody. They don’t have to download this specific tool and feel the stigma of
having dyslexia and having to use a special tool. But let’s go directly to the most
interesting experiment out of all the tracking experiments that we made. So this will
make you an idea of what we understand of dyslexia. Please read this test, you have 12
seconds because this is the average reading pace of a person without dyslexia, so read.
Okay, so how many errors did the text have?
>>: 7?
>> Luz Rello: 7?
>>: No I said several.
>> Luz Rello: So 7 I mean 1 number, come on take a risk 6 or 7?
>>: 7.
>> Luz Rello: 7, hmm well very good. So the text has 12 errors and these 12 errors are
errors make by people with dyslexia. So what we made is we put on the first test we got
people to read them, people without dyslexia, don’t tell me you have dyslexia,
mentioned, “Yes we saw that the test has errors, I don’t know 7 or 8 errors.” People
without dyslexia didn’t notice the errors. So we tell them, “Okay don’t worry you have a
second chance.”
>>: [indiscernible].
>> Luz Rello: With dyslexia didn’t notice the errors.
>>: Okay.
>> Luz Rello: So we say, “Okay you have a second chance. We are now going to read
the text again. Now you know that the text has errors so let’s go.” So the people read the
test again. The people without dyslexia then the got everything right like, “Okay there
was like 11 or even 12.” People with dyslexia starting to notice the, but still only saw a
few of them. So the most interesting thing about this is that when we went into
comprehension we found out that for people without dyslexia the fact that a text has error
made you comprehend more of the test. Your comprehension decreased significantly and
for the first time ever, ever, ever, people with dyslexia were better than people without
dyslexia. I mean the presence of the errors in the text didn’t make an impact in the
comprehension of the text.
So we couldn’t believe this because if something was true for all experiments that we
made is that people with dyslexia read significantly slower and comprehension is
significantly lower and what is good for people with dyslexia is good for people without
dyslexia. So this condition was like, “Okay we don’t understand it.” So we repeated the
experiment. We repeated it with fewer errors. We repeated it with more errors. We even
repeated with text with jumbo letters. Are you familiarized with this test that is at
Cambridge University? Okay, this was not at Cambridge University we did it, okay. So
how the text with jumbo letters affects the comprehension and what happens.
It happens that it doesn’t matter how the text mash up is for people with dyslexia the
comprehension maintains the same. It doesn’t matter if it 50 percent of errors, 16 percent
of errors, 8 percent of errors, no errors, the comprehension is the same, it doesn’t matter,
they have super powers. Well with people without dyslexia first of course they
comprehend much further and when there are more errors its like, “Okay what’s going
on?” So this affects comprehension and then when they realize that everything is about
errors they are on automatic pilot and then the comprehension goes up again.
So what can we learn about this? The thing is that in literature we always read that okay
yeah people of course, because people with dyslexia are blinded, this is why they don’t
see the errors. Then we were like, “No, but we have the recordings.” Then when we
went to the recordings we see that people with dyslexia actually see the errors. So there
is something here, they see the errors; this is like the first reading. Let’s do that again, I
really like this. They see the errors, they see them, okay these are two errors in Spanish,
but they consciously don’t see them because then when they are asked they don’t tell and
it doesn’t affect their comprehension.
So here we have something with dyslexia. This taught us that the errors of –. Yes?
>>: So you are saying they see the errors? They see the word right? You can’t
technically say that they see the errors?
>> Luz Rello: Well we know that they significantly stare more times in the words that
have errors, this is what we know.
>>: Yeah you don’t know if they saw it.
>> Luz Rello: Not if they saw, no consciously for sure they didn’t because from a
conscious point of view no, yeah they stare significantly more in the words with errors.
That is exactly the research result. Thank you for that point.
So what happened here? What did we learn from this? We learned that errors might
have the key and that they might be important. This is something that we learned. So
conclusions from the reading point of view we have set of content and text presentation
recommendations that improve the reading ability of people with dyslexia. This was
integrated in tools. They have tens of thousands of downloads, for you that probably
doesn’t mean anything, but for us we were like super happy to have these amounts of
downloads. We also found that errors are important.
Let’s go to writing. So now we are going to see how we can improve the spelling of
people with dyslexia. How can we improve their spelling when they are children and
when they are older? So this is the point where the errors made us think that maybe the
errors could have the key or could have something with people with dyslexia. So what I
did was I called all the mothers that I knew with children with dyslexia from all our
previous experiments and said, “Okay give me the errors now. Give me all the errors that
you have from your children, let’s crowd source all errors.” So we had meeting groups
with the mothers, they brought their notebooks and we took the errors, put them in a
database and then analyzed them. We analyzed them from all the points of view that I
could imagine.
So we analyzed them from the phonetic point of view, from the [indiscernible] point of
view, from the structural point of view, kind of words that the errors are and we found
tendencies in these errors and we compared these to the errors that were found in English
be this was done only in Spanish. We also saw that even the kind of errors that we found
in the text written by people with dyslexia was even comparable with English. So now
that we have the errors and the kind of errors that people have like, [indiscernible],
[indiscernible] and [indiscernible] of letters we took the errors and said, “Okay let’s make
exercises again with these errors.”
So normally when you are at school and you need to learn a word you have to repeat the
word 20 times to learn it. So here the methodology is the opposite. Instead of repeating
the word, the correct word for so many times is to give the child the error and maybe if
you get to play with errors maybe you get to learn from your errors. So taking into
account their errors we said, “Okay solve fix this word,” this is an error that they have,
cut this word, choose the correct ending. And for the distracters, so here is like
[indiscernible] right, [indiscernible] instead of [indiscernible], so for the distracters we
also took into account letters that we know that they are more likely to be confound by
each other because they share phonetic and [indiscernible] features. Everything based on
the error analysis that we made before.
We integrate this in a tool with two frames of mind and the tool is studied to
[indiscernible] and being used by used by professionals. We hear what professionals told
us so now the tool has graphic evolution. We also heard what the children told us so now
the penguin, at the beginning there was no penguin, so now there is a penguin and now
the penguin can grow, can dress as a dinosaur and all the things that the children told us
that they wanted the penguin to be dressed like. So here is how it works, this is a video
that I found off YouTube from [indiscernible].
[Video]
>> Luz Rello: This is segmentation. This is the very first version of Piruletras that we
had.
[Video]
>> Luz Rello: Okay so that was the idea and we were like, “Okay, but we need to find
out if this really works or not because our motivation is make this work in the real world.
So we went to a school in Barcelona to the Lestinnac School and did collaboration with
some psychologist’s from [indiscernible] children with dyslexia. We ran a [indiscernible]
study where the children had to play either Piruletras or word search. We found that after
12 sessions of playing them significantly improved their writing skills. We told them this
was a contest so they were super excited to make as many points as possible either
playing with word search and Piruletras. This is where they got their prizes, because
there were winners in the championship.
So what will happen when these children with dyslexia grow up? Well they need a spell
checker because in the real world you need a spell checker and there are some words that
the spell checkers do not catch. So for example real-world errors, errors that are correct
words, but are not the words that you were trying to write. You are much familiarized
with real-world errors. So you are like, “I am not sending the paper/I am now sending
the paper,” and these kinds of errors can actually make a lot of confusion and these are
not caught. This is a real world error from my e-mails and they are very frequent. So, up
to 20 percent of the errors that people with dyslexia write are real-world errors.
So we made a method that we will be presenting in a couple of weeks to correct realworld errors. So here again we started from the errors written by people with dyslexia
because now we know that errors have the key. So we collected errors, then from the
errors we created confusion sets. So we took the errors, we made the linguistic rules;
from there we made confusion sets. So these confusion sets are groups of words that are
very likely to be confound with each other. Then from these we made a probabilistic
model simple. We only took into account Google Ngram books and found out how
probable it was to find this word in this context in comparison with the other words
within the confusion set.
So we also crossed that information with a language model, so the language for
[indiscernible] University. So it’s like, “Is this Ngram likely to be correct Ngram?”
That is another filter and the second filter was a dependency person. That also gave us a
score to find out whether this Ngram was likely to be a grammatically correct Ngram.
And we made a comparison with a wildly used spell checker. So TexEdit, Pages,
OpenOffice, Microsoft Word, Google Docs and we found that our method is more realworld oriented so we could detect more real-world errors than the rest of the spell
checkers.
And when it comes to correction, so to show suggestions to the real-world error only
Google Docs overcame us. They have a very precise oriented algorithm. When they
suggest a word they are almost always right, but they suggest very few words. Most of
the errors are not highlighted. Then we also made a user evaluation with 34 participants,
half of them with dyslexia and they had to fix sentences and when real-world errors were
selected and suggested these made them to fix the letters significantly faster and a more
accurate outcome.
Okay, so conclusions from the writing point of view: we have a method to detect and
correct real-world errors for people with dyslexia. We have a method that significantly
improves the spelling of children with dyslexia. Both of these methods are based on the
knowledge that we can extract of the errors of people with dyslexia. The tool Piruletras
was downloaded 30,000 times in more than 70 countries and we won the first price of
Vodafone Foundation Mobile of Good Europe Awards, yeah! In Europe this is big.
Okay, so let’s go [indiscernible]. So we have reading and we have writing, but what can
we do to get rid of this funnel? We need to early detect dyslexia. So as I mentioned
before nowadays the most frequent way to detect a child with dyslexia is when they are
already failing, when they are already having difficulties in a school. So what if we can
detect dyslexia before it happens? This funnel will then disappear. So why is detection
of dyslexia so difficult? It’s like, “Okay we are not the first ones who thought about
that.”
So it is very difficult first of all you know it is a hidden disability, it is very difficult to
detect by everyone. The second is even in the UK where dyslexia is exemplarily treated
it is studied that only 5 percent of people with dyslexia in the UK have a diagnosis of
dyslexia. It is also expensive; a diagnosis of dyslexia goes form 150-1000 Euros. You
can also detect dyslexia using neuroimaging, but that is very expensive, maybe only
Angelina Jolie’s can do that and of course it’s not scalable because you need a specialist
or [indiscernible] to detect dyslexia.
So the first thing that we tried was to use the eye tracking data that we had from previous
experiments to classify readings of people with dyslexia. So we use all the data that we
had from previous experiments, apply support vector machine and we found out that we
could detect readings from people with dyslexia with 80 percent accuracy. And we think
this approach can be done to other languages because previous results in psychology have
shown that eye movements when reading dyslexia are different from people with dyslexia
and without dyslexia. But first this is not fun, because at the end you are reading text and
this is what you are doing when you are being diagnosed, you have to read tests and it’s
not fully scalable because you still need an eye tracker.
So while I was presenting these results last May, on a coffee break, at a [indiscernible]
conference I met Jeff and Abdula Ali from the University of Maryland and we were like,
“Okay this is not right, we are going to do games to detect dyslexia,” and this is how the
idea came because at the end of the day there are many dyslexia indicators. If we go to
psychology literature there are so many things that are indicators of dyslexia. I mean we
have phonological memory, working memory, visual attention, visual memory, there are
all these things that are there that are indicators of dyslexia, but if we take into account
only the eye tracking measures what we were actually targeting was reading speed
because at the end fixation ratio number were really [indiscernible].
So we started with an approach of taking into account orthographic processing and
phonological awareness and see what’s going on. So we implemented this in a game.
Let me do a demo of the game. We made 40 children, 20 with dyslexia and 20 without
dyslexia, play this game. It is in Spanish.
[Demo]
>> Luz Rello: Click over E. So it’s like a grid with letters. So this is like orthographic
processing. I want you to see levels.
[Demo]
>> Luz Rello: So now you have to find all the G’s. So you see it is not easy. So the very
first levels are very easy, but very soon it is going to get very complicated.
[Demo]
>> Luz Rello: So here we enlarge the grid, we start putting in more distracters and the
higher the level they get the distracters are more likely to be confounded with each other.
We do this also based in the analysis of the errors written by children with dyslexia. So
for example here we need to find the V. So now you see this very fast starts getting
complicated. So we will stop laying Dytective.
So we made this experiment with children from 7 to 10 years old and as you have seen
first they have to map the letters that they hear with the letters that they see, the frequency
of the letters, everything that makes it more complicated for people with dyslexia. Then
they have to map the sounds, they have to map the syllables. So we also take into
account how syllables are language, because these levels are like after these ones. If we
see where the errors of people with dyslexia are made worse they are made in these ones,
not in these ones. So we take into account which one is more, which are check syllables
like consonant, consonant, vowel. These are the syllables that are more likely to be
mistaken by with people with dyslexia.
So we make our statistical analysis and I tell you I have never seen a smaller “P” values
in my life. So it’s like this [indiscernible] and for every single variable that we were
measuring. So we will be presenting this in Lisbon in 2 weeks as a poster. So we were
like, “Okay we are on the right track.” Yes?
>>: So are those character anagrams selected manually by [inaudible] or are they
somehow?
>> Luz Rello: They were selected by me manually and I took into account, like you will
see now every single thing I know from the error analysis point of view, from the
linguistic point of view, linguistic [indiscernible], everything from psychology,
[indiscernible]. So we tried to cross as much information as possible. And right now
there are 212 exercises and it took a lot of time to –. I mean its single distracter is
thought.
>>: Are they in [indiscernible] or language specific?
>> Luz Rello: Right now they are only in Spanish, but we are developing it for English
and German. Yes?
>>: Do you see training effects of people [inaudible]?
>> Luz Rello: It is very soon to say this because right now the data that we have is only
for children that test only once. We only allow them to do it once. So I think there must
be training effects for sure as there always are, but I don’t have data on that. We will
soon, but I don’t have data on that.
>>: So I am not a researcher so when you say it is the lowest “P” value you have ever
seen?
>> Luz Rello: It means the most distinct the groups are.
>>: So you can very accurately detect [inaudible]?
>> Luz Rello: Not at this moment, this doesn’t detect. This only tells you they are
different, but this doesn’t detect. Now we are going to detection. This is like the very
first step when we were like, “Oh my god, oh my god, this is going to work,” but this
doesn’t detect, not yet. Can I carry on? Okay, okay, okay, so because we found
significant differences here we were like, “Okay now we are going for real.” We are
going through every single dyslexia indicator that we hear about and we are going to put
it into a game and see what’s going on.
So we took into account basically everything we hear about from the last 5 years. And to
make this content design apart from taking into account these dyslexia indicators we took
into account of course the errors as a source of knowledge. So every single exercise we
take into account all the patterns that we found in errors, so the linguistic and visual
patterns. The exercises that you have seen, they tell you linguistic [indiscernible]. So we
have 17 stages with 32 levels, 200 exercises manually, totally manually created and from
stages 7 to 12 we are using –.
Okay, so you remember the game Piruletras, the one with the errors, and the one with the
penguin. So from all the logs that we have from Piruletras from all this time we took the
exercises that were more challenging for people with dyslexia. So we thought that these
exercises were going to be able to distinguish better populations because they were more
challenging for people with dyslexia. So we also took into account the logs from
Piruletras and the game lasts for 15 minutes. So we have here some children in the south
of Spain playing these 3 weeks ago.
[Video]
>> Luz Rello: The video is not really good. We have to make a different one. Ah, this is
super difficult.
[Video]
>> Luz Rello: So to be able to apply much learning we need like a lot of participants.
We contacted the dyslexia association from Spain, well from all over, but the ones that
really collaborated were from these 4 countries. And let’s go now through the –. We
managed to have 243 participants, 95 with diagnosed dyslexia. So what these children
were doing; so apart from the levels that you have seen they were also recognizing words.
They were also recognizing numbers. So words that do not exist, but they hear. So this
proves like phonological memory and visual attention and of course the higher the level
gets the more similar these known words are to each other from a phonetic and
orthographic point of view.
They also had to do letter differentiations. So like find as many letters that are different
in this amount of time. So at the beginning letters were more different, but the higher the
levels they have like mirror letters and then they have rotation letters and also letters that
have phonetic similarly. This stages you are very familiarized with them because I
explained to you before these are based on the errors that the children with dyslexia make
based on the logs of Piruletras. So yes, find the right letter and reorder the letters. We
also took into account the frequency of the [indiscernible].
So that is there are some words that defer from each other only in one letter. So here for
example if you see this, Lunes this means Monday in Spanish. So Monday is super
frequent. So when you see this you are just like, “Okay Monday,” but then you find out
that here there is no “N”. So it’s is like Luces, which means lights and this is 50 times
less frequent than Monday. So we also play with the frequency of the distracters. Am I
making myself clear here? Okay, good.
So because we know that words that are phonetically similar and orthographically similar
are much more challenging for people with dyslexia. So we cross this with how the
language is acquired and the frequency, because the more frequent the word is the less
time it takes a word to be decoded. So did you remember the experiment of how to
detect errors that people with dyslexia couldn’t see? Well they could see, but not
consciously see. So we have two levels about this.
So here they have to detect errors that affect the functional words and here they have to
detect errors that effect lexical words. So this way we are addressing syntactic awareness
and lexical awareness. We also take into account location, so how frequent this group of
words appears in language and this task for people without dyslexia is very easy, but for
people with dyslexia is really challenging.
Okay, we are finishing here. The other one they have to memorize a sequence of letters.
This is very much used in intervention. So we start with vowels and consonants and then
we start like mixing consonants that are alike with each other and also taken from
phonetic features and visual features. Then we end up with capital letters. So they also
have to make the transformation of finding the letter and then finding the keyword. Then
we end up with word writing and non-word writing. So again the same kind of
[indiscernible] that we take into account. So the logs, the error analysis, the frequency,
how language is acquired and we make the right words using this key word or to write no
words. So they have to remember.
So basically we went through all the dyslexia indicators that we thought could be a game.
So we are not taking into account for example like speech and other indicators, only
being a game. So now we can detect [indiscernible]. So here we use a straining data and
the performance of 243 people from 7 to 70 year old and from each of the people we
could gather almost 200 features. We had almost 50,000 data points. So we also took
into account the age, the gender, the second mother language, because they always had
Spanish as a mother language, but you know there is transference of languages. So there
are many people who are bilingual so we also took this into account and whether they
have ever failed a Spanish subject. We performed cross-fold validation using Support
Vector Machine because that is what worked the best. And we could predict whether a
player had dyslexia with almost 86 percent accuracy.
Then we made very basic feature analysis, but we found that every single stage matters.
So if we –. Yes?
>>: How do you get ground truth for whether they have dyslexia?
>> Luz Rello: Because we ask them for the diagnosis. So every single person with
dyslexia was diagnosed with dyslexia. So our ground truth yes, that’s a very good
question because of course sometimes, diagnoses fails. So they are diagnosed with
dyslexia by a professional and then one year later it turns out that it was attention deficit
disorder and not dyslexia. This is a weakness of our research. We take professional
diagnosis as our ground truth. So in future research we what we want to do, we already
have it arranged with two medical doctors, is that we will take a percentage of our results
and the children will be evaluated by them manually. So then we will find out in the long
run how our algorithm is, but right now our ground truth exists in diagnosis of dyslexia.
>>: Can you guess what the cross correlation is between two professionals making a
diagnosis?
>> Luz Rello: I don’t know, no. I tried to find this data and I couldn’t. I mean I know
from personally speaking some families that told me, “Oh my child was wrongly
diagnosed.” I don’t know, so it can also be due to –. I mean hospitals because of trying
to protect their privacy they don’t disclose this data. So this is something –. I don’t
know, if any of you know I would love to know.
>>: My question is what conclusion do you draw from that? The conclusion could be
that it’s less accurate. The conclusion could be that’s its more accurate.
>> Luz Rello: I think it’s more accurate to be honest because just to give you an example
that happened last night one of the participants is a family that we are very close with, so
there are two doctors. One doctor has, how do you say this in high intelligence?
>>: Yes.
>> Luz Rello: She has a really high intelligence. So she made the test as a control group
because she has no dyslexia. Then when I was doing error analysis I found out that she
was classified as having dyslexia, but her scores were high. I was like just incase I am
going to ask the mother. I asked her, “Do you know by any chance she has dyslexia?”
She was like, “Well she never failed a subject. She is a really honored student.” But
because we really trust your research we know you from like 5 years we are going to
spend our money and make a diagnosis. It turns out I have the e-mail from last night that
she has dyslexia, really.
So these results are very recent so we don’t know yet for example how they interrelate to
each other. I mean given this it is for sure that not only a higher score in the game with
determine this, it is also how the different cognitive skills that we address interact with
each other. But these are very recent results. I don’t know yet, but I think we are on the
right track of detecting dyslexia. But we don’t want to diagnose, we only want to detect
and make sure children in school don’t fail. Our motivation is not to diagnose dyslexia;
our motivation is to avoid unnecessary school failure only by warning parents and
teachers. Not to put attach into a child that they have dyslexia, but just to say, “Hey pay
attention.”
>>: So you made a comment earlier that the eye tracking wouldn’t scale because of the
eye tracking hardware that I am assuming that’s needed.
>> Luz Rello: Yes that’s what we thought and also because you need someone who
knows how to use the eye tracker and not only the hardware, but the person needed. So
here if we put this into an application disk and go to schools like [snap] and it is cheaper.
The idea of this with dyslexia where there is no profit or organization is to really try to
bring this for free if possible.
>>: Put the test makers out of business.
>> Luz Rello: Yes I mean some companies have contacted me and I don’t know maybe it
is because I have dyslexia I don’t want to make a business out of it. So I really believe
that detection and intervention must reach everyone, but of course to make it sustainable
we need to make a little bit of business. We need to have a technical service. It has to be
something that really works. So we will see, I mean if we find enough sponsorship then
it will be for free. If we don’t find enough sponsorship then it will cost a little bit, but to
make it sustainable because right now our tools that are up there we are implementing
them in our free time and the more system, how do you say? Well every time that there
is a new system for lists we have a problem.
>>: I have an idea. I just spent 20 thousand dollars on 2 daughters to get some full battery
dyslexia test and that seems crazy.
>> Luz Rello: And for some families this is not possible. I mean in Spain especially I get
e-mails from some families that they don’t have the money to pay for this.
>>: [inaudible].
>>: When you say there is 85 percent accuracy in the diagnosis are the errors mostly in
false positives or false negative?
>> Luz Rello: I will explain to you. So we have like 3 kinds of errors here. So for the
little children the errors are children that are tagged as dyslexia and they are not dyslexia.
The reason here is because we think that with reading education and with language
education each child has a different pace. So I think that children that are in a lower pace
are tagged as dyslexic and then they are not. So these are the other things. Then for
these ages, so for late teenagers in their twenties, the kinds of errors that we get are
people who are dyslexic and are tagged as non-dyslexic. So here what is happening is
that these people were diagnosed with dyslexia when they were children and they already
had like a lot of treatment and they somehow overcame dyslexia. So this is why our
algorithm is tagging them as non-dyslexic.
Then for the older people we find both kinds of errors, people who don’t have dyslexia
that are tagged by dyslexia and vice versa. And here what we think, we don’t know, but
what we think is that when you are grown up the more you read the better you get. So
with grownups I really depend on the job that you have and many other things. I mean
how exposed you are to text. So we think it may be that, education issues, but these are
all hypothesis. We haven’t gone through the data and seen each single reason of what
could be. Yes?
>>: So these results are really interesting. I am just curious if a person has some other
disorder would they still be classified as dyslexic? So how unique are these results at
classifying a dyslexic individual verses an individual who has say a visual challenge or
attention deficiency?
>> Luz Rello: That’s a great question. At this moment we don’t know. This is very
recent, like from 2 weeks ago we got this. So I don’t know when we get more data and
we hope we get more data we will know. So right now we are asking in the test, “Is there
anything else, any other disability, any other difficulty, any thing else that you think we
need to know?” And people are writing like, “I have dyscalculia. I have dysgraphia. I
have attention deficit disorder.” So we are gathering all this data, but at this point I don’t
know.
>>: Let me ask a previous question in a different way. Do you know what the false
positive array off this classifier is? Meaning that what percentage of half the population
is classified as positive?
>> Luz Rello: At this moment I don’t know this. So we know that 50 percent are not
currently classified, but I don’t know out of this 50 percent I don’t know which –.
>>: The reason I ask is because you used cross-validation on the data set that’s balanced
in a different way than in natural population.
>> Luz Rello: Ah okay I will explain to you how we manage this. So how we do it is we
test each individual differently. So we have the training data and we take one person and
this person is classified. Then we take another person, then we take another person and
we are not doing cross-validation. So because we are only classifying one –.
>>: You leaven one out.
>> Luz Rello: Yes leaving, yes.
>>: That doesn’t change the fact that the grace population is different from your 250
verses 90 something subject ratio.
>> Luz Rello: So there are two things here: so first the experiment that we are doing now
is going to be with 90 percent population without dyslexia. This is how we have we have
enrolled all these schools. So we are solving this thing first and secondly I am not sure
how open this question is because I ask people at CMU from the matching learning
department about this and they told me that we were doing the right thing, leaving one
leaf out. So I am not a matching learning specialist myself.
>>: Okay we can talk about it.
>> Luz Rello: So we can talk about it, because I would love to know more about this.
>>: But the random baseline, like if you were just randomly to guess, that’s 50 percent?
>> Luz Rello: Yes, yes so that is one by one.
>>: Following off the previous question I am just wondering if you have thought about
mixing in non-reading tests, math tests, math games, things like that?
>> Luz Rello: We will do this. It is not in this test, but we will do this with math and
with seek.
>>: Does dyslexia itself happen on a scale or is it binary?
>> Luz Rello: So it actually does. So our results are binary as diagnoses of dyslexia are
binary, but psychology research shows that there is continuum. So yes, and our ground
truth are with, yeah. Okay I will go to the last slide, yes.
So conclusions, okay we have a new method to detect dyslexia which is based on
dependent measures that haven’t been used before that are derived from games, attention
and linguistic. It is cheap, it is scalable and we only need to have connection to the
internet to use it.
>>: A quick question, what’s the age? What’s the earliest age a kid you would give a test
to?
>> Luz Rello: So right now this only works from 7 years old to adults. This was because
we only could find ground truth from 7 years old, because children start being diagnosed
at that age. They don’t start being diagnosed before. So this is why our results are from
7 to adults. We are going on the 22nd of this month; we are running an experiment in
Chili, in a kindergarten with not language related exercise, but with figures and attention
related exercises to find out if we can do early detection before we read, with pre-readers.
And for having results of this I think we need to wait for a couple of years until they start
develop reading so we know if this is going to happen. So from this result we only can
show the ones that we have a ground truth, so from 7 years old even though it can be
done from 6 years old. We made the pilots and 6 years old can do it.
>>: And I know dyslexia is manifesting differently on different languages. So English is
probably the hardest one for dyslexia people and it is manifested earlier and stronger than
other languages. So this research is done in Spanish. How about other languages? For
example I know Russian is not manifesting as early or strong. It could be hidden
[indiscernible]. With English it is manifesting right away.
>> Luz Rello: Yes so right now it is 3 and we are doing it in 2 more languages. So for
English it is [indiscernible]. Yes for English the main way to detect dyslexia is due to the
errors where in Russian it is reading speed. This is how dyslexia is mainly detected
because speed influences [indiscernible]. In English it is more about the errors. For
German Maria Rosenberger will adapt the method for German. So for this we need to
start from the beginning, start analyzing the errors written in English and German which
are very different from the ones written in Spanish. Then how this dyslexia indicators
differ from different languages. So in Russian we don’t know. I know it’s also –.
>>: It’s very different.
>> Luz Rello: Yes it’s very different, but the [indiscernible] is consistent. So this is
where the manifestations of dyslexia are more hidden as in Spanish.
>>: And you mentioned that you graded them and everything manually yourself. How
will it be done in English and other languages? Will it be you work with someone?
>> Luz Rello: Yes.
>>: To see that it will be right.
>> Luz Rello: Yes, yes we are working in teams. So for example with Maria the game
you have seen Piruletras is already available in German and the game for writing skills.
It took us 8 months to replicate the methodology. So she went to the schools, analyzed
the errors and for analyzing the errors we did it together because we had to come out with
new classifications of errors. So in German you have like capital letters, you have
segmentation’s that are different; you have words that [indiscernible], you know all these
things. So the methodology can be transferred, but it needs a lot of work. It is not
straightforward, no.
>>: And your main focus to detect dyslexia are also to train or treat, not treat, it’s not
treatable. To train a person to –.
>> Luz Rello: So that is our challenge. So our future work is tree lines. So the first line
is using not early detection, but also intervention. So we know that at least levels from 7
to 12, the ones that were based in the previous game, we know that they can be used for
intervention. So we want to do this. This is the first line of future work. The second line
is doing it in different languages. We are staring right now and the other things are
dreams to just really make this available for everybody.
>>: Do you work with specialists in dyslexia? I know several here working with
children, they are very successful.
>> Luz Rello: Fantastic, put me in touch with them please.
>>: Yeah because they have special lessons and they are very successful.
>> Luz Rello: Please put me in touch with them. I know in the Spanish community I
know them. For English I know the people that are in the National Association of
Dyslexia in Miami and Texas with successful methodologies, but in this area if you can
put me in touch that would help us a lot for the English part that is coming.
>>: I will.
>> Luz Rello: Okay, so I thank you very much.
[Applause]
>>: On your last slide you mentioned that you are doing some things [inaudible].
>> Luz Rello: Yes so what we are doing, so we have like 17 stages, so what we are doing
is the ones that can be integrated [indiscernible], because not all of them can be
integrated, put them in [indiscernible] and then make them share in the social media. So
we finally know how many people with dyslexia are in the web. If many people see this
in their Facebook probably dyslexia will stop being stigmatized and start being normal,
not a hidden disability. I think [indiscernible]. So we are doing this with Katarina, I am
sure you know her. We are doing it in English and in Spanish, but for doing this first we
need to really find out which are the stages that play the greatest role because right now
the game lasts 15 minutes and for [indiscernible] we need it shorter. This is wonderful
current work.
>>: You had a link to a demo site.
>> Luz Rello: Yes.
>>: Can you please site that for me.
>> Luz Rello: Yes, but this demo is only 3 stages. If you want to get to the whole game
it is dytective.com. They will ask you for a special code and I will send you the special
code. You can just put “test please” and you can see the whole test. It is in Spanish, just
to let you know. It will soon be in English.
>>: So I guess a lot of the question is: How can we help you? I love what you are doing
here. Is there anything we can do to help?
>> Luz Rello: So many things. So, one of the challenges that we have is finding for
example participants. So when we do this in English if you can somehow advertise this
study in your web site, in your social media this will help us a lot. Another thing, when
this is available for English and we want to bring this to schools there are some schools
that have computers, but some other schools do not have, especially in South America.
So if Microsoft has like a donation thing of tablets or devices that you don’t use you can
donate to us. I mean there are so many ways.
>>: Well is there any way we can stay in touch with you to find out what you need?
>> Luz Rello: Please. So give me your contact. I mean there are so many ways. This is
just starting. So we are just starting the non-profit where this is like a first preliminary
result, the experiment, this is only 250 people; we are aiming to 10,000 to have free
results. So there is so many ways you can help us, really.
>>: Are you in contact with Dyslexia Association which is here in the Northwest?
>> Luz Rello: No, please put me in touch.
>>: Can you repeat that?
>>: Dyslexia Association. So people are working with dyslexic children. They are
testing the, teaching them, they are doing lessons.
>>: Like Hamlin Robinson.
>>: Yes.
>>: There is a dyslexia school in Seattle called Hamlin Robinson that’s pretty well
known.
>> Luz Rello: Let me write this down. Can you tell me now the –.
>>: I will send you by e-mail the contact person.
>> Luz Rello: So right now in English we really need the contacts. I have been working
in Spanish for 5 years, but not English.
>>: They will have many, many participants for your Spanish, because they have children
for [inaudible].
>> Luz Rello: Great, that will be great.
>>: It’s called Hamlin Robinson. We will connect with them.
>>: So we are going to work with Liz, but I should introduce myself just briefly as for
help. So Mike and I –.
>>: This is Jeff Petty.
>>: Yeah I am Jeff Petty. Mike and I lead the learning tools for OneNote Hackathon
Team that set out and we worked with folks like Robert [indiscernible] and Greg
Hitchcock. We worked with over 20 people from Windows, and from Office and from
Microsoft Research to create an inclusive reader. We explicitly set out to leverage
proven interventions for people with dyslexia, but to create a non-stigmatizing tool that
would benefit all readers. So we met earlier today with Liz and we are going to get her
feedback on the tool. She already has some recommendations for additional things that
we can do to improve it. We are going to have a relationship where we are trying to
figure out how to support Liz on an ongoing basis. We will figure out what that looks
like.
>>: That’s great.
>>: But I think one of the asks that we have is that we would like to get the mailing list
from today and just reach out to everybody and say, “If you are interested in contributing
or just being aware of what we are doing we would love to take advantage of that.”
>>: And we have an Alpha out to a limited number of schools. It’s a OneNote add in
currently. We are hoping to get it built in to maybe all of Office or maybe all of
Windows, but even now like Hamlin Robinson is one of the schools that we are trying to
work with and there is a school in California and some schools locally. We are just
trying to get Alpha feedback and then improve it and hopefully release a Beta in mid
November time frame that’s public. So we are just basically trying to get feedback right
now. Also talking to experts because we have got some good experts here, Greg and
some reading/font rocket scientists at Microsoft, but we are also trying to meet with other
people outside of Microsoft who have expertise.
>>: So when you say, “Help from this group,” what kind of help [indiscernible]? I came
today because I have a daughter that’s been, oh in second grade she had ADHD and then
we had her diagnosed with Exotropia, a tracking issue, and that was causing her delayed
reading. Now people are telling me they think she has dyslexia. Are you looking for
feedback and experiences or are you looking for my child to come and use your tool?
>>: All of the above. So in some cases we have a dog food group so people are getting
early [indiscernible] of the software and they are providing feedback. We also have
people that are active in the dyslexia community and they are just helping us make
connections, like the Eide’s. I don’t know if folks know the Eide’s, but they have written
the book The Dyslexia Advantage. They happed to be, in addition to being leading
thinkers about dyslexia, they happen to be local and are also willing to plug-in and help
us. Then we are working with folks like Marianne at the National Language Processing
team and doing some other things in August. So we interested in building a community
within Microsoft and with external partners like Liz to help move things forward. People
can contribute in lots of different ways. So if you are interested we will let you know
what’s going on. Then if you have a specific contribution you want to make we are
pretty open to folks wanting to help.
>>: And how do we contact you?
>>: Jeff.Petty@microsoft.com, but Merrie did everybody accept the appointment?
>> Merrie Morris: I don’t know that’s necessarily –.
>>: [inaudible].
>> Merrie Morris: I think the appointment mentions me at the bottom as a host. So if it’s
easier for you to look me up and then I can forward you mail to Jeff. So either you can
contact Jeff or I and I will make sure the information get’s to Jeff.
>>: Yeah.
>> Merrie Morris: Did you say there was a Yammer group as well?
>>: Yeah. So we will get you all that stuff. We have a Yammer group where we post
new builds and then we’ve got an alias. If you have contacts in the community too we
are interested. We don’t want to be flooded with 100 people in the community, but if you
know people who can connect us or give us greater feedback that’s definitely perfect.
>>: What’s the Yammer group?
>>: The Yammer group is learning tools.
>>: Dog food?
>>: Search for learning tools on Yammer and it should pop up. There is probably a
learning tools dog food or learning tools something or other. Learning tools is sort of the
brand we are calling it. Again we ultimately hope to get this technology into Office and
Windows, then again maybe for everybody and not just a high powered expensive tool.
>>: So you two are connected to Liz?
>>: Yes to Liz, definitely, recently, but yes.
>> Merrie Morris: Well thank you all so much for coming and thank you Luz.
>> Luz Rello: Thank you.
[Applause]