Integrating Latent Semantic Analysis and Language
Model for Character Prediction in a Binary
Response Typing Interface
Seminar on Speech and Language Processing for
Augmentative and Alternative Communication
Masoud Rouhizadeh
Introduction
 Most of the word or character prediction systems make use
of word-based and/or character-based n-gram language
models.
 Some works for enriching such language models with
further syntactic or semantic information. (Wandmacher et
al. 2007 & 2008).
 Predictive powers of Latent Semantic Analysis (LSA) for
character prediction in a typing interface developed by
Brian Roark (Roark 2009)
Roark's binary switch typing interface
 Binary-switch
 Static/dynamic grid
 Different language model contributions
 Different scanning modes:
Row-Column
RSVP
Huffman
Latent Semantic Analysis (LSA)
 A technique to model semantic similarity based on co-
occurrence distributions of words
 LSA is able to relate coherent contexts to specific content
words
 Good at predicting the occurrence of a content word in the
presence of other thematically related terms
LSA, an example of documents
1. The Neatest Little Guide to Stock Market Investing
2. Investing For Dummies, 4th Edition
3. The Little Book of Common Sense Investing: The Only Way to Guarantee Your
Fair Share of Stock Market Returns
4. The Little Book of Value Investing
5. Value Investing: From Graham to Buffett and Beyond
6. Rich Dad's Guide to Investing: What the Rich Invest in, That the Poor and the
Middle Class Do Not!
7. Investing in Real Estate, 5th Edition
8. Stock Investing For Dummies
9. Rich Dad's Advisors: The ABC's of Real Estate Investing: The Secrets of
Finding Hidden Profits Most Investors Miss
Preprocessing and tokenizing
 Tokenizing
 Removing ignored characters
 Turning everything into lowercase
 Removing stop words
Term by documents matrix
Cosine similarity
 Each word is represented as a vector
 A[0, 0, 1, 1, 0, 0, 0, 0, 0])
 B[1, 0, 1, 0, 0, 0, 0, 1, 0])
 0.4082
Integrating LSA and language model
 LSA is a bag of words model and is shown to be
reliable to predict a word within a context
 Making it more sensitive to context
 Pa is estimated by cosine similarity of w1 w2
 Pb is estimated by bigram probability of w1 w2
 P(w2|w1) = λPa + (1-λ)Pb
From word to character prediction
 In Roark's typing interface we are interested to predict
characters, rather than words.
 Sorting the upcoming words based on their
probabilities
 Evaluated by RSVP simulation
From word to character prediction
computer
association
accessories
arts
architecture
...
bags
backup
backpack
batteries
backgrounds
brands
...
_, e, a, i, c, f,
o, n, d, g, ,, t,
r, h, m, ., ", s,
l, p, b, -, u, "",
w, k, j, q , $, y,
v, x, z, :, ;
B
From word to character prediction
computer
bags
backup
backpack
batteries
backgrounds
brands
brain
...
a, r, e, a, i,
c, f, ....
A
From word to character prediction
computer
bags
backup
backpack
batteries
backgrounds
...
c, g, t, e,
a, i, c,....
C
From word to character prediction
computer
backup
backpack
backgrounds
...
k, a, e, a,
i, c, f, …
K
From word to character prediction
computer
backup
backpack
backgrounds
...
g, p, u, e,
a, i, c, f…
U
From word to character prediction
backup
p, e, a, I,
c, f, ....
computer
P
Evaluation
 Simulation mode
Trained and tested on a small part of NY Times portion of
the English Gigaword corpus
RSVP
Results
2400
2200
2000
Average key-stroke
per sentence
1800
1600
1400
1200
1000
Character frequency
scanning
LSA+Bigram
Model
17.79 % keystroke-saving per sentence
Conclusion
 Word-based language models shown to be effective in
character prediction
 Integration of LSA and bigram language model works
well in predicting upcoming words
 With larger LSA and bigram models we expect better
results
Thank you.