Vivek Punjabi
Andrew Menzies
CS 2610
Homework 1
Virtual Keyboard
1
INTRODUCTION
Our goal was to make a virtual keyboard program on the computer screen with the ability to type by
clicking on the keyboard and by tracing words on the keyboard. The program would also provide
suggestions based on the user input and give feedback to the user based on that input. While designing
the program, we chose to prioritize accurately reading the user’s input over maximizing the potential
speed at which the user could type.
2
LAYOUT:
Our program layout includes a text box for the goal sentence at the top where the user can enter the
goal statement (using a physical keyboard) through the Enter Goal button. It also has a text box that
shows the actual words the user typed right below the goal sentence, so that the difference between
the two is easily visible. We have five buttons, one of which is the current word button which stores the
default word to be chosen. In Tap mode, this is the exact sequence of characters the user entered,
while in Trace mode, it is the best suggestion found for the user’s swipe. (The exact string of characters
they swiped over is unlikely to be a sensible word). We also display four suggestion buttons displaying
words the program thinks the user might be trying to enter. The suggestions are displayed in order such
that the most likely match from our dictionary will be saved to the leftmost suggestion buttons as it is
the closest to the current word button. Then, there is the layout of the keyboard of the traditional
QWERTY-style 27 key (26 alphabet keys and a space key) keyboard. Our project has an option to select
between the Tap mode and the Trace mode where the user can use only the tap features or only the
trace features.
3
USAGE:
Our project by default begins in the tap mode where the user can simply click the keys from the virtual
keyboard, add a goal statement, select suggestions and compare their input with the goal statement.
The current word button shows the user’s input and the suggestion buttons show up the top four
suggestions based on matches from the dictionary file. The user can either press the current word
button or press Space key to select the user’s input without any correction or select one of the
suggestions that might appear before completing the user’s intended input. The project can be switched
to Trace method by simply selecting the Trace mode at the bottom right corner. Here the user is allowed
to trace over the keyboard and he may select from either the best possible suggestion in the current
word button or less likely suggestions from the other suggestions buttons. We have added some
features to improve the suggestions and efficiency of our algorithm in Trace mode.
1. The user can enter double letters in any word by simply traversing through that key just once.
For each letter traced over, the keyboard actually puts two copies of that letter into the
sequence of letters to search for, meaning the algorithm will find words both with and without
the double letter at that point. The user can move off a key and back onto it, though, to
increase the chance that a word with a double letter at that point will be suggested.
Vivek Punjabi
Andrew Menzies
CS 2610
Homework 1
2. If the user accidentally extends a trace one key beyond the last letter of any word, our algorithm
will check for the suggestions with the starting key and the ending key as well as suggestions for
the starting key and the key just before the ending key. This improves suggestions at the cost of
an (imperceptible) increase in dictionary search time.
4
FEEDBACK
In order to help the user understand what their actions are doing, the program provides several
feedback mechanisms. Since the keys do not look exactly like typical application buttons, in both Tap
and Trace mode, hovering over a key turns the key green (showing it responds to input) and clicking it
turns it blue. While the keyboard is in Trace mode, the line the user traces is drawn in light blue to make
it clear where exactly the user’s tracing has been detected (and which keys the trace covers).
As the user enters a word, before the user has pressed the Space key or chosen a suggestion, the text
for that word in the user text box is black. After the user confirms a word (by pressing Space or clicking
the “Current Word” or any of the Suggestion buttons), that word in the User and Goal text boxes
changes colors to show the user’s accuracy. Letters in the corresponding goal word that the user
omitted turn yellow. Letters that would have to be inserted in the goal word to produce the user’s word
appear orange in the user text box. If a letter in the goal word does not match the letter in the
corresponding space in the user word, then that letter in the goal word and its counterpart in the user
word both turn red. Green letters indicate matched letters. As an example, consider the goal sentence
“brown fox” and the user input “bbrow fix”. The result is
Goal: brown fox
User: bbrow fix
In each case, the program decides which letters are considered omitted, extra, or mismatched based on
which choice results in the lowest edit distance between the words, where each letter designated as
“omitted”, “extra”, or “mismatched” counts as 1 unit of distance. The colors make it easy for users to
see if they have any mistakes, and with a bit more effort, they can figure out what their mistakes were.
5
WORD SUGGESTIONS
For the two modes of input, we use slightly different suggestion-finding algorithms. However, both
make use of the same data structure. When the user starts the application, it reads through the
dictionary file to find each word and its frequency of use. It constructs a 2-dimensional 26-by-26 array
of tries, where the trie in array index [i][j] contains all of the words that start with character i and end
with character j (where character 0 is A, character 1 is B, etc.). We chose this structure because in both
modes, we know the starting letter of the user’s word, and in trace mode, we know the likely ending
letter that the user wants. Thus, this structure narrows down the searching the program must do.
To find suggestions in tap mode, the program moves down each trie which starts with the first letter
tapped, entering the nodes for each successive letter in order. It then recursively visits the rest of each
of those tries’ nodes below the node for the last letter the user tapped. It keeps track of the four words
with the highest frequency values that it finds this way, and displays them as suggestions.
Vivek Punjabi
Andrew Menzies
CS 2610
Homework 1
In trace mode, the keyboard portion of the program feeds the search algorithm a sequence of (letter,
necessity) pairs in the order that the user swipes over the letters. Necessity indicates how likely the
program believes the letter is in the word the user wants. The first and last characters swiped have a
high necessity, others have a low necessity, and duplicates of letters that are added to account for
double-letter words have no necessity (so a word with double letters will not necessarily appear over a
word with one of those double letters reduced to a single letter). The program searches in two tries,
both beginning with the letter the user pressed the mouse down on, but one ending with the letter on
which the user released the mouse and one ending with the letter the user traveled over before that
one. The algorithm finds all words that are subsequences of the sequence of letters, ordering them first
by the total necessity of the matched letters and then by highest frequency. The five best words appear
as suggestions.
6
DESIGN INSPIRATION AND CHANGES
Much of the inspiration for our original design was the swipe keyboard for Google phones, which we
briefly tried out before starting programming. Our final application is close to what we originally
planned. However, a few features are different. For example, the “Reset Input” was not originally part
of our plan, but we realized that often we wanted to try typing with the same goal multiple times, and it
was tedious to click “Enter Goal” and reenter the same goal to reset the user input text area. We also
originally planned to use angle detection to give certain letters higher necessity, thinking that if the user
makes an acute angle on a key, that user wants to include that letter (otherwise, the user would have
swiped past it in a straighter path). However, we decided against including this for two reasons. First,
our simple subsequence-search algorithm where each letter except for the first and last has equal
necessity worked well in our testing—there were no cases we found where a word was in the dictionary
but did not appear as a suggestion when we traced it. Second, while testing we noticed that in trying to
swipe quickly, we would often overshoot a key we meant to include, swiping completely through it
instead of turning sharply on it. Favoring keys where we made sharp angles was not likely to bias the
suggestions in favor of the words we wanted. This tendency to overshoot keys is also what motivated us
to look for words ending in either of the last two letters swiped over, rather than just the last one.
7
TRADEOFFS AND ROOM FOR IMPROVEMENT
In a few ways, our design had to sacrifice speed and convenience for accuracy. We require the user to
use a radio button to change input modes so that we read the user’s intentions correctly (if the user
drags the mouse slightly when tapping a key, it doesn’t begin a trace). Also, it would be faster to
automatically confirm taking the best suggestion for a swipe if the user ends it and begins another swipe.
However, we thought it was more important that the user be able to retry a swipe that failed to bring up
the word he or she wanted as a suggestion. Thus, beginning a swipe without confirming any suggestion
from the previous swipe cancels the effects of the previous swipe.
Some possible avenues for improvement involve searching for words even if the trace does not cover
every letter, searching for suggestions even in the middle of a swipe, and using sentence clues to predict
the user’s next words. Also, given more time, we would have liked to have the blue trace line appear
with less flickering, though the framework the application is built on (javax.swing) made writing code to
draw the line difficult.