1

1

>> Andy Wilson: So I think we should get started. It's my great pleasure to have Jake Wobbrock over this morning to give us a talk and update on his latest work and his group's latest work on touch typing on touch screens. And Jake has been a great partner and a great collaborator with MSR over many years. In fact, he was one of my very earliest interns back in 2003.

He was an intern in the surface computing group.

>> Jacob Wobbrock: I was only this tall then. I grew up.

>> Andy Wilson: So it's been great to watch him progress through his career, and he's now an associate professor at the I-school, University of Washington and is kicking butt basically. So anyway, let's hear what they've been up to.

Thank you.

>> Jacob Wobbrock: Thanks, Andy. So this talk is called from plastic to pixels, in pursuit of effective touch typing on touch screens. Of course there's been a lot of work on text entry for touch screens and a lot of that work has taken the form of stylus and then kind of single finger and thumbs.

We're actually looking at, as the screen shows, touch typing. So really trying to get to where we can put our hands down and type as you would in a sense on a plastic keyboard, hence the title here.

And so part of the questions are can we do such a thing, and can we do it well.

And if so, how.

A brief self-introduction. Although given the audience, I probably didn't have to do this. But you all know me already. But this is just from the dub retreat on Friday. Thank you to all of you who could make it. And I mentioned a couple areas we're working on.

Today, I'm talking about accommodating of situational impairments in mobile use. So you can see some images here for some mobile text entry stuff and then at the bottom some predictive and adaptive input techniques generally.

All of these people here deserve a lot of credit and in particular, today I'll point out a few in a few moments. So my roots are in entry. My dissertation work, as some of you know, is in text entry on some EdgeWrite that I created and basically put on me device that had edges and even many devices that didn't

2 have edges, like track balls and isometric joy sticks and even an eye tracker.

I've gone on and done quite a few things since then, but it just seems to be inescapeable that text entry keeps coming back into my life. Somewhat reluctantly on my part, I'll be honest, but it's still a rich area with a lot of challenge.

This is, for example, the four-button version of EdgeWrite which is really just a demonstration on a number pad of how you can enter kind of gesture texts about without a gesture surface, just by punching discrete keys that correspond to corners. People could write actually reasonably quick, 15 words a minute or so, just kind of twiddling their fingers. The reason this is on the slide and the reason that this relates to me today is that even from the unlikely places,

I'm still hearing these things.

So this is the setup of a girl who, a young woman who emailed me a few months ago about having pain and problems in her arms. I'm not sure exactly what her medical or health condition is. She's a programmer. She's an undergraduate computer science major, and so she actually rigged up this dance mat to this software in order to write code on her PC. So she programs with her feet as she moves around to actually enter text. So you never know, sort of once you've done text entry, you never know what your future's going to look like.

I've already done it with the introduction, but I did want to mention a similar thing, which is I feel very lucky and fortunate to have worked with Microsoft over the years. It's been, I think, some of my most fun collaborations with including people in this room, Andy Wilson was my manager or mentor in 2003, and then these other fine people here. I've also had the privilege of working with and publishing with them. And also teaching, on the teaching front, some people who don't appear on this slide, some quick highlights just from that time. Some work on modeling pointing errors, a dollar recognizer that's another one of these things that seems to keep popping up for stroke gestures.

User-defined gestures with Mary and Andy, access overlays, making surfaces more accessible to people with low vision and who are blind. So there's been some really fun outcomes. So thanks for that. And it's great to be here.

So touch screen typing is the topic for today and I'm going to cover probably a little too quickly, but still going to cover five projects, all of which are pretty recent. You can see three of these are at CHI 2012 in a couple weeks, and one is at graphics interface coming up. And they all hit on different

3 parts of entering text on touch screens, particularly with the focus on touch typing.

So getting to where we're not just sort of single point of input.

The real brains behind all this work are Leah Findlater, who is a professor at university of Maryland. Mayank Goel, who is in the room, and a first year CSE student at UW, and Shiri Azenkat, who is also a CSE student at UW. So they deserve all the credit and then some.

I want to start with a love story. I put love in quotes, because it's a story of commitment. I'm not sure it's a love story at all. It's actually a story of commitment. But, you know, we could start by observing how much has changed, right. This is a Daugherty type writer from 1893, and I like this image, because it's so exposing of its guts. You can see into this thing pretty clearly much more than the ones they used to put in these little, like, mahogany wood boxes and things, right. You can actually see the levers and so forth.

But, you know, obviously when we look inside today's devices, this is a disassembled iPad, things don't look anything alike. Yet, you know, maybe not that much has changed in some ways.

So this is from the 1800s, and here Steve Jobs is talking in front of a QWERTY key pad that's broadcast large behind him, projected large behind him. And it's in some ways quite amazing. The key arrangement is exactly the same.

Even over those beyond, what, 125 years.

Why do we type? I mean, you know, it's worth stepping back and answering ourselves this question. What's the deal? I mean, we want to type even when there's nothing to type on. I say nothing in quotes, because there's still a surface there and that's part of our exploration. But this Conesta idea never really came to fruition, but I think there was a working prototype at some level and we want to type wherever we are, it seems.

One of the answers is because it's fast. You know, if you can parallelize our hands, high speed camera studies definitely show fingers moving in parallel.

We can use, I say, nine to ten fingers, because often people don't use both thumbs. Chords are even faster.

And just to give you some sort of comparison, cursive handwriting which is as

4 fast as you can really long-hand handwrite is 25 to 35 words per minute.

Stylus keyboards are 20 to 25 words a minute. Touch typing, 50 to 90 words a minute and natural speaking 150 to 220 words a minute. And then court reporters can even go beyond natural speaking, so that they can catch someone speaking very quickly like I am right now and they can get even up to 300 words a minute if they're very good.

So if you look, touch typing in here, it's not fast enough to transcribe natural speaking and if you've ever tried to do that, you know you can't quite keep up. But it's still, so far, it's a multiplier of two or three beyond stylus keyboards, cursive, and so forth.

By the way, this image is very interesting. These were women typing up world

-- sorry, Social Security cards. So punching numbers, basically. And the only male I can find in the scene is the guy who probably is doing the least work.

The manager type who is strolling the aisle and hawking over people.

I think we also type because readers like it. You know, good luck with this, right? So that's from Newton. This is from Einstein. Part of it's what we're used to in the norms of the writing of the day and reading of the day. Perhaps these are even considered neat and nice. But the only thing I can really read is this, because that's printed. And the signature because I know who it is.

These are challenging for readers. Typing helps alleviate that.

Computers like it too, right. So these are a figure from McKenzie's text entry textbook. But these are different approaches to dealing with the problem of segmentation in handwritten text.

Another answer to why we type is because we, quote, always have. So this is the famous Sholes-Glidden first QWERTY keyboard. Type writer. There's no lower case option here. It's only capitals. You can see it's a pretty minimal key set, but this is where it all began, and we're still with the same layout.

Which, to dispel any myth, is to not prevent -- is not to slow people down.

It's actually to alternate hands to prevent mechanical jamming, which is sometimes --

>>: [indiscernible] only lower case?

>> Jacob Wobbrock: I don't actually know, but that's a good question. Maybe

5

-- I'm not sure exactly the time frame. Maybe the telegram era, those messages were all upper case so people kind of inherited the upper case experience.

Actually, so the shift key came in around 1922, which from a purely mechanical perspective, you can imagine was a little tricky to pull off, right.

We love this keyboard. In quotes, we kind of hate it but we love it. We put it everywhere, on places where there's none of this parallelization of hands and all those arguments about it's fast don't really even apply. We even put it here for accessibility aids when we're using a mouse on a screen, even though actually moving across this space due to the alternation of hands is not sensible and, of course, there's been a lot of work in optimizing that.

The QWERTY keyboard put other keyboards out of business. This is the World 1.

It's one of the first index typewriters, and it's, of course, gone. This is another index type writer where you move the lever and you get the letter you want this way.

>>: Is this purely because of sort of historical momentum that, again, QWERTY was big and then we continued, and now we can't break it?

>> Jacob Wobbrock: That's what I'd argue. I'm not a historian. That's not my sort of main contribution here is to unpack this history. But I think it's, as we'll see, it's just caught on and keeps going. And I'll show you actually that in images in a minute.

Here's an attempt at lower case. So there's actually two separate QWERTY key sets. No shift. So that's called the double keyboard. And then let's jump from 1897 and, boom, 1968, and alas we've all seen and you can't quite read the letters, but I can promise you they're the same layout. And the alto has the same layout and you can read those letters. And the star, the same layout.

And on we go. The first IBM PC, the Apple Lisa and before long, we're typing on glass.

This is a lot different than all those other pictures, right, in terms of the mechanics of the surface we're on. There's no plastic, really. It's pixels, so to speak, it's glass. And it deserves to be asked whether or not we need to type here in the same way. And if so, can we even do that. Can we do it well.

So there's a pun on this slide, but I want to make sure no one missed it. The

6 key challenges. Some of them, at least, are so there's obviously a lack of subtle tactile cues. There's no bumps, edges and ridges like there are on a physical keyboard. There's no travel distance. You know, where you touch, you stop, basically, except for the little bit of squishy flesh on the tip of your finger, but that's pretty small.

Small keys, fat fingers. Of course, fat finger problem is known. And there's also, I think, a state problem here in that sort of Buxton 1993 state model idea where with a physical keyboard, I can be off the keyboard, I can be on the physical keyboard, and I can be depressing a key. So there's this three states of action for the user. With a touch screen, you're either on it or you're off. So you lose an entire state, which turns out to be, I think, a big challenge.

This is a cartoon, a little fuzzy, but this is kind of sometimes how we feel, right? People chuck their high tech stuff on the floor and want to go back to what feels natural and right. So here are some of the ways people have solved this problem. They've just attached a physical QWERTY keyboard to an iPad so they don't have to type on that screen. This is a product called the ikeyboard. I've never tried it, but it's basically a key guard, which is inherited really from a system technology but also can help people sort of more accurately hit the keys. Here it is again, although I think it should be noted this person most certainly didn't write what's on the screen because that's The

Tale of Two Cities.

Touchfire is another product. Again, I haven't tried it. But it's basically this peel-off kind of silicone overlay so you get a little bit of travel distance when you strike a key. But obviously, the tradeoffs for touch screens are big, because now every time I want to have the screen display something, I have to take it off, put it back on, kind of fold it back and forth.

So there's a lot of attempts and wide acknowledgment this is an issue. So the idea for today is can we burden the bits? Can we invert the finger-keyboard relationship. So instead of having your fingers subtly adapt to the keyboard that's under it, can you actually make the keyboard adapt to your fingers and just do what feels right?

This is hard. And I'm not going to claim we solved it. But we've taken a sort of first stab at it and I think future work can begin to chip away further.

How well can we actually touch type on flat glass. There's a lot of related work that touches all this. Text entries in old area. Touch screens have been

7 around for a while. I won't have time to review it all. But single-point entry keyboards, there's a lot of work there. Adaptive keyboards, often not personalized to specific users and often actually not even evaluated. Just kind of algorithmically proposed. And stroke gesture keyboards are relevant as well.

Not much work really in looking at touch typing, like actually trying to touch type. It's not been approached. And on a really small device, which is where a lot of this work began, touch typing is compromised by just the lack of space.

So in looking at kind of the five projects. We're going to start with a study of finger strikes. How do people actually strike glass. So this is the first part here.

How do fingers strike glass when trying to touch type and what are the implications for the touch screen keyboards were the questions we were interested in.

So actually, was sitting next to Stu Card at HCIC last June, and it. Doesn't show up in the projection, too well. It's too bad you can't see my screen, but there are a bunch of smudges on here, and they're actually really indicative of the underlying touch screen keyboard. And I just looked over at him and he turned off his iPad, but he'd just been typing. I looked at the iPad and I saw this shimmer because the angle I was at was just right. And there's a full keyboard that you can see. I thought hey, Stu, can you take a picture of your iPad, I'd like to see the emergent kind of keyboard from your finger smudges, your oily fingers. So he did.

And it kind of reveals that striking glass does, of course, form these patterns. Of course, when he's doing that, though, he's looking at a keyboard.

Here's from online, I found a site that actually put up some smudge images.

This is from a mail program, and you can see the typing layout here a bit and some scrolling behavior from smudges on the side. This is a photos app, where there's a lot of swiping in the center and then Angry Birds, which I've never played. But if you've ever played Angry Birds, is there something about -- yes. I figured someone in the audience would know. I figured, okay, I'm going to do that, right. Maybe I'll have to play just to see what the motor patterns are.

8

So it's interesting, these kind of physical traces we leave, right, as we interact. So this is a surface that was generously donated by you guys, by

Microsoft, thank you. And Daniel Wiggdor helped get this into our lab. We did build a stand to get it a little higher because touch typing like this is a little tough so we wanted people to be able to sit and be a little higher.

And so as I said this primarily is a study, how do people strike glass. There are three conditions. Today is going to be a bit of an overview so I apologize that I'll go quickly. It's not going to be the level of study detail you'd find at a CHI talk. If you have questions, we can deal with those or talk about them later.

But there were two touch typists. That was including criteria. They were 85 words a minute on a physical keyboard. So they were actually good typists.

They were transcribing phrases and there were three conditions. There was one where there was no keyboard and no feedback at all. That came first. These were counterbalanced after which is no keyboard and then some asterisk feedback and a keyboard and asterisk feedback. And I'll explain the asterisk feedback in a second.

We don't have a recognizer here trying to figure out where they're striking, right. We just want to know where they strike. So you can't actually show letters unless you choose to show the right letter every time, which is also maybe problematic because it gives maybe a sense of confidence and accuracy they didn't have.

So to explain this condition a little better, again with no keyboard and no feedback, we present a phrase. These were areas just to register the thumb so they were kind of in a comfortable starting position. So they were prompted the input area and then the users could figure those thumb placements as they like.

Here's what this looks like. So again feedback. Just typing on glass. You can see that the hands actually start down and as they begin, they kind of hover them slightly. That was a sort of natural behavior we saw. But the palms and the arms are resting, which for a vision system, another thing you have to make sure you ignore, spurious touches and so forth.

So the next condition was no keyboard with asterisk feedback. So they're actually going to get some feedback in the form of asterisks that help them

9 stay aligned with the text. The one character you are seeing that we did input is the space. We had a big kind of space bar area that was reliably hit. But otherwise, we're just showing when the system saw a strike so that they can stay aligned. So that's what this looked like.

And, of course, there's no recognition of what those strikes are, just that they are. And that's why for the -- yes, we did have back spaces in the form of right to left swipes, which, of course, is not the same, not as ecologically valid as reaching for the back space key. But the reason we did that is we wanted to go with what we thought probably would be more likely of a reliable way to back space on a touch screen. And also not to confuse it with potentially striking, like, the P key, which is in the direction of back space.

People found that was comfortable enough.

And then the third condition was actually with a visual static keyboard.

Again, this asterisk business, let me briefly explain. If there's a strike, there's a question, right. Is this an I? Is it a mistake? Was it spurious input? By showing the asterisk, we kind of resolved that for the user, that at least we can believe that they can believe that they're entering what's next.

So it helps correlate those.

And then if they actually made a mistake, like here it didn't catch the space or they did not hit the space area, we asked them to stay aligned so they would actually back space through that word and begin again and we would disregard any miss like that. So the result is we can correlate input events with the letters.

So here is some of the other data collected. We can see that we can see all the fingers down there. That's, for example, would be synchronized with an A, that touch. This might be synchronized as well. We can see the hand down, so we have touch down and up events. And we also did some simple computer vision stuff to grab convex holes and things, though I won't be talking about that today.

So some of the results we found so here are all the key presses from the whole study, and everything that was a finger or thumb is blue and palms and spurious kind of stuff from the palms are orange. And you can clearly see some patterns here that emerge.

So one, we can see the separation of hands. That's fairly obvious. Another is

10 that there's a heavier use of the right thumb than the left, which is typical for right-handed participants, and all of them were. You can go back to your office and see if you do that now. Or maybe you can kind of feel that here.

That tends to be typical.

There is a nice arcing of this top portion on both sides, actually, and that's kind of interesting, because it suggests certain sort of configurations of the would-be keys. You can also see these kind of split trends, these sort of curves that come down through the middle here.

The typing speed was 58 words a minute, which is about 31% slower than their physical keyboard rate of 85 words a minute. And this is just with all those touches. So breaking down if we look in some of these what I think are kind of pretty pictures, we can see that with a visual keyboard, their key centers are aligned pretty well with the intended keys. Without the visible keyboard, they're overlaid for illustration now, you can see it's a lot cloudier.

They're more dispersed, as we might expect, right. The speeds, though, are about the same. 27 words a minute, 28 words a minute. These weren't significantly different.

So obviously, if you're going to type with no visual representation, you're going to have to account for some spread there. We can do one standard deviation contour ellipses and look again at the structure here that emerges.

The visible keyboard and non-visible keyboard here is on the right.

Some comparisons. We get more arch on the non-visible keyboard. We also get greater space between the hands. That was typical across participants, and larger key press spreads out near the extremes. So, you know, when you're not drawing a keyboard, you get these effects that are kind of magnified versions of what the visible keyboard allows for.

And I think we kind of just reinvented this or rediscovered it, which is actually kind of good. I mean, it would be a built of a surprise if it was wildly different. But the same patterns actually are very much present there.

>>: Can you go back one slide. So there's this big enter keys and things, are those A? Does that correspond to A over here? Does it actually correspond to the enter?

11

>> Jacob Wobbrock: No, right. These are only letters, sorry. They're not tab, caps on. And perhaps, you know, what we see here is that these letters, there are some -- I'm sorry. Let's see. This keyboard doesn't show them.

Some keyboards have bigger keys on the outside a little bit and they're beginning to go that way. So that an older split keyboard. I think the newest ones would show something more in that way.

If we actually do some very simple key press classification, just the distance to the centroid and some ten-fold cross validation. You can see with a user independence, across all the subjects, we get about 93% on the visible keyboard and only about 70% accuracy on the non-visible keyboard. That's obviously not even remotely good enough to type on. But that gives us something of a kind of user-independent baseline.

If we personalize to different users, so here's three of them, for example, we can see, look at this. You can see the split here between hands. It's kind of interesting. We actually come up quite a bit. So it's not -- I don't think this is the case where we would insist these things have to be user-independent. I think personalized would make sense because the systems can observe our typing and create models for us.

So that's just distance to centroid and we're going to see as we go through this talk that we do quite a bit better than that as we get more sophisticated with our classification.

So what classification accuracy is good enough and does a visible keyboard that obviously was better, no practiced -- these subjects just started. There was no real practice here. More sophisticated touch model, perhaps language model.

We're going to leave language models out today. They could be overlaid with any of these techniques so we'll explore these.

So we went on to look at actually personalizing keyboards in this way. Leah

Findlater was the main author on this. So we're here now. You know, and I guess in a way, we're trying to see if we can get away from this situation, right. This is, at least with our experience of the surface, the pass around the keyboard thing is a little bit clunky and there's no way to set it because the margins aren't that big. There's been some really nice work on keyboards on surfaces done by people in this room, the three of the four in here, along with Bjorn Hartman, and this is really interesting work but there's a much

12 bigger surface here and there's multiple keyboards and people have their own and so forth. So it a little bit of a different problem.

So better classification. How can we do here. So distance to centroid, I think, did show some promise, 90% or better. But it would be nice to do better. So we took more people, more data from different keyboards and more transcriptions. I'm moving quickly over the gathering of that, if that's okay.

Here what the features were that we kind of ended up exploring.

So with a given touch, a down/up, we have the usual features you might expect, the fingertip, the finger center, some travel between the two, the angle between the two, and the major and minor axis and so forth. So these are some possible features per touch event.

Trying to figure out which key does that touch event map to. That's the question, right. So through some base feature selection, we ended up with the starred items. And they're not real surprises. The tip and the X coordinate relative to F and J. We do everything relative to F and J, because they kind of provide the sort of origin, if you will, because we want the hands to be able to kind of split wherever they go.

So you can see some of the others there. Now there's a big question here about do you actually show the adapted visual representation or do you just adapt the underlying model but don't show it.

And I actually think this is an interesting question. It's certainly been raised in adaptive interfaces generally. Do you kind of show the model or not.

But when we're touch typing, we want something that's very motor driven. So how much you show is kind of how much you maybe lean people towards looking and that actually came up. So there's at least three challenges here.

Let me describe this. This is an adaptive keyboard but the layout remains static. It's going to remain fixed. Here's one where we're showing the underlying adaptation model. Of course, you can't really show everything, because there's some features that aren't just X and Y centroid. So it's an approximation, but it's closer to showing the model.

So you get this cycle of adaptation, right. So the visual representation changes. I think it kind of looks like the continental United States here.

Then the user changes their behavior to compensate and so the system adapts

13 again and you end up with this kind of cycle that's been observed many times.

And what if someone aims a key press, like deliberately aims, and that's what

Tim peck is unfortunately sick today, but he's worked on some of these problems and developed these key center anchors to avoid that problem.

Our approach here was just to disable the classifier after a back space or when typing is slower than a key stroke per second. We found that worked fine. But these are complementary approaches.

And another issue is excessive overlap. So, you know, keys can become very overlapped. If you look here the F key is kind of -- it's with an underline.

F is a bit hidden there, and other keys as well, J as well. So we actually maintain a minimum of 30 pixels between the key centers. It's just a change in the visual space. It's not change in the model. We kind of push out from the

F and J keys to do that. And that way, things will go a little bit cleaner.

So the process is start with a static layout that's seated and then we, as the user types, we observe the key presses. We use a decision tree classifier. I was trained on those features you saw. We used Weka to do this. Optionally adapt the layout of the keyboard. So the underlying model, but not visually as we just talked about, or showing the visual portrayal and then things get, as you can see, kind of crazy.

>>: How many key strokes is independent in what you have shown so far?

>> Jacob Wobbrock: That's right. That's a good point of future work is getting much more sophisticated on those kinds of relationships, yeah.

So here, you can see an image of the keyboard and in this case, the person's hands are spread to the point that the two halves have gone with them. You could type like this if you wanted. I don't think that would be very ergonomic, but it's able to do that. Unfortunately, I don't have a video of the keyword in action right now.

So we did do an evaluation. So we had touch typists again. Their physical keyboard speed was 79 words a minute. These aren't the same people as before.

And each key was initially seated with five points. Adaptation starts at ten strikes per key and the history per key was about 100 strikes. So we had a phrase transcription task for the training and learning and then also pangrams

14 to make sure that enough of the Q and Z and so forth were given enough data.

>>: Pangrams?

>> Jacob Wobbrock: It's a phrase in which every letter in the alphabet is contained at least once. Quick brown fox jumps over the lazy dogs.

>>: What's the other one?

>> Jacob Wobbrock: That's right, pangram, right? It's not race car. Race car

-- palindrome, that's the other thing. Yeah. They both start with P.

Different. And then we had three conditions. Keyboard conditions. I should say three levels of the keyboard factor to formalize that. So we had an adaptive keyboard that did not visually adapt that we talked about. We had an adaptive keyboard that portrayed that visually and then just a static, conventional keyboard that did nothing.

Three 1.5 hour sessions, so they access lightly longitudinal. They came in three times for a total of four and a half hours. And 4,500 phrases were collected. And what's nice is we actually have a significant effect of the keyboard type on words per minute. And what we see with that is that the adaptive keyboard that did not visually portray its adaptation was faster than the other two. And you can see even from the start. The visual one kind of starts at a deficit, gets a little bit better and then by the end, and we don't know if we could go further if this would continue to improve, the visually adapted one. But the nonvisual one was the winner. We'll see what the speeds were. Over 30 words a minute.

The uncorrected error rates were near zero so we didn't statistically analyze those. There's just tons of zeroes in that data.

Some preferences were expressed as well. These are from the NASA TOX scales.

We did a capita to actually see that they were only measuring about three things so we grouped them. These are the ratings from the participants. A lot of people preferred the non-visual adaptive one, the one that they were fastest with. But they actually indicated that the natural kind of comfort was owing to the adaptive visual one. Of course, this response could be because they thought it should be more natural, since it's visually adapting. We don't know for sure.

15

>>: Why are there so few responses on least frustration?

>> Jacob Wobbrock: This is one scale's worth of responses so there's 12 people. This is 24, because there's two scales, and here's 36. Sorry, I know

I'm blasting through some of the finer details so please interrupt me if you care.

Some comments. I felt I was constantly looking at my hands about the adaptive visual keyboard. 8 of 12 participants actually made similar remarks about this.

Not in session three, though, so they did seem to be getting used to it. The adaptive visual keyboard seemed easiest this time, as long as I didn't look and see the odd spacing. So it's like ironic, right. If you adapt it visually, they don't want to look. So why bother, right? Why do that? Because they're freaking out, like whoa, that doesn't look like. It's like turn your head, and it's okay. It's kind of an interesting trade-off.

Most subjects preferred the adaptive nonvisual keyboard but did not really know why, because this had no visual distinction from the static conventional keyboard. It appeared to them exactly the same as that. The quote here is interesting. With the iPad, you see the wrong letters and it corrects in front of you, using language model correction is what they're referring to. But with the adaptive nonvisual keyboard, I know I'm hitting in the wrong place and the letter comes up, which is nice. I'm just not used to it.

So I think there's some interesting thoughts here about whether or not to adapt visually and what the benefit really is there.

>>: There's something you're doing with the visual representation here that you haven't commented on, and that's that you're -- I mean, I look at these things and they look like little Gaussian blobs. So I'm wondering if you explore sort of the design space of that like to actually make the representation correspond to the distribution for each key and therefore sort of change people's expectations about whether or not they really struck the key.

>> Jacob Wobbrock: Right. So that's a good observation. So the distribution of hits that we think belong to the key are trying to be reflected by what we're drawing. And so but we didn't -- I wouldn't say we really explored the

16 sort of visual representation options that might be possible. And that's actually an interesting point for future work. I mean, we're trying to suggest there's not these sort of rigid boundaries like old school containment testing.

But yeah, there's definitely room to move on that.

We also saw another pattern was that people's adaptation with the visual portrayal was greater than without the visual change. So this is the non-visual one, but now rendered visually. So you can actually see kind of how much there was adaptation going on and you can see it's more subtle, whereas this is for the same participant but with the visual representation. So it's as if you don't actually show the visual change, they don't actually necessarily move with as much variance. If you begin to reflect the visual change, they move with more variance, perhaps feeling like they have to reach for keys as they get further outer and then they enter this cycle of I'm reaching, I'm reaching, I'm reaching, and it's driving things further out.

That's a bit of an open question.

So in summary, we learned a few things. We did improve the typing speed over the conventional static keyboard. Subjectively, it offered good performance, but participants couldn't really tell why. So that's kind of nice, because they don't know. We'll see that happen again with walk type a little bit later.

We seem to require more visual attention with the visual adaptive one. But perceived to be comfortable and natural to use. But maybe that's a novelty effect.

So I think we can do more with touch screen keyboards than just touch type.

They afford certain other things. This is not gestures for cussing, despite the title here. Yes, Shiri.

>>: I'm not really familiar with Microsoft service, but is there a keyboard?

>> Jacob Wobbrock: Yes.

>>: In the standard --

>> Jacob Wobbrock: Just a static, yes.

>>: And it's not like a windows-owned type keyboard or tablet?

17

>> Jacob Wobbrock: Not to my knowledge. There might be such a thing now, but back in the day, does anyone know?

>>: [inaudible].

>> Jacob Wobbrock: So I think it's just transplanted from the desktop.

>>: Reoriented, and that's about it.

>> Jacob Wobbrock: So maybe there's some room to improve. So I would like to talk a little bit about using touch screens for what, in part, they're good for, which is, in fact, making gestures and strokes and actually adding that as a complement to some of the modes in which we're used to. So I know this is not from a surface. This is from an iPhone, but how do we find equals? So we go here and then we go here and we find it, hopefully, if we know even to look there. But the point is we're using a lot of mode switching and flipping the screen. So on a small screen, that makes some sense because you have limited real estate.

But actually think that we can draw, right? I mean, we're on a surface. Let's draw things. So here was our idea. Four or more fingers go down on the offhand. Then on the dominant hand, in this case the middle finger, but it could be any finger, we simply draw what we want. I actually am not sure we even need this, but we didn't get kind of far enough to explore just leave the mode out and just draw. But certainly, that helps disambiguate.

Your hands stay in typing position. There's some nice kind of gestural affordances of this. It's also kind of consistent with Guiard's kinematic chain, frame the area, act in detail inside it. And so you can see some of these here.

You'll notice there multistroke and, in fact, even multitouch gestures as well.

So some of those benefits, you know, it's an active mode. That's another nice one so you know you're doing it, like shift. Low visual attention. You don't have to really look. You can feel. And, in fact, you can keep your eyes on the text you're entering. And participants' comments reflected that.

We used, to design the gesture set for punctuation, we used the gesture guessability approach that I'd used with EdgeWrite and Mary and Andy and I used

18 for the surface gestures study and work we did in 2009. In this case, you can see we had 20 participants and a bunch of different punctuation and symbols they were working with and also some commands, space, shift, back space and enter.

And we allowed them to do single-touch, multi-touch, uni-stroke, multistroke.

That was all open. But they were stroke gestures, not sort of like palm of hand and things.

Here's the agreement scores, which some of you are familiar with what this means. But it shows a sense of consensus around the gestures. We had one trial where we just showed the name of the symbol. So asterisk writ. Out.

Because if we showed an asterisk, that might bias them toward making that kind of asterisk, that typographical mark.

On the second trial, we showed the name and a symbol in many different fonts, you know, kind of like ten fonts, asterisk so we try not to bias them toward a particular representation. You can see there's some good agreement in the beginning here. Question, dash, pound, colon, a few others. But then we plunge into the commands being way down here shift being the least. Here are some examples of shift. You can see that people are coming up with different things. This probably is a four-finger upward swipe, two fingers one side or the other. And so people come up with different things.

Then comma and period were low enough and common enough we decided to leave them on the keyboard. This is the final gesture set and actual gestures from real participants. You can see some variation we'd expect to see with the guessability maximization. So the dollar sign with one vertical versus the dollar sign with two verticals and so on.

We did decide, based on what we saw, that the preponderance of multitouch gestures, not just multistroke, which many of these are, but multitouch for these gestures was high enough to recommend that they be included as multitouch gestures. So equal sign, colon, two fingers. The pound sign with two fingers slashing both sides and then quotes and things. But most are them are still single touch, single point of entry.

We also added some marking menus. If you're on a key, a letter, and you want to say shift, that key, were an I or something like that.

19

>>: You didn't use multiple gestures for each one, equals, this one or this one?

>> Jacob Wobbrock: Yes. That's right. So their alias just according to -- that was just -- yeah. So that's part of what that guessability procedure gives you is kind of anything people come up with. As long as it's not colliding with something else, it can be included.

So this marking menu idea was just another addition where you can hold done on a key and then stroke, and that gives you, for example, control-A or shift or whatever.

So we evaluated this, but we needed phrases that included punctuation, obviously. So we did word pairs and inserted the punctuation between them. If it was the kind of punctuation that needs a partner, it would be partnered at the end.

Six participants did this. We had two keyboards, gesture versus a mode key, like you'd expect. This does not have to live instead of the mode key. It can live with a mode key in perfect harmony. So it's really a complement. It doesn't need to be sort of better. It just needs to be good or useful or whatever.

So, you know, the goal here was to not really replace mode key, necessarily.

But we found no statistical difference on either errors or speed. In fact, they were almost identical. So that, in a sense, means it's not any worse. I do think we had enough statistical power here so we would have seen a difference.

Some comments, this is great, it feels very seamless. My opinion --

>>: I have questions about the mode key. People share actually how

[indiscernible].

>> Jacob Wobbrock: I actually don't have a close-up of the mode key. But there was a symbol key where the shift key is on a regular keyboard, both on the left and on the right. When they hit that, it would switch the key labels.

>>: They'd get a different keyboard? A menu thing?

20

>> Jacob Wobbrock: Yes, sorry. I know I'm going fast. This was not evaluated. This is just another cool thing you can do with gestures and touch keyboards. Sorry for the confusion. The study was just of the mode key versus

-- so a shift key, basically, and the keyboard keys didn't move, but the labels on them flipped to the --

>>: Another question that I have, then, the way the mode key condition worked is so I know that like on the iPhone, for example, you switch the keyboard because it's so small. But in theory [indiscernible] a wicked big keyboard that was one keyboard that had them all, maybe it was harder to --

>> Jacob Wobbrock: Right, around the top, around the sides.

>>: Is this something you --

>> Jacob Wobbrock: We thought about it, but we did not decide to make that evaluation yet. Mostly because we thought there are so many symbols and things that if you do that, you just end up with this kind of ugly -- we were skeptical people would really use such a thing. It's kind of an aesthetic more than anything, not a real principled rejection of that idea. Certainly but now you can almost model that one.

>>: [indiscernible] every single symbol, but the symbols that are present on a real physical [indiscernible] analog to the real physical keyboard. Some of them still require moves to get to it. Some only require one press.

>> Jacob Wobbrock: So I guess we're talking about the sort of six or so that don't require shift that have their own key?

>>: You can imagine ones that are above the number.

>> Jacob Wobbrock: It's a fair point. Basically, the question we have is really driven by do you switch the set or not. And you're saying with enough real estate, you don't really have to switch the symbol set. And I think that's a fair point.

>>: Do you have any thoughts on the timing required to actual switching to your gesture mode? Was that sequential? Was that kind of done in parallel?

21

>> Jacob Wobbrock: I'm not quite --

>>: Putting four fingers down. So did people like kind of put four fingers down, then draw, or was it more like --

>> Jacob Wobbrock: I see, the two-handed. So no, actually, they not surprisingly performed just like the kinematic chain theory would suggest, which is that the one leads the other. We didn't do -- we have the data, I guess. We didn't look at the exact timing between the two hands hitting. One led the other.

>>: Do you feel like that you potentially don't need the model? Like if you can quantify how much time they wasted on --

>> Jacob Wobbrock: Right.

>>: What would you foresee or the slide being -- just curious.

>> Jacob Wobbrock: So I have two comments on that. One is that the hands were already in place so dropping the four fingers was just a matter of vertical descent. It's not really, you know, it's a very small, unaimed task. It's really pretty incremental.

>>: [inaudible].

>> Jacob Wobbrock: Sure. The here thing is that I guess in a sense, it's a little bit misleading to present -- not misleading, but it's a little bit off the point to present speed things here, because people find -- for example if they found stroke gestures much more satisfying and comfortable or something like that, then I think that is easily warranted add be such a thing, even if the speed was, say, a little slower. So yeah, it's always nice to be faster.

I think that's possible. Also, they have to hover their offhand if we didn't have the mode. They'd have to put their hand somewhere. So I'm not sure that resting it is any sort of worse a choice than hovering it or whatever.

Let's see. My opinion of gesture things that is there's a slightly higher learning curve but they're usually faster once the fingers know how to do them.

That's the learnability question.

For the mode-key keyboard, I always had to look down at the symbols to find

22 where they were. With the gesture keyboard, I felt I didn't have to look down really at all. If I did look down, it was just to figure out if I was on the home row. So to Mary's point, if you actually had all the symbols on the main keyboard, there would still be visual search time and learning of position.

But you'd learn it over time, just like you have to learn gestures over time.

>>: The people who did this experiment with comparison, are they the same people who created the user defined --

>> Jacob Wobbrock: No, no, sorry.

>>: And you taught them the available set or did they just guess?

>> Jacob Wobbrock: I didn't say this because I'm moving quickly, which I've learned a lesson, which is if you say it, you'll actually go faster because then you will prevent questions.

They only had three to five minutes of play time, basically, for practice. So it was pretty unconcerted, and I would say, actually, having seen all this, that it was very guessable set. There was almost never something they tried that didn't work. So I guess in a sense, it validated the guessability procedure again.

Okay. So let's talk a little bit about mobile and situational impairments that still involve trying to type on touch questions. Of course, now, we're not talking about ten-finger. We're talk about two fingers, two thumbs.

It's sort of unfortunate that our devices have very little awareness of the context in which they're being used. We move through the environment and as an accessibility researcher, as part of what I do, I'm actually really interested in how situational impairments actually cause accessibility needs for all of us.

So we're actually impaired by the situation, by the context, by the environment, not just sort of because we have a health condition of some kind.

So, for example, when you're walking, probably beyond this, but you at least have divided attention at some level. Some body motion, which also could be part of vibration or maybe you're moving on something vibrating. Maybe you're in an elevator or something. Uneven terrain like stairs, ambient light levels, ambient noise levels. All these things kind of fight for your attention and

23 resources.

There's been a few studies of this stuff, but really very little innovation.

So it's not that uncommon to hear people talk about it, but they it's like well, what do you did about it? And there's almost nothing yet. So I think it's pretty wide open.

>>: Are you familiar with the email and walk application?

>> Jacob Wobbrock: Email and walk, that's not one I've come across, actually.

Maybe we can talk and you can send that to me. Email and walk. So is it accommodating for some of these.

>>: Just make sure you don't run into pillars.

>> Jacob Wobbrock: Exactly. I actually -- well --

>>: [inaudible].

>> Jacob Wobbrock: So you can see where you're walking, right. I have heard of that technique, okay. So that's actually maybe one of these rare points of innovation as well. So quickly, to what WalkType is, go straight to the heart of this. This is [indiscernible] wells' work. So how it works, it uses touch features from the -- touch screen features, accelerometer features, inference about gait, which I'll talk about in a moment, and key-center anchors, which I mentioned earlier coming from Tim Peck and others here to actually cut the tex entry rate in half while you're walking.

>>: What are key center anchors?

>> Jacob Wobbrock: So the idea here is that if you actually do dynamic resizing of keys, whether, you know, well, mainly thinking about when you do that such that the motor space of each key is changed but not the visual portrayal, you can get keys that are almost impossible to hit because they're so crowded out by the keys around them that are more likely. And so key center anchoring is to preserve the super point of each key which is some little space so if a user deliberately aims a strike on to that key, that's get that letter.

It prevents keys from being blocked out by their neighbors.

24

And the insight here is from mow the device is moving, we can infer how the user is moving. And how the user is moving systematically causes errors. And systematically is why we can do this, because it's actually somewhat kind of lawful.

>>: I don't know if this is relevant. Are you familiar with some of the work, like on image deblurring? Because it seems really similar.

>> Jacob Wobbrock: It is.

>>: In spirit.

>> Jacob Wobbrock: It is and Mayank will tell you all about that after we're done.

>>: You some.

>> Jacob Wobbrock: If you want. So it's kind of image stabilization-esque, but for sort of input, right, as opposed to output. You're here, do you want to say something about that?

Okay. So you're not meant to be an orb this whole slide. I know it's a lot.

It actually a set of different models that were tried. The classification models from the left across the top to the right and the right is the key center anchor features. But these indicate the features that are being used in the different models that we explored.

And I don't have time to go -- if you want to see this in detail, come to CHI, because in a week I'll be talking about this whole thing very much in depth.

But there's touch features that are green, there's these accelerometer features that are blue. I want to focus on one here that is the phase of gait, the gait phase, the walking phase.

So here's what we saw. It turns out that when your left foot moves forward, the device slightly rotates counter clockwise as this arrow indicates and then when you're intending to say, for example, hit S, you've got a little rotation and you land on D. This is a systematic pattern that emerged. Similarly, with the right foot forward.

And so because this is somewhat systematic, we can actually do something about it. So this is X axis acceleration. And if you actually take the mean of this

25 signal and then you can tell that sort of below the mean actually does correspond to left foot behavior and above the mean corresponds to the right foot action.

And if you fit the dominant frequency here, we take a feature of the amplitude, that's one of the things we care about, and we also take a feature from the

X-axis to the touch point. These dots are the touch point, the moment of touch down and those time elements indicate kind of where we are in the dominant frequency.

So what happens for classification, then, is if the key center is struck, the anchor trumps all. So you get the key, guaranteed. That's the idea from Tim

Peck and his co-authors. But if not, then we have these different models. And

I'll show you their performance in a moment.

We have the Euclidian model we all know, distance to the center. Acceleration and displacement model and then this walking pattern model, which includes the gait features which I just briefly mentioned. And so the majority wins there.

So they each make their vote. And if there's two out of three or more, that's the classification of the key. If they all disagree, this one wins because it was the highest individual accuracy. And we used a decision tree for this as well.

So if you look here, just briefly, it does start at 75 or 70. But you can see that the walking pattern model is individually the best. Touch screen features only without the accelerometer does okay, but not as good. And then when we put the last scheme I just showed you, we call that WalkType and that does the best.

And so an evaluation of this with 16 participants, two keyboards, WalkType was on or off, but visually there's no change so the participants don't know which they're using. Two postures. We actually did sitting and standing. 30 phrases and 57,000 key presses. The words per minute while walking is improved. I actually think this is due to the prevention of errors. So we're not spending time correcting errors and you can see here that actually sitting the mean is higher. But that wasn't statistically significant.

>>: Why is just sitting the WalkType helps?

>> Jacob Wobbrock: That's a good question. Actually, let's talk about it

26 after I show the next slide. Here's the bigger result is that the error rate while walking was halved and so there's a win there. But again, the error rate is lower while sitting too. And that goes back to the question, why is that improved. And you can see here, this is the same data without WalkType, you know, those red dots are off the key they were intending and here they're kind of cleaned up. You still have some, but not as many.

So we can leave this slide for a second. So why does it help when sitting? So it seems that when you're sitting, there's still some of the same kind of body motion that happens a little bit in a subtler way as when you're walking. So as I move, you know, without the step, the sort of major thing, I still have a little bit of, you know, body perhaps. There's some still noise in there that gets cleaned up a little bit.

Did you want to add anything to that?

>>: [inaudible].

>> Jacob Wobbrock: So the stepping exchange rates that, but the hands are still part of that story. So obviously, very good question. Thanks for asking that. On the user preference we saw that 14 of 16 users preferred WalkType and, again, they didn't know which keyboard was which. They just used both.

They looked the same and he said, oh, I liked this better. That was a better experience.

The last thing I want to touch on is typing for blind people. This is called the Perkinput system that Shiri Azenkat, who's here, has worked on.

So the question, a couple questions. Can we lower the visual demands of text entry on touch screens to zero, like really zero. And if so, might we have a touch screen text entry method that's useful for blind people. So this is a

Perkins Brailler, and a text entry and a sense entry. Text creation device that, if you know Braille, you can use. And so you place your fingers here.

So here's the Braille letter for P, so Braille is six dots, if you don't know

Braille. It's six dots in two columns of three each and -- sorry, right, two columns in three rows. And the keys, the Braille dot is like a number in though way and then map those to the Perkins Brailler.

So we would have our hands, three fingers on each, and that's a space indicator in the center. And so with two devices, and I'll show you actually we have

27 multiple versions. If you use two devices, they would map very straightforwardly, as you can see there. And, in fact, we have three different versions. So we have one device, one-handed version. This is the same hand acting once for one column and then a second strike for two columns. So two hits per letter.

And in this case, the letter R is shown here. The Braille R and you can see that those fingers match the same.

And then two-handed on two phones, you just saw that. Two-handed on a tablet, you would see that. So these are straightforward mappings so the Perkins

Brailler, whereas the top one, you hit twice. But it's the same mapping within each hand.

>>: A user begins typing by registering reference points.

>> Jacob Wobbrock: The pinky is being used to enter the space, since that's not a standard Braille finger on the Brailler.

>>: A user can also enter text on the phone twice as fast using two phones.

>> Jacob Wobbrock: So this is a process of providing registration points where the fingers will be.

>>: Perkinput can also be used to enter text on a tablet.

>> Jacob Wobbrock: Her fastest phrase on a tablet is 57 words a minute, which rivals touch typing for many people on a physical keyboard. It's still a long way from a stenographer's situation, but it's, yeah, Braille-based input.

So hit testing, I put in quotes, because it's the not the kind of conventional containment style. You saw the registration points. What we do is based on maximum likelihood. Assuming Gaussian distributions around these reference points so if you have one strike, it's pretty straightforward to type the key whose probability is most likely. But if you have two, you want to maximize that over the two. So if you look at this figure here, you know, the combination of these two points in their respective distributions is what gives the overall highest probability.

28

So quick evaluation here, we did -- the evaluation wasn't quick, but I'll quickly go over it. It was actually fairly long. Shiri's like no, that cost me three weeks of my life. It was a longitudinal study to start with. So we had kind of three phases. And the first was the one-handed version, because most people are still just going to have one device, against Apple's voice over, which is something of the kind of main, state of the art competitor, I suppose. We had eight subjects, but they came eight sessions. So we had a fair bit of data from those.

And then in a second phase, we took one of those people who was willing and -- was she the one in the video? I think she was. And she went on to do six more sessions to kind of gain more expert proficiency. And then in the third phase, she also went on, the same person went on to do two more sessions on two devices and on the tablet.

The words per minute voice over was frankly not very competitive so that was sort of what we expected. But that was borne out. And you can see that over the six sessions or so, or the eight sessions, rather, the participants on average were almost eight words a minute. Not super fast, but this is the one device version. Things will speed up considerably as we add devices. Or space.

Uncorrected errors was also much less than voice over so we don't have a speed accuracy to trade off, which is nice. This expert, in her extended use, went on, the axis here is not to ten words a minute, it's to 25. Went on to be, I think her peak satisfaction session was about seventeen and a half words a minute, which is not too bad for a single device, for a single touch screen.

>>: You have to keep in mind that she's getting feedback every time she touches the screen. So that will slow things down.

>> Jacob Wobbrock: That's a good point. So there's this delay to hear the feedback. So it's not just a motor kind of performance outcome. And then in the two-handed devices, we have these results so this is the last bit where she did on two devices and then on the tablet. You can see with two hands on the tablet here, words we are minute in the, I think it was 38 words a minute and her fastest phrase was 57 words per minute.

>>: Do you think it might help for a follow-on to give audio feedback once you

29 have the word completed?

>>: Definitely. But as a starting point --

>> Jacob Wobbrock: Yeah, that's on the next version stuff, you know.

>>: Why is there a discrepancy between the two devices?

>> Jacob Wobbrock: Yeah, exactly. Actually, before we answer that, let me come over here, because the uncorrected error rate and corrected errors, you can seat errors made during entry, both of those cases, the two devices a fair bit higher than the tablet.

Actually, I'll let Shiri answer that, because I'm not sure.

>> Shiri Azenkat: We only did one session, we had one practice session and one real session with the two-device setup and the tablet setup. So the two device setup we did second. She was tired. I did see --

>> Jacob Wobbrock: I wasn't aware that there was a fatigue thing going on. I wasn't sure if you notice when she used the one device, she actually used her offhand to stabilize it a little bit.

>> Shiri Azenkat: That's true too, yeah.

>> Jacob Wobbrock: And with two, you can't. So there might have been a little shimmy.

>> Shiri Azenkat: Yeah, there was an issue with the device.

>> Jacob Wobbrock: That's what I thought would explain that.

>> Shiri Azenkat: But I do think fatigue played a role. We only did one session.

>> Jacob Wobbrock: That's good to know. So Braille-based input can be quite fast, 38 words a minute. Two hands on a tablet were faster and more accurate than two hands on two separate devices. But I guess that deserves an asterisk now in light of the potential con found.

30

A Perkins Brailler can be simulated with just one hand, but speed suffered.

So going forward, just do wrap up here. So we talked about these five projects, and they're all concerned with the same issue of touch screen typing, but what have we actually learned --

>> Shiri Azenkat: Can I just add one?

>> Jacob Wobbrock: Yes, you may. That's why I brought you today.

>> Shiri Azenkat: Just one more tiny thing that was cool and wasn't demonstrated, let's say she was typing with two hands on the tablet, she could easily switch to one hand by just registering only with four reference points on the one hand. So it's kind of cool to like type with two hands and then read, if you want to read Braille, with your other hand and transcribe it, you can do it.

>>: Be useful to actually hold the tablet and type.

>> Jacob Wobbrock: Right, right. Yeah.

>>: Any ideas about doing a [indiscernible] interface? Seems like uniquely suited to your three fingers space.

>> Jacob Wobbrock: So actually, there's --

>>: I hate to mention other interfaces that way. But all jokes aside, I think it's actually quite --

>> Jacob Wobbrock: No, so actually there's a project, so you may not be able to tell quite, but he's holding the screen towards us with his hands on the right so we know what this is like. So there's a project at Georgia tech called -- what do they call it, Braille touch? And that's actually the orientation they picked and they're coming from both sides and doing it.

Their paper hasn't -- an evaluation hasn't been published. We don't actually know. Maybe you do by now. I don't know how fast that worked out to be.

>>: Seems like with your dual hand input could be very easily mapped to this.

But you have to account for some finger being there all the time to actually

31 hold it, probably pinkies and then pinkies -- other fingers could be on it.

>> Jacob Wobbrock: That's very much the posture there.

>> Shiri Azenkat: So I think the key insight here is that we're not using fixed hit testing, so I think there's definitely some exploration we can do.

The other projects that have done something similar, they just lift of the screen up [inaudible]. So yeah, I agree that there's more we can try out with this. But I think that the key thing that we've done so far is just developed this technique of using [indiscernible] which fingers are touching the screen

[inaudible].

>> Jacob Wobbrock: Thanks, Shiri, and thanks for your question. So what have we learned? I think we've learned a few things. I think we've learned how fingers touch flat glass when touch typing. At least something about that.

The distributions, the kind of shapes and patterns that come from there.

We've learned how adaptive keyboards might use features beyond just touch-down location to classify strikes as key presses. So touch-down is just one of kind of feature, but there's these others we can use and we put them to use.

How adaptation can be visualized or not and some of the issues with visualizing that underlying model.

How text entry speeds can be improved through adaptation. So we saw with the adapter keyboards significant improvement in text entry speed. Which I'll highlight that. That's the first time that an adaptive keyboard study for touch typing, and I'm not sure, it might, probably not in any possible keyboard study, but I know for the few touch typing attempts out there, have never shown an empirical improvement. So this is the first time that we've seen any actual benefit of adaptation in terms of actual text entry performance so that's nice.

How gestures can be used for punctuation and how marking menus can be used for commands. How accelerometers can be used to halve the error rate of text entry while walking and also improve the speed, I should say. That's another benefit. How a Perkins Brailler can be simulated using one or two devices with one or two hands and that, actually, people are on the whole relatively comfortable typing on flat glass. When we started this work, we weren't even sure that was true.

So it's not all that horrible to do this, especially with just a little bit of

32 getting used to it. Yeah?

>>: Have you or others looked at all at how adding things like audio feedback to the flat glass technique screens would also help?

>> Jacob Wobbrock: We have not done that and I think there's just so many dimensions of this. That's another point of good future work, which is basically --

>>: Like click, like because you're missing --

>> Jacob Wobbrock: Ticks, even slight little vibrations from motors or things.

It's harder on a big surface than a device, but that's probably, could you make the screen vibrate just a little bit. I think that's all open for exploration.

That's an option on the small. But I actually think in the touch typing sort of ten-finger space that's an interesting question. If you can make the screen feel like it gives just a little. I know there's the [indiscernible] electric approaches to that, but even with just a little vibration or something, that might change things a lot, I don't know.

>>: Most of your approaches reduce the fingertip to a single point. And then you sort of aggregate that over time, right? Basically look at the distribution of those.

>> Jacob Wobbrock: Right.

>>: Have you looked in any of our features or any of our [indiscernible] not considering finger as a point but as an area touch, and correlate that, I mean, there's a lot of work from, you know, [indiscernible] understanding touch.

Where exactly that point is, right.

>> Jacob Wobbrock: Mary is actually lecturing in my class on that tomorrow.

>>: It seems like everything you're doing right now is kind of a step above that. I'm just wondering if introducing those points, those features --

>> Jacob Wobbrock: Right. So I'll remind -- let's see. It's worth maybe popping back. Sorry. I don't know how easy this will be to do. Maybe if I go this way. I'd like to show you the features that were selected, because you'll notice that a number, a few things, at least, that begin to get towards area,

33 not area itself as a single scaler, but major and minor access, for example, they speak to that.

They weren't actually that useful, the least in this particular approach. I'm not saying they couldn't in other ways. There's a little bit in terms of the actual movement of things, although you're right that it's being measured from different coordinates.

One thing about the touch, perceived touch and so forth an is that, you know, my understanding is that that work goes to what a human thinks they're doing when they touch, versus what a system is able to say you're touching here according to the way it's sensed, right.

In this case, it's a little different, right, because the points are consistent, even if I don't feel like those are the points I'm hitting. And so if I'm able to just type, especially with something adapting, I don't actually know quite where the keys are. But the point, as long as they're consistently represented, it should still, I think, produce an outcome that I want.

So I don't know how much we'd get from that or not, but this is becoming a theme. This is also sort of a worthwhile next step to look at that more closely. This is definitely sort of machine learning lite, right. I mean,

WalkType was more sophisticated but the rest of this is kind of a first cut with sort of simple approaches, like Andy's comment about things aren't really being correlated across multiple strikes and so forth. That's all very true.

I think with more expertise and more digging, things could get better. But part of the take-home is this is actually not that bad. It's actually giving improvements. We're getting words per minute. We're getting better accuracies in terms of WalkType and stuff. If we're doing that now, if we actually beef that up, it's only going to get better, right?

>>: I guess to the point of my question, I wasn't so much looking at adding more features in your feature table, but the more I'm looking at basically area-based approaches, you basically have these heat maps of coverage where you are and right now, you're considering hit this think on point against those heat maps and moving the point. If you're looking at a Gaussian from the heat map. But in general, you also have the information about the touch itself.

Like the area of touch and how it was there and the point of impact and also some other stuff there that you might confuse against your hit testing up

34 against these areas, maybe that will give you. Potentially much more informative than the single point.

>> Jacob Wobbrock: I see what you're saying. Sorry I kind of missed your point.

>>: I think this was part of the question.

>> Jacob Wobbrock: Okay. No, we haven't looked at that. I'd love to talk to you more about that, actually. That is a nice step. That's a nice suggestion.

We're kind of towards the end here. So let me drive us down toward the end here. So for future work, including some of what you guys have mentioned, I think we touched on a few things here. There's a tension between inventing entirely new methods for sort of text entry on touch screens in terms of can we make the QWERTY experience really good. I mean, part of my initial part of all that inertia behind QWERTY is it's fun to try brand new things and I think it will be new in the sense of perfecting QWERTY will require new things. But I think it's still kind of the same old QWERTY. It's like how much do we yield to that versus try something totally new. Bimanual text entry tech next haven't really been fully explored. Especially if we go on the new way, touch typing is obviously bimanual, but are there ways we can use two hands in interesting fashion that can make larger touch surfaces useful. This hasn't been explored as much, because on small devices, it's not as relevant.

Is the stylus making a comeback? So the stamps in galaxy node, wow, that looks like a palm pilot, except it's a nicer screen, you know. What are the implications of that, you know. I'm not sure. But I think it's another thing to think about.

Better understanding of the cycle of adaptation is a general problem, but I think it does raise its head here quite a bit. Language models effects on typing behavior. So we avoided the language models. We actually have one working, and it actually does pretty well. I kept that completely out, because we wanted to isolate the sort of touch base approach, but we can add a language model on top. But what are the effects on people's typing behaviors. Like a fair question.

Better features and classifiers and different approaches like bank was saying is certainly part of future work. Can we touch type on a phone-sized screen is

35 also a little bit of what you were suggesting, but even in a QWERTY fashion, is there a way to do QWERTY on an iPhone? You know, with maybe hands in kind of interesting ways, but you could kind of cram them in there, and could you get good enough? I'm not sure.

Again, that's kind of owing to the inertia of QWERTY.

>>: Touch typing?

>> Jacob Wobbrock: Right which WalkType was about. And then additional sensors, which for some of you in the room is like your hot spot of expertise.

But what kind and for what? I'm not clear exactly on what else we could get and I'm sure there are answers to that. But that would be actually interesting to think about.

So another way to put the challenge less texturally is just this way. So, you know, all of this, right, it's trying to solve some problem. And the question in a succinct fashion is can we just do this somehow without feeling the need for any of those things.

Thank you.

>> Andy Wilson: Thank you very much. Mary is so excited, she didn't even want to clap. I take that as better than applause.

>>: [inaudible].

>> Jacob Wobbrock: People may want to leave. If you want to ask real quick, then feel free to leave.

>>: You were talking about additional sensors. I have a question, phone. I don't know if it would be -- like the front-facing cameras, like the web is looking or not, because that seems like it would add to the confidence

[indiscernible].

>> Jacob Wobbrock: So actually, I have a project sketched up in an NSF grant using the camera looking at the person to figure out different situational impairment stuff, right. And I haven't thought about it in terms of improving the text entry accuracy, but that could be really interesting sort of feature.

>>: It's almost like typing, that --

36

>> Jacob Wobbrock: So my general thought was could you kind of implicitly mode switch things in a way that's uncomfortable or natural by way of the way the person's looking or not. And even if you look back, take Patrick's

[indiscernible] technique so I can see the last thing I did. My eyes are drawn to it if you're familiar with the green glow paper.

So I hadn't thought in terms of texting. It was interesting, how would or could the sort of model or the keyboard or whatever change in a way that's useful. Certainly, you could take up the rest of the screen. I don't know if you'd want to move your hands. You don't look up and move your hands around.

But if there's a way you could reuse parts of the screen, because if you're not looking, you don't need to show anything.

The project I did with Sean King a few years ago I was mentioning before the talk called slide rule was for blind people to access touch screens and screen read with their finger and do what Apple calls a split tap now. We call it a second finger tap.

But that was an exploration we had, and we just drew nothing for the screen because they're blind. And for sighted people to have the same experience. So

I think when you're not looking, there's maybe ways to reappropriate the screen. But text entry is sort of heavily motor dependent, fluid-tasked that it's not obvious to me how we might do that. It's an interesting question for exploration.

>>: So want to talk about that a little bit, because I recently had contact with the steno community, and they think we're all idiots, because they're like we solved this problem. The answer is steno, you know. You can.

>> Jacob Wobbrock: Court reporter, stenographer. Sten graph.

>>: So as you pointed out, they go at conversational rates easily. They say, you know, it is something that takes some degree of training.

>> Jacob Wobbrock: They go to school for it.

>>: You get licensed. There are tests that, you know, you have to be at, I think it's 250 words a minute to be a licensed stenographer.

37

>> Jacob Wobbrock: So the range I had.

>>: So they think we're all nuts.

>>: So why aren't we using steno right now?

>>: I don't know. Well, one reason that they give is they said that the machines are too expensive. So that they run 6, 7 thousand dollars apiece.

>>: Doesn't seem like sort of the first type writer.

>>: So there's an open source movement doing this program called clover, which it's open source steno, so you can do steno on standard keyboards. And this is how I found out about it, because our keyboards is one of the few keyboards that lets you cord large numbers of keys simultaneously.

And, you know, it's kind of funny in talking to them, because some of the applications they push out. So there's a lot of people who have had throat cancer who can no longer speak. To be able to converse at kind of normal conversational speeds, they teach them steno.

>> Jacob Wobbrock: So basically for output purposes, right?

>>: Exactly. And being able to, you know, buy a $30 keyboard and be able to do steno is incredibly enabling.

>>: The improvement is worth the pain, whereas before I've never seen it to be a worthwhile trade-off, as sort of alternative layouts.

>>: But given how much we all now spend our time communicating via text, do you think it would be worthwhile? I mean, should we be teaching our kids steno?

>> Jacob Wobbrock: So I think it's a good set of observations, because -- and

I've done this too. It's often dismissed out of hand because, oh, it's too hard to learn, whatever. It's an expert thing and we can't all expect to be experts, right.

But I think that is a bit of a punt, because if we did teach it in school, right, you could imagine -- we teach our kids a lot of things, right. If it

38 were worth it. I guess I have some skepticism around it's mapping to an iPhone sized device, you know. I have some skepticism around some of the other kind of translations from its original, you know, use, particularly in the court situation or other cases, not just for that. But so I think in the case of the throat cancer person, right, that's obviously a case where you have a highly motivated person for whom it's really worth it, and I just don't know that we're there yet.

I can't think that fast probably to write any faster, much faster than I would use a QWERTY. So it seems for communicative reasons it might be better. But not necessarily for me in my office doing steno into my computer.

>>: One of the main problems is that there's such a high barrier to achieve some useful level of competency, whereas with a QWERTY keyboard, you start off, hunt and peck.

>> Jacob Wobbrock: That's true, you can't easily in the same way you would hunt and peck. The Braille type -- the Perkinput work and typing Braille work is obviously meant for a certain kind of profile of user, which is someone who either knows Braille or is willing to learn Braille, who is probably blind to begin with or has a reason to have learned Braille.

And so there aren't very clean statistics on how many blind users actually know

Braille. I've seen very different -- I know that seems surprising, right, but do you have any insight into that? We've talked about this before, and it's.

>> Shiri Azenkat: I haven't been able to find anything about the number of people who know Braille now, but there is -- because speech is so available now and it's available for cheap --

>> Jacob Wobbrock: Learning rates are going down.

>> Shiri Azenkat: There are fewer children learning Braille, which is a huge problem, because they're illiterate, basically. So it's kind of a

[indiscernible].

>>: [indiscernible] keyboards is really bad news is typing passwords on a phone. Do you have any insight into that? It's the upper case and numbers and special symbols. And with the phone, [indiscernible].

>> Jacob Wobbrock: Well, so part of my answer is stay tuned because Shiri's latest work has been on something very much related to what your question is.

39

And at her request, I'm not going to say more right now.

>> Shiri Azenkat: But there has been --

>> Jacob Wobbrock: She can say more if she wants.

>> Shiri Azenkat: There has been other work done in auto correct for password entry. If you Google fast words, it's -- what's the guy's name?

>>: For password --

>>: Sometimes I can set you a review on it, but I'm working on something that's a different paradigm altogether, but -- I can't remember the guy's name.

I can send it.

>> Jacob Wobbrock: One thing I did a couple years ago was see how well people could log in with a single sensor, single button or touch, if you will. So the password became, instead of spatial, which is really what our passwords are now, became temporal. So you tap a little song, like a jingle, like happy birthday to you. Turns out people's individual differences on their down and up times are pretty clear. So we don't actually internalize that song in the same way in our timings. If you only time down, they're similar. But when you look at up, when people release their finger.

>>: So they're always in the same tune?

>> Jacob Wobbrock: Yeah. And turns out to be really hard to eavesdrop. So just a quick aside, Mary's arm is going to get tired, but I will call on you.

Now you know how our students feel, right, in our class.

So we had people do this within, like, a foot of someone looking. Knowing what the song was, and then trying to replicate and log in against their model, and no dice. It original worked, 8%, 9% of the time, which is too much, but if you're giving them the perfect eavesdropping situation. So you could log in on an iPod nano or something if you had personal information like that, or anything with just a button or a single touch pointer capacity sensor or something. But it's a song. It's a temporal, called tap songs instead of passwords, right. Tap songs. So I think there's some innovative work, not just mine, and sort of alternative password approaches.

>>: [indiscernible].

40

>> Jacob Wobbrock: Mary, okay.

>>: Now I have two questions. Tap words, so you're saying you [indiscernible] variation could totally leave for regular people but did you test it in the

[indiscernible] musicians would have much more refined.

>> Jacob Wobbrock: They can sort of eavesdrop each other better maybe or log in against each other. We didn't do musicians, own on my development, I'm an amateur musician for many years. So yeah, I played, yeah, the piano and the guitar for many years.

And so I don't know, I mean, my sense is that is the timings might get a little tighter or something.

>>: You told me what drummers, there's a study on drummers, their ability to type discrepancies in timings is much higher. It's actually --

>> Jacob Wobbrock: There's actually a whole field called psychology of music.

Some of the early references I spent time with was the early 1800s, 1830s and so forth about basically rhythm finding because it does tell you something about transmission time from hands to the brains when they're talking about neural transmissions.

If you think bit, tapping with a signal, with a beat means you actually have to fire things before they arrive. They're isolating those things. I never in that time saw a specific musician, how good are they compared to the Joe

Schmoes of the world. Their book is this thick on psychology of music. It's motor control meets music.

>>: My other question is when we were talking about, like, oh, steno and

[indiscernible] I was actually curious for even touch typing, I mean, what percent of people can touch type? I feel like [indiscernible].

>> Jacob Wobbrock: I don't know. Gosh, that feels like quite a gaping hole in all this, doesn't it? I've never been asked that before and didn't ask.

That's the blind spot, I guess, right like you said, we're all in this room at least pretty good touch typists. Most of us. But many of us, anyway, I should say. No, that's a slightly embarrassing --

>>: I didn't know if it was like 1 percent of people or 90 percent of people.

41

>>: What do you consider touch typing? Not looking at the screen and using like the correct --

>>: I bet those are pretty closely related.

>>: Not necessarily.

>>: I don't describe, but I can type [indiscernible].

>>: Yeah, because people --

>>: You have a non-official way of touch typing.

>>: So the question is how many people are formally trained in it.

>> Jacob Wobbrock: I would consider myself a touch typist because I don't have to look at my hands. I don't even know which fingers I use. Maybe some of them are nonstandard. I think I use my pinky for the P, but sometimes I think

I use my ring finger. I was never formally trained. And I expect that anecdotal description is applicable to many people.

>>: So you'll have to find this out.

>> Jacob Wobbrock: You're giving me homework?

>>: Tomorrow.

>> Jacob Wobbrock: One last question.

>>: When I can't see the keyboard, I find I'm easily one letter off.

>> Jacob Wobbrock: We had a project a couple years ago, actually. I have to say one last thing and I'll close, I promise. I know you're all hungry, and so am I. A couple years ago, we had a project that was Sean King in 2008, it was called true keys and it was motivated by a computer science professor at UW, who is an emeritus professor, but his son worked at the med school. His father, informer computer scientist, had peripheral neuropathy, which meant his fingertips were numb and not very sensitive and he couldn't sort of feel the bumps and ridges and all the things we've just been talking about.

42

And so he would often land on the keyboard being off a row or column or whatever, and he'd have that experience that we've all had at some point where you're just off completely. So we developed a system that took keyboard geometry into account, physical keyboard geometry into account, and could rectify that so you could actually be off a row and it would just work. So that actually happens occasionally. I'm not sure if that's solving the world's greatest problem, but it was a fun project. Okay. Thanks a lot.

1

Related documents

Products

Support

1

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib