22445
>> Sumit Gulwani: Thanks for coming. And it is my great pleasure to introduce
Professor Joseph LaViola from the University of Central Florida. So Professor
LaViola is a SIA faculty in ECS Department here at the university. He's visiting us on a short trip to collaborate on a project, interdisciplinary project on building and drawing intelligence that describes sketch recognition with some learning theory. So Professor LaViola is a leading expert on sketch recognition and application to building intelligent tutoring systems and today he's going to tell us a bit more about his work.
>> Joseph J. LaViola: Thank you, Sumit. So welcome. Thank you for coming.
What I want to talk today is on moving towards using mathematical sketching, which if you don't know what that is, I'll define it, in intelligent tutoring systems.
And I actually gave a talk here at Microsoft maybe about five years ago on mathematical sketching and something called Math Pad, and at that time we were interested in trying to develop dynamic illustrations for a variety of physics concepts. Now what I want to talk about is sort of I'm going to give you a little bit of history to see where we were and then I'm going to look and see where we are right now and where we want to go in the future.
And we've sort of developed a new system we call it Physics Book. My student
Salman is sort of the chief person on the project. It's his Ph.D. work.
So pen-based interfaces is sort of the area that one of the areas that I work in.
And we're talking about interaction with a stylus or perhaps your finger, using a
Tablet PC or interactive white board or various touch tablets.
And one of the interesting things that I've seen over the last couple of years is that pen-based computing, pen-based interfaces or pen-based interface research is sort of maybe not as important anymore or people aren't doing it as much and people are all focusing on multi-touch. And touch-based interaction.
But I still believe there's a significant amount of work to be done in the pen-based computing world, specifically when we're talking about applications that actually take advantage of that technology.
And one of the things that it takes advantage of is it allows you to mimic pencil and paper with one of these tablet computers or interactive white boards.
They're really good at entering 2-D notations things like mathematics, chemical diagrams, drawings, musical notation, things like that, anything that has a two-dimensional representation it makes sense to be able to use a tablet or a stylus to enter that information, because you don't have to worry about some one dimensional language that you have to transcribe into the actual notation.
And what this then provides us with is what I feel is some very relatively new interaction methods and styles that focuses specifically on STEM education, science, technology engineering mathematics.

So our sort of overall vision is we want these pen and touch-enabled devices to offer natural user interfaces, natural user input as I say close to pen and paper, pen and paper and provide an easy learning curve to be able to use some of these tools.
And in order to do that, what we need to do is to do sketch recognition, that sketch recognition involves the ability to do handwriting recognition, understanding mathematical expressions, understanding free form drawings, and then what we end up doing is augmenting pen and paper with the power of computation.
What this then does is hopefully provide us with some interesting STEM education tools.
So I sort of structured the talk as a past, present and future talk. So we're first going to go back a little bit. When I first started this, I made the observation that these diagrams and illustrations that you see in physics books and mathematics books, they're useful for explaining mathematical concepts. But they only really aid you in the initial formulation of a particular problem.
They're static. And therefore what we want to be able to do was to try to make them more dynamic so they would provide more information to the user as they are working through a particular problem or understanding a particular concept.
And that's where mathematical sketching came from, which is basically the notion that you sketch diagrams, you write down mathematics. You associate the mathematics to the drawings, and then you perform some animation.
And this was manifested in a prototype we call math pad. And what I want to do is just give you a couple of quick examples. I know this is old, but I want to give you a couple of examples because we're going to see why we need to move forward with this, and what the limitations are in the math pad system.
So let me go ahead and I'll start I'm not going to worry about showing you that you can recognize handwriting mathematics. It's pretty much -- math recognition,
I wouldn't necessarily say it's a solved problem but it's pretty close to being a solved problem that it just works. I've got some precaned mathematics here.
And this actually represents damped harmonic oscillation.
So I will go and I'll draw ceiling and a little spring and a block. And I will nail these objects together and I will grab the mathematics and attach it to the spring.
Needs a little bit of time here. There we go. And I'll run it. And you see the animation. Okay. And that's all well and good. It gives you a sense of what's happening in terms of a dynamic illustration. And one of the other things that we did was I wanted to go further and see how far I could go with making the illustrations.

So what I wanted to do was give the ability to actually write little programs, little more sophisticated programs that would allow you to do various interesting little things.
So here's an example of air drag, or [inaudible] subject to air resistance. What you see here is there really is not a nice closed form solution for this. So what you have to do is you have to essentially doubly integrate the differential equation using a simple, you can use a cutter or in this case I'm using oil integration.
And I go ahead and I draw my little floor here. And I'll say this is 100. And we'll put, we'll label this guy as P. Okay. Let me turn sketch trails on. I'll run it. And you can see in this case you can see how the air drag coefficient is sort of pushing the objects so it looks more realistic.
And I'll just show you one more. This is sort of my favorite one. This is mathematics that represents the integration of an orbital motion differential equation. So I'll go ahead and I will draw something that looks like a sun, I'll label it. And then I'll draw a ball and label it P and then I'll run it and you'll see it does an orbit around -- in this particular case the orbit is a circular orbit, approximately circular.
And the cool thing about this, when I designed it, was that you could just change a variable. So I'm changing the initial position of the object and I run it again, and you'll see that it provides you a nice sort of dynamic behavior.
So it really gives you -- the ability to sort of see how different parameter changes affect the animation. Okay. Now, this was all well and good. But the problem was that we had to essentially provide a full mathematical description. And that's okay in some cases, but if I'm trying to think about how we want to use it more in a physics class, we needed to sort of deal with some very fundamental problems.
That's where we get to the present. Okay. And what we're interested now in doing is sort of taking that, taking mathematical sketching in math Pat and try and bring it into intelligent tutoring.
The features of intelligent tutors, of course, is it helps build students knowledge, it helps them understand problems and provides scaffolding to help them as they work through problems.
And in this case another ideal situation for pen-based interaction because students are used to solving these problems on pencil and paper, and once again it provides more natural interface.
And then what we can do is use these animations to provide intuitive concepts or intuitive understanding of these concepts. Now here are a couple of physics problems. Couple of simple ones I pulled off the Internet. In this case we've got
200 Newton block suspended by three cables.

And we have to find the tension in each cable. Now, if we were to take this and try to animate it in math pad, we couldn't do it. All right? Because, first of all, the problem is not asking us to specify how an object is going to move through time.
Or how we might integrate a differential equation to have an object move through time. It's not asking us that question. It's asking us to simply find the forces that represent the tensions in each wire.
Here's another one. In this case what we've got we've got two blocks, one on top of each other, and you're pulling them across the floor. You've got kinetic friction and static friction happening with each of the blocks and you want to see what the maximum force will be before the block on top actually starts to move in the opposite direction.
Okay. Once again, we couldn't do this. We can't do this in math pad, because it's asking us for a simple number, an answer. And what we find actually in most physics books, especially in high school and introductory books in college is that
90 percent of the physics problems ask for a number. And therefore we need to improve mathematical sketching capabilities in order to support this type of thing.
If our overall goal is to provide an intelligent tutoring system with mathematical sketching.
So how do we go about doing this? Well, we have to once again as before understand the sketch that the user's written. We have to understand the mathematics that the user has written. We have to try to facilitate the user learning in some way with scaffolding or other types of tools and we don't want it to be so burdensome that it's so difficult to use.
>>: I'm just trying to understand the difference between the previous class of problems that you [inaudible] problems previously and the next the first class was about animations.
>> Joseph J. LaViola: It was basically about doing animations and trying to illustrate things dynamically with the caveat that you needed to specify how things move through time.
In all the examples, you always specify something as a function of time, whether it was a closed form solution, F of T equals T or whatever, or whether you actually had to take a differential equation and solve it. Okay? But each one of those, in each of those cases, the problem was always specifying how am I moving at each time step.
>>: So I recall from physics classes there used to be two different kinds of chapters, more statics and one was dynamic, the previous one was dynamics and now you're talking about statics.
>> Joseph J. LaViola: Yes. I mean -- okay, that's a very good question. It turns out that in both statics and dynamics there are problems that we want to animate, we want to provide animations that provide insight into the problem in both cases. And you're going to see an example of how we do that for a statics problem. Okay?

So obviously the goal is to do a ritro set of animations and develop an understanding of the problem and so on and as a result we can hopefully understand the student solution.
And what we've developed is something we call physics book, which is a next generation of math pad, and what I want to do is show you some videos of what physics book can do in its first version and what I'm going to -- I'll highlight some of the features as we move along, and then we'll talk a little bit about some of the details.
So let's go to the first video. Okay. Now, I apologize for it being a little bit slow.
But when we recorded it using Camtasia it took a little bit of time away from the runtime. Now, in this first example, one of the things that we did was we added a physics engine into the system.
So we have an underlying physics engine that sits behind the scenes. All right.
And what that allows us to do is it gives us lots of flexibility. We can simply make a drawing without having to add any mathematics at all, or we can go ahead and actually supply mathematics to and associate that to the drawing.
In this case what we're doing is we are applying basically a position as a function of time which is what math pad had before because we want to make sure we've got the functionality that we had in math pad into the new version, because I'm trying to keep it in and expand upon it. So in this case what we're doing we're writing the equations of position and you can see this is sort of a projectile equation in this case.
And we've added some things here. You'll notice one of the things that this is in real time, the recognition is in real time. You don't have to circle it. And then have the system recognize it. It's recognizing it in real time.
And what you're going to find here is that there's an initial bumpup and then it drops down. Some other differences as well you'll notice is that we actually do an analysis of the drawing and then we beautify the drawing, and this was something that we decided to do to make it a little bit more easier to understand.
You'll also notice there are vectors attached to any object that show you what's happening as you move through. We put a ground plane in there to see.
You want this one here. So that one we showed with position -- this one we're showing velocity. So we have the ability to enter velocity equations, position equations, acceleration equations into the physics engine and the physics engine understands those equations and then essentially throws them into the overall physics engine architecture and allows us to to do that.
Okay. Do I want to show this one? I'll show this last one here. This one's a little bit interesting as well because we use two springs and make two springs and then we have a circle or a sphere that we put on top of this structure.

And in this case all we need to do is essentially just enter the mass of each of these objects. And so what this does is that it provides flexibility in terms of what you can specify into the system.
If you want to do a full specification, you can, if you want to do a minimal specification you can. And that's important.
>>: In this are you assuming that the sketch is always going to be perfect or what happens, how does the communication relate using the sketching --
>> Joseph J. LaViola: Well, if, first of all, if something is not recognized correctly what we end up getting is the ability to simply erase that offending ink, called the offending ink, and then rewrite it or if one of the drawings is not recognized correctly -- see, the beauty about this you can simply erase it and do it again.
>>: How does the system know that it has not recognized a rank correctly, given a different interpretation and vector saying this is the final --
>> Joseph J. LaViola: Well, right now it doesn't necessarily know that. It's sort of -- we're making the assumption that you're doing things correctly in terms of setting up the problem. You're going to see an example where -- let me show you. Okay. Go ahead.
>>: Can you turn off the beautifying of the pictures? It looked much nicer.
>> Joseph J. LaViola: That's an interesting question. Currently we don't turn it off, but we certainly could do that. One of the interesting things about leaving them in their natural state is you have to make sure that there's the underlying representation exists so that you can do a proper animation, because unfortunately when you make the drawings, they're not precise. They're imprecise. So you have to do some rectification, and what we did in this case was actually just do the rectification and show that to the user. But that's something we can add very simply as part of it. I like to always try and maintain that notebook style, if you will.
Did you want to say something?
>>: Just [inaudible] so the goal here is we need to allow the user to be able to animate it given any form whether it [inaudible] equations or gives partial, so you just mention gives you a value for the [inaudible] you should be able to -- this gives you the equation for velocity whether in vector form or XY coordinate form.
So the focus really was to make the math work. But that could be -- that's really --
>> Joseph J. LaViola: I'll talk about some of the more challenging issues that we're struggling with right now as we get through.
So here's another -- this actually was -- the video I showed you was physics book version 0.5. And now this is physics version one. So you'll see a few changes here. A little bit of different style.

And what I wanted to show in this case was so here we're going ahead and we're doing once again creating the velocity vector. And then going ahead and writing down the vector equation. You're going to notice something here, is that A is not actually specified. And this particular case, since A is not specified you're going to see it's not behaving exactly the way we want it to. So we can go ahead and replace A with G. G is the gravitational constant, built into the system. We run it again and we see the behavior that we want.
>>: Do you have it shown for the built-in equations and constants?
>> Joseph J. LaViola: Yeah, you can do that. We don't do that. Over here we see it says what gravity is right there. So now as you can see we're starting to get into a little bit more about of the problem domain where we have specific problem. So we're trying to find the force that acts on this object. We go ahead and we, whatever the prewritten math or we've already recognized it. And drag it to the object. And actually in the very latest version you don't have to do much actual circling and dragging to the object. It kind of does it for you.
So once again we have the ability to experiment with different values. You can see we're trying to keep stuff that was in math pad and bring it into the new system. But there are subtle differences here. Let me see here.
Of course when you hover over the object it shows you all the mathematics associated with it to help you understand what's going on there. And as one of the ways to -- [phone ringing] to keep track of current values is we actually show those to you on the drawing. So here's an example where a little bit more sophisticated example where we're looking at modifying inclined angle and you can see we use slight adjustment there into what the actual angle was and we associate the math to the ball and the mass to the ball and then specify the normal force.
What you can do is look for different quantities as you're animating. So it shows you the actual, the sum of the forces as it moves through the animation.
>>: Sliding or rolling?
>> Joseph J. LaViola: It's sliding/rolling. It's not rolling. It's sliding.
Now, here's sort of the one with the initial problem we looked at. We've got the tension in two wires attached to a block. And we go ahead and we make the appropriate associations. As you can see, we make those associations the drawing gets rectified. And they want to find the tension.
So then what happens, is once we associate the tensions to the individual wires, we're going to run the animation. And this is something that's very interesting, because there is no animation, and there shouldn't be, because we're in equilibrium.

So what we're doing actually here is showing that you've gotten the problem correct by not showing any animation at all. Because that's what the problem calls for.
Now, if the user made a mistake in calculating that tension, let's say they chose a different value for it and calculated a different value, what we would get then is you would see -- well, you'll see what happens. So the tension wires break and the object falls. So what we're trying to do here then is visual demonstrate what is happening with a problem, regardless of -- so it's not specifying equation and motion.
In this case we're taking it in the opposite direction. We're saying statics problem, if you get the problem right, nothing should move. If the problem is wrong, it won't move. Yes?
>>: Does the system -- thinking it was incorrect because the problem could have been, you know, some weird state.
>> Joseph J. LaViola: Yeah, we sort of make underlying assumptions.
>>: [inaudible].
>> Joseph J. LaViola: Well, we know what the tensions are if you look at the sum of the forces, you'll be able to determine whether or not you're in equilibrium.
If you're in equilibrium the forces are equal to 0 therefore nothing happens in the forces engine. So that's kind of the way it works.
>>: But you're making the assumption that the way the problem is drawn is that it is in equilibrium state problem and not like a specification of state in some moment that could have occurred such that you end up with this other kind of behavior?
>> Joseph J. LaViola: Yeah, pretty much. One of the interesting things about this stuff is that we sort of have all these physics problems to go by as effectively our sort of skeleton problem domain. And so we can take some liberties and not having to deal with all the absurdities of physics and stuff.
So anyway, that's an example. Let's go back to the slides.
>>: If you're not writing this things down on the side and associating it with an object can I notate the object with an L or something that we normally see?
>> Joseph J. LaViola: You can, yeah, future version we'll -- I'll briefly mention some of that later. So a little bit what's under the hood, what's happening. Basic architect is relatively straightforward. I've got user interface which deals with the ink and gestures and renders the animations.
I've got a sketch analysis component, which deals with the recognition and the rectification component, and then the simulator, which represents the physics engine, which not only has the physics engine but also deals with specific entities

in such as springs and wires and all of that sort of flows with the association inferencing system which allows us to associate the math to the individual drawing elements. Part of the most interesting part of this is the physics engine because we could not simply use an off-the-shelf physics engine, because what we needed was one that provided us with a way we could sort of inject mathematics into it as it's running.
So what we did was we based it on a typical game physics engine. And it uses traditional approaches in a certain sense, position is updated by double integration of the acceleration or single integration of velocity. And then the collision detection, resolution is sort of standard. Although what we do is we use exponential smoothing to keep track of net motions which is very important when we want to do rest and sliding context. Because that turns out to be really tricky.
But what we can also do with it is we can replace the update equation with any equation we want. What the system does is it looks and it figures out, okay, this is a velocity equation, this is acceleration equation or this is position equation and then throws it into the physics engine and that takes over.
Now, then you might ask what happens if you collide with something an object, well, unfortunately, in those particular equations the user is not specifying what happens after a collision takes place. So what we simply do is we just break the user defined behavior once a collision hits. Which makes sense, right? Because that's sort of the best you can do with a user defined behavior unless you actually provide information about what happens when you collide with another object.
And then, of course, we provide what I call hooks into the physics engine and those hooks are for other things, other physics concepts, and we'll get to that when we talk about sort of the version two of physics book. Yes?
>>: When you say collisions, is that user defined, that you mean you just stop the system as it is or you complete a new set of forces using more sophisticated
[inaudible].
>> Joseph J. LaViola: What we do is once we know we're going to get a collision, we sort of say all right. The user defined equations are no longer -- let's take sort of the last piece of information we can from those equations and then use that to feed that back into the physics engine to calculate new forces and positions so that we get it so that it's sort of a continued flow, right, in the animation.
So in terms of recognition, we do some relatively straightforward things in the recognition. All ink strobes are resampled and we do that to remove noise and ensure that all the points are equally spaced.
And we also made an observation that typically the diagram components that you draw are going to be larger than the mathematical expressions and that helps us to distinguish between a drawing and a particular mathematical symbol, especially if we're labeling a particular drawing. And then in terms of the recognition we have to recognize, we first recognize the diagram components.

We look for specific shapes, wires, springs. And then the remaining ink strokes are recognized as mathematics.
In terms of shape recognition, we do some very straightforward things. We look at number of cusps and self-intersections. Cusps are just components that have very high curvature if the stroke is closed or not. That also gives us a sense of whether or not we're dealing with polygons or circular objects. And we use the measure of circularity where we look at the standard deviation of the angle about the centroid and that also gives us a good indication that we're dealing with a circle.
So we're very simple with the shapes we have right now. Polygons and circles.
Circles, you've got two cusps and high circularity and polygons you've got more than two cusps.
Of course, as we move forward if we want to add other shapes we have to get more sophisticated.
In terms of wire and springs we sort of handled them a little bit separately because they sort of have unique structure to them.
One of the things we have to do is we have to make sure that one end touches a recognized shape, and so with springs you have to have more than three self-intersections and with wires they can be relatively straight lines. That's sort of a way to make determination between those.
As far as the mathematics recognition, as I said, when I started doing all this stuff, you know, 5, six, seven years ago, something like that, math recognition was still, people were still working on it and stuff and I did my own math recognizer. But we now have some pretty math recognizers out there we use one called Star Pad or math recognizer in Star Pad which was actually funded by
Microsoft when Microsoft was supporting the Pen Center Brown University. So it's a pretty nice system and works pretty well.
Now the other thing that we have to be able to do in trying to go from a sketch to an animation is what we call rectification. And rectification sort of says let's take the drawing and get it ready for an animation by dealing with, by sort of -- by going from approximate to precise. Because if you want to make it look proper, you have to make sure you've got precise specifications in your drawings, even though they may look approximate.
And this can occur during the recognition process. It can also occur when you make associations between an object. We saw an example of that where we changed the angle of the incline plane and in doing that we actually changed the triangle.
And this can be applied to both individual items or groups of items. So in terms of shape recognition, one of the things we sort of ensure is anti clockwise ordering polygon vertices we do that because it makes it easier to understand how we want to do the rectification.

We also deal with vertical and horizontal alignment of edges. So if you draw something that's approximately horizontal or approximately vertical we'll treat it as vertical or horizontal.
Other cases, for example, let's say that I draw classic example is with incline plane problem and I draw a circle on top that represents a ball. What does that mean? Should that ball be on the circle? Should it be within the circle? Above the circle? We sort of make the assumption what you're trying to do is have the ball placed on the circle and we move it there to make sure it's proper.
Associating the angle with a particular vertex of a triangle is a little bit tricky because depending on how the triangle is drawn we have to figure out exactly what angle that we're moving. So we have a different algorithm for different cases in the way that triangles are drawn differently in terms of different orientations.
Now, in terms of the springs and wires themselves one of the things we have to ensure is you may draw a spring or a wire and it may go inside the box a little bit or right above the box, we make sure we identify the object and then we make sure that the end of the spring or wire is actually touching the block, let's say.
We also support vertical and horizontal alignment in this situation as well. Most of the time what you're dealing with is a vertical or horizontal. But as with the tension example sometimes you know it is an arbitrary angle.
Okay. All right. So that's just really a little bit under the hood. I didn't want to get into too much detail of that because that's not the purpose of the talk.
Now what I want to talk about is the future, where we're going, this is where the really interesting stuff is starting to happen. And I want to talk about right now is a relatively important strategy that we've come up with and it's actually really, really simple. But it makes a lot of sense if you think about it.
And we want to be able to support our ideal goal is to support 100 percent of all the physics problems in a physics one textbook. That's our ultimate vision.
Right? It's also very challenging vision.
So in order to do that, we need to be able to -- we have a lot of coverage. So we started thinking about strategies and ways that we can get as maximum amount of coverage as possible. And one of the ways to do that is if you look at newtonian mechanics, you see that force equals mass times acceleration, represents the fundamental equation that all the other concepts are relatively based on.
So if we have an understanding of how the other concepts in physics that are not directly related to F equals MA, how they translate to F equals MA, then if we can identify what type of problem we're doing, then we can figure out what the mathematics is necessary to go from whatever concept we're interested in to F equals MA and if we can do that then we can do an animation of something.

Therefore, what I'm saying is if I do a problem with work, where I'm asked to find the work of some object or the kinetic energy or the momentum of some object, or potential energy, for example, I can then translate that information into F equals MA and then do an appropriate animation.
So that really expands the type of problem set that we can support. And that's one of the strategies that we're working on now.
And that's where physics book version two comes into play. This is sort of -- this is sort of present/future, because it's under review right now. So I can't really talk too much about it. But I'm just going to give you some of the highlights.
And you'll notice what we're starting to make it look more and more like an intelligent tutoring system. You can see we've actually got the question up there.
And in this case you've got a mass that is lifted by a certain height. And the forces applied by a person pulling on the rope. And you have to find the work done by what happens when you raise it by four meters. So what you have to do is calculate the work and then in the system, right, it will tell you whether or not this object moves four meters. And if it does, based on this work calculation, you've got the problem correct. If it doesn't move four meters, then you know you didn't get it correct.
So that's sort of the next generation. In order to do that we have to sort of add some other types of things to the drawings. In this case we're adding a displacement to the drawing. And this, frankly, is something that you would do anyway as a student. The proper thing to do, right? Maybe not all students would do this, but it's the proper thing to do going through the process of solving these problems. Not only does it help you make the animation work but it's something you should do anyway.
And so what this allows us to do then is newtonian mechanics, kinetic energy, potential energy, simple poise and incline planes, things like that, where you're not actually specifying where you're specifying the force or work or whatever.
And that really sort of helps to make it much more robust to starting to think about using it in a physics classroom. Obviously our ultimate goal is to get it in front of physics students and have them use it for a semester and then see what happens.
All right. Now, all of this sort of leads into what we're sort of thinking about now and into the future. And one of the things that I realized was you know, what you could do is have a set of canned problems, and understand those problems completely. And then provide them into the physics book.
But I didn't think that would be that very much fun. So what I'm interested in doing is taking any problem and trying to understand what it's telling me. And if I know what the problem is, if I understand the problem, that will lead to understanding of what solution you might do in order to solve that problem.

So what we've been doing is actually we have a rudimentary version of this. We do natural language processing on the physics problem itself, to extract important pieces of information. So we look for things like, you know, number quantities, what the problem is asking for we look for keywords, velocity, pulley, acceleration, force, things like that. And from that information we can then, or our goal is then to define exactly what the problem is asking. And if we understand what the problem is asking, then we know the types of solutions that we can have for that problem and if we know that information then we can provide the scaffolding necessary. So as they work through the problem we can identify where they are potentially in a reasonable set of steps that you would go to solve the problem and provide help along the way.
So that's something that we're moving forward with. And what we would have essentially is a knowledge base of solutions, right, based on certain concepts.
We know if we have a certain concept we know this is the approach, general approach you would take, then you would try to match that approach to the actual problem.
And the other thing that we're realizing that we need to do is bring the solution into the drawing. Okay. One of the things we're also realizing is that some problems just don't provide us with a good way to animate them, to show that you've got them right. The way I describe this to laymen is I say think about the back of the book of your physics book, and the answers. And what I say the physics book is the great question and answering back of the book you've ever seen. That's our ultimate goal. And some problems just don't -- they don't amend themselves to this type of behavior.
So we've been thinking about ways in which we can fix that. As an example, okay, let's say I have a problem that is asking me at T equal five seconds what's the height of a projectile?
So you write down the height. Now, how do we animate that? Because you could have the object animate and the projectile motion, but at five seconds how do you know you've actually got it right.
So we've been thinking about providing a mechanism where we can throw the solution, have the user actually enter the solution into the problem. And, for example, something like this, you know, you've got a ball here and you know this is the height or that you want to get to and let's say it's a T equals 5. And I can say, all right, just doing this and I say that the height is going to be ten meters, let's say, then I could, what we could do is then animate this piece here to show that as this ball is moving, right, then this is going to increase and decrease as, with the height of the ball and at T equals 5 we show what is that value, it's equal to 10, okay, I know I got the problem right because that's the answer.
And we would have like something like this that would say this is 0 and this is 10 units or whatever.

So we're thinking about ways in which we can add this type of behavior into the drawings so that we can then provide an appropriate animation that makes sense.
Now not every problem is going to be animatable, as I call it. They just don't facilitate that. But what we're trying to do is figure out ways in which we can do that as best we can, as many problems as we possibly can.
Okay. So basically what I've talked about today is mathematical sketching is I believe is ideal for intelligent tutoring systems, especially in the physics domain.
And I think physics are a classic example because of the fact that there's so much motion involved. Physics is really about motion.
And what we've done is we've sort of developed a new system called physics book, which is starting to look at how mathematical sketching can be brought into an intelligent tutoring system and we've developed a couple of different versions and continuing to work on it. The biggest thing is integrating the physics thing into the system, because it provides more flexibility in how you specify a mathematical sketch. And we're also, that also amends itself to being able to do problems we couldn't do before such as the static problems or problems where it's asking you for other quantities other than force or acceleration.
And, of course, understanding the word problem itself is I believe very important.
And will go a long way to helping us understand the student solution on a high level. A high level, I refer to it as high level semantic recognition. Mathematical expression recognition is you recognize the syntax and then you recognize the parse. High level semantic recognition is understanding that that equation that you've written is an equation projectile motion or equation for kinetic energy. You can understand that type of thing then it goes a long way to help you understand what's going on.
And of course there's tons of more work to be done which is why Salomon is going to be working on this problem for many, many years to come. And we'll do a great thesis.
So I just want to thank my sponsors. The primary work is supported by the career award, and I'll take any questions that you have.
[applause].
Any questions?
>>: The automatic, or the natural language processing, so the idea there is to just sort of copy and paste some existing word problems?
>> Joseph J. LaViola: You would copy it. You would get it off the Internet or you can actually write it, too, if you wanted to.
>>: So it seems like that -- error prone.

>> Joseph J. LaViola: Could be.
>>: In practice --
>> Joseph J. LaViola: I mean, we're really just touching the surface on that right now and what we have is the ability, it actually extracts variables and important keywords and numbers and then what it does is it says, okay, given this variable, tell me all the mathematics that I think would be related to it in terms of physics problems, and that's really where we are at this point at this point. And we incorporate it into the physics system and it will hopefully provide us more information. One thing I didn't talk about is what happens when things go wrong, right? That is a huge problem. And I've sort of been punting it for a long time because it is such a hard problem, and I wanted to focus on trying to make, to actually do the first problem of seeing what type of animations we could make.
But at some point we will have to tackle that, and that's where reinforcement of problem math and drawing altogether can hopefully provide enough information to let us know what's happening and what's going wrong so the user can enter things if a mistake is made.
>>: It seems like for this part, in particular, there was an understanding of word problems there may be something that's expert or other people can sort of annotate them.
>> Joseph J. LaViola: You can do that, right. That's one way to go. There's work guy by the name of curt vanlien who is now at Arizona state, I believe. He was formerly at Pittsburgh, he developed something called the Andy's physics
Tutoring System. What it does is it's very similar to what physics book does except it's not pen-based. Doesn't allow you to write anything.
What they've got is a set of canned physics problems, because they've got those canned physics problems they know the exact steps you need in order to solve that problem and they provide the scaffolding. I thought, yeah, I could do that.
Buttion I wanted to be cooler, I guess, try to do it automatically.
>>: Seems like there could be a lot of sharing.
>> Joseph J. LaViola: Yeah, but that also begs the question, you know, I can solve a kinetic energy problem in a number of -- actually, I can solve a force problem in a number of different ways. Or like I'm asking for the velocity I could ask kinetic energy to find velocity or I could use force to find the velocity. So there's like multiple ways to solve a problem and how do you deal with that, can you identify that there are multiple ways to solve the problem and then figure out that the user is actually taking one of those paths, which is also very interesting.
>>: Tell somebody which is --
>> Joseph J. LaViola: Yeah. That's a lot of -- this is why they call it a career award, because I've been doing this for the rest of my life. Any other questions?
Yes.

>>: How far do you think we're away from using these systems in the classrooms?
>> Joseph J. LaViola: You know what, it depends on the system. The simpler the system, the more readily available it will be in the classroom quicker. As an example, we have a system right now which animates simple data structures for
CS 1, link lists, trees, things like that, where you actually sketch out a link list, you sketch out a tree, and thrown you can actually, you actually write code and then the code will animate the drawing you made.
That we're going to start, we're going to use it starting in the fall. It's relatively straightforward. For something like this, it's so many different pieces, so many different components, that it's going to take a little while.
But I think what will end up happening we'll end up doing it on a very constrained basis where we'll say we want you to use this for this particular physics concept and you're going to spend two weeks with it and make it really robust for that concept and then see if it actually helps people.
But actually saying that I can take this and give it to a physics one professor and say all right all your students should use this throughout the whole course is a long way off.
>>: So are you targeting high school, the high school in particular or is it more
[inaudible].
>> Joseph J. LaViola: It would work for both AP physics in high school and also physics one for engineering and engineering majors.
>>: CS 1 system that you talked about this one is that CS 1 is presumably for undergraduates.
>> Joseph J. LaViola: Yes.
>>: Therefore it's easy to force to use the system, but bringing this technology into the high school might have a different set of nonparticular problems.
>> Joseph J. LaViola: Yes and no. You know, the high school teachers that I have talked to would be very -- they would love to be able to use it. Of course, we would have to provide the hardware that would enable them to use it, typically what we would do is have a tablet, portable tablet lab and we'd have them, obviously having them take it home would be -- you could do that because this would work with a simple welcome tablet device, thank you to Windows 7. And it enables you to do pen anywhere, as long as you have the right hardware.
So it's really just based on the scope of the experiment that you want to run when you're using it, for the students or the teachers. And one of the things we can do right now very close to being able to do is actually have the teachers use it to teach with. Because they're more -- it's more constrained user defined set. And

then the other thing you can do you can look to see if the students themselves actually gather more understanding when they're being taught with this tool than if they're not being taught with the tool.
That's something that we definitely should be able to do. But yet giving it to the students and saying whole semester you're going to use this, that's going to take a while.
>>: Another question, do you also massage these systems being used on smartphones, smartphones [inaudible].
>> Joseph J. LaViola: That's also a very interesting question. Because we're in the world of touch now [Apt] ( I was at kooi two weeks ago maybe 30, 40 papers on touch. You know how many papers are on pen? 0. A lot of stuff on phones and stuff like that. I think you could potentially do something on a small device.
The problem with a small device is the handwriting, and entering the mathematics. And that poses a bit of a challenge, because, first of all, you have a much smaller screen space to do it. You typically don't have a stylus.
Although some phones do.
I know that there's been some work in trying to recognize simple characters using your finger. There's actually I believe there's an Android application out that allows, that does character recognition with just writing with your fingers. But simple characters is a lot different than X squared times the square root of X plus
3 over 2. So it's possible.
The other thing that you could do is something that I've thought about, too, is you don't provide all the functionality to somebody on the phone. Maybe you just provide the viewing capability. So they at least get to see things and then provide them with the ability to change parameters. So from sort of a student learning point of view you're not giving them everything but what you're doing is you're giving them at least stuff, enough machinery to explore. And it may not help you in solving the problem but basically it will help you explore the visual, the physics concepts. That remains to be seen, I think.
>> Sumit Gulwani: Any other questions.
>>: Do you actually depend on pressure information from this group or do you just --
>> Joseph J. LaViola: It's just discrete.
>>: You could use the capacitive styleless.
>> Joseph J. LaViola: Yes, the thing about pressure, you know, I actually, when
I first started doing this, I was using pressure, and I took it off, because --
>>: I wasn't implying you should use pressure, I'm saying since you don't use pressure, do you use capacitor stylized on test devices?

>> Joseph J. LaViola: Yes.
>>: And append has a rate certainly not as good.
>> Joseph J. LaViola: Yeah, one of the things you have to be careful with those types of devices is your hand can get in the way. So you end up writing like --
>>: Yes, that's true.
>> Joseph J. LaViola: But there are some pretty decent palm rejection algorithms out there that work. So, yeah, definitely.
Anything else?
>> Sumit Gulwani: Okay. So let's thank the speaker again.
>> Joseph J. LaViola: Thank you.
[applause]