22905
>> Michael Zyskowski: Good afternoon, everyone. Thank you all for coming
today. My name is Michael Zyskowski. And I have the pleasure of introducing a
high school intern that's been with us this summer. Her name is Tara
Balakrishnan. Hopefully I'll get that right. Tara is a high school student at
Redmond High and will be a senior this fall, 2011, and she's been working this
summer to evaluate the services for cloud-based services from the Project
Hawaii effort on a Windows Mobile 7 device. And she's made a really nice
project that she'll go into detail called ChatArt.
In addition to that, I thought I would mention that Tara was a finalist on an
intern-wide competition for all interns at Microsoft. Another WP7 application she
wrote outside her time at Microsoft called Pictionary. So I encourage folks in the
audience and viewing online to take a look at that if you have a chance.
In addition to being a rock star developer here at Microsoft this summer, Tara's
also into robotics. And she participated in a VEX robotic championship last
summer. And she also has some hobbies outside of the technology field where
she has been studying Indian classical dance since the age of three and has
performed in various places around the Pacific Northwest.
So with that, I'd like to welcome Tara to the stage.
>> Tara Balakrishnan: All right. So, hi, everyone. Good afternoon I'm Tara.
And I wanted to thank my mentor, Mike, for helping me out this summer. And
today I'm going to talk to you a little bit about an application that I developed
called ChatArt. And it's a Windows Phone application. So a little bit about me.
I am going to be a senior at Redmond High School. And this summer I worked
with Microsoft Research Connections and I worked on developing an application
for the Windows Phone that diverges the Hawaii services, specifically the relay
and rendezvous services.
So the goal of the relay service is to connect users across the cloud. And I
wanted to use the service in my application. My application is called ChatArt,
and it allows users to interact through a pictographic messaging system. So
instead of just sending text messages you can draw an image and send that to
either a single friend or groups of friends.
So let me tell you a little bit about project Hawaii. So the purpose of the Hawaii
services is to utilize the cloud's computing power instead of just doing everything
on the client device, you can send something to the cloud and utilize that power
for what application you're trying to accomplish.
There are four services. One is speech-to-text. The other is optical character
recognition, and the two that I worked with were the relay and rendezvous
services.
The relay service allows you to send messages across the cloud and the
rendezvous service lets you associate a device with a friendly name.
So let's talk a little bit more about the relay service. The relay service allows you
to create an endpoint which is a unique ID for each device which can be used to
transmit and receive messages. The endpoint is a location in the cloud where
you can send and receive messages from.
You can also have a group which allows you to send messages to a number of
people instead of just a single endpoint.
The relay service also has a storage queue so you could read messages from.
So you don't have to read it instantaneously. You can read it at any point. And
the rendezvous service allows the user to associate a friendly name with that
endpoint ID. An endpoint ID is a series of characters and numbers that's like ten
or 12 characters long. So it's not very user friendly to have to type in that ID
each time you want to send a message.
So instead, you associate a friendly name with that ID. So the services that it
provides are allows you to register a friendly name, deregister the name,
associate the ID with the friendly name, disassociate it. And look up the ID that
is associated with a particular friendly name.
So this is my application is called ChatArt. And here are some of the main
features of it. It allows you to communicate across multiple devices. Send and
receive both pictures and text messages.
And since it's on a Windows Phone it's obviously a touch-based drawing. My
goal was to design an interactive user interface for the application. And I
employed the native storage of the Windows Phone for the chat history.
So here's just a couple of screen shots of the application. It shows in the first
screen shot you can see the home page. And when you click new chat, I'm
creating a new endpoint. So it checks to see if there are any old endpoints on
that, that aren't in use any longer. If there are such endpoints it deletes them and
creates a new endpoint.
Then it allows you to select either friends or a group of people that you want to
chat with. Right now I don't really employ the rendezvous service entirely. The
only instance I'm using it is actually for Player two. So you can either type in the
entire group ID or select one of the existing IDs that I have that will later be more
fully implemented. And this is the basic drawing interface. I'll later do a demo.
But it just shows you have a color palate since drawing in black and white is
obviously not that exciting.
You can type in a text message and send that as well ascend a drawing. And
the last picture shows the history. So you can see the messages as people send
it to you.
So I'm going to talk a little bit about the architect of the program and how one
phone communicates with the other. So what I have is -- on my first phone, I
draw a picture. And then I draw the picture on a canvas element and I convert
that element into a writable bit map and store each individual pixel in a byte
array, which I send over the relay service.
And that information is sent to either a specific endpoint that I identify or a group
based on which group or user I join with.
And phone two then decodes the byte array into another writable bit map and
paste that image into the history panel.
We'll go a little bit into how a drawing works as well as storing the image. So for
drawing, I record touch data points. So every time you touch the phone it
records a point and it connects a line between those points.
And that's how the drawing works. I use the rendezvous service to store a group
ID with the friendly name which was player two. And I access the group ID from
the cloud each time I click on the player two button.
And then for using storage, as I receive a message, I add the image to the native
storage on the client and I access the storage to display the image.
So this is just a bit of feedback that I got from asking people what they thought of
the user interface. And for the most part they seem to think that the GUI is pretty
intuitive. The main suggestion I received was to work a little bit on the adjoining
group page to have that incorporate, to be more intuitive, to understand and have
maybe better instructions.
So with that, I'm going to go into a demo. If we could switch over to the Wolf
Vision. Okay. So here's the launch screen I'm going to start up the application
on both my devices. When I click "new chat" I'm creating a new endpoint. And
here I'm going to -- when I click on group, you can see a group ID that I just
created. I can either type that ID in or I can use one of the existing group I have.
For sake of time I'm going to use an existing group.
So right here in the red you can't really see it, but it just tells you what your
endpoint ID and your group ID is, just to make sure you're joining with the correct
group that you want to be associated with.
So I'm going to go into the palate on this phone. You can see there's different
colors you can choose from. I'll just choose green.
And I'm going to send maybe some grass over to another phone. Yes. So here I
can also enter text, and when I click the add button it will add it to the canvas.
I'm not actually sending a text message I'm sending a picture with text in it. So
let's just add enter text here. And I'm going to send it across to my other phone.
So this is the main tab. And in the history tab you'll see the pictures. Okay.
Yeah, so I clicked it twice, actually. Or I guess more than twice. And you'll see
that the grassy field has appeared on the other phone.
So something that I'm actually working on is having, being able to edit a picture
that has been sent to you. But right now all you can do is respond by drawing a
new picture and sending that across. So if we can switch back to the -- yes,
thank you. So these are a couple of -- hold on. All right. A couple of challenges
that I faced while creating this. One is the writable bit map. I initially had
absolutely no clue how to convert a UI element into a bit map. So I asked some,
a couple of people for help and I found the writable bit map class that allows you
to converge a UI element into a bit map. And then I copied each individual pixel
into a byte array which I sent across the relay service.
The second issue that I had, not necessarily on this application itself, but on
another application I was making, was to produce a smooth drawing experience.
And so I had -- the other application I created was a Pictionary app. And instead
of sending a completed message over, I would draw and have it update in real
time. And since each call to the relay service is an asynchronous method, it was
creating a number of threads over like a couple hundred. And that was causing
the drawing to lag a bit. So I had to find a way to limit that.
This still needs to be implemented, but right now the data is being stored across
several instances of the application. And it's not -- so I store each message
locally, and I need to clear that queue between instances of the application.
So I'm still working on that. There's a little bit to be improved on the application
itself.
So some ideas for future investment. One thing is to improve the user
experience. And to particularly enhance the adjoined group page to make that a
little more intuitive so people can understand without having me explain it to
them.
So going on to group control. So fully employ the Hawaii rendezvous service.
Right now the only instance as I mentioned that I use it is for player two button
and maybe I had a help guide and various other features that would make it a
little bit easier to use for others.
Additional features that I could add is having the user be able to edit sent
pictures and also be able to save a picture to a photo library. Say someone
sends you this really nice picture that you really, you want to save and maybe
send to all your friends who don't have this application. So you want to be able
to save it to the photo library and send it out to whoever you want.
So these are some uses for the application. Communication in emerging
markets. So help people who are otherwise unable to read and write,
communicate. So especially with the increase of popularity with mobile devices,
if people can just send a picture communicating what they want. So say they
want to tell their parent to go out and buy milk and they don't know how to read
and write they draw a picture of a milk can and a cow and they understand my
son or daughter wants them to go buy milk.
Of course, the main idea is just to have fun. Sending and creating pictures,
sending it to all your friends and maybe playing fun games like tic-tac-toe.
And the last way it could be used is as a teaching tool. Teaching students
different languages they previously have no experience with.
So associating like maybe a picture with a word, and that will allow it to be a little
bit more memorable for them.
Does anybody have any questions? [applause].
>>: Any thought about being able to augment existing pictures so I take a picture
of something with my phone but I need to write on it and say this is the piece I'm
missing here or this is where my house is here and how hard do you think that
would be?
>> Tara Balakrishnan: I haven't worked with pictures that you actually take with
the phone. So I'm not entirely sure how hard it would be. As long as you can
convert it into an image, then it's very easy to add to the canvas and draw on top
of it.
>>: [inaudible].
>> Tara Balakrishnan: No, just any image.
>>: Any?
>> Tara Balakrishnan: Yeah.
>>: Tara, can I ask a quick question. It's interesting [inaudible] you were
mentioning the receiving end. What if I draw something and I say oh I forgot to
add a smiley face, can I add to an existing and then send that one?
>> Tara Balakrishnan: Yes.
>>: And you send the whole thing again, or is there some update mechanism that
it uses?
>> Tara Balakrishnan: So you mean you've already sent the image.
>>: For example, you sent the grass, now I say I want to add a sun or
something ->> Tara Balakrishnan: What happens when you send it the entire image isn't
deleted. So if you do, for example, happen to delete the image on your canvas,
then, again, working with this, if you click an element in the history box, it should
paste it onto the canvas and then you can continue editing it. There's also like an
erase button, if you draw with white on something it will go back to the original
state.
>>: To answer the question, when you resend the whole image?
>> Tara Balakrishnan: Yes.
>>: Okay.
>>: I think it would be difficult to [inaudible] because then you have to make sure
that the state on the other side is ->>: That's right.
>>: Were there any features to the relay or rendezvous service that you're going
to find useful or to add?
>> Tara Balakrishnan: I think one thing would be maybe deleting a message in
the queue. I couldn't find ->>: You send but nobody has read?
>> Tara Balakrishnan: Yeah.
>>: Okay.
>> Tara Balakrishnan: I think that would be useful. I tried doing that once but I
couldn't find ->>: Everybody has to read it. There's no way.
>> Tara Balakrishnan: Yeah.
>>: Have you talked to people, thought about how it might actually be used, what
kind of scenarios other than just sort of, I want to send a picture, the others like it
but something larger happening?
>> Tara Balakrishnan: Yeah. In terms of using it, I've mainly thought about ways
that people would have fun with it. That's the primary purpose that I would use it
for. I haven't really given much thought to how other people would use it, though.
>>: Roughly what file sizes are the bit maps that you're sending?
>> Tara Balakrishnan: They're around, I think -- well, the array I send is about
900,000 bytes.
>>: [inaudible].
>> Tara Balakrishnan: [inaudible].
>>: I think to be intelligent about how you compress and send it, you just send
the points ->> Tara Balakrishnan: That's something I've considered instead of sending the
entire image is just sending the points that you draw.
>>: That you draw.
>>: [inaudible].
>>: Even if it is a new image if you had access to some nice compression ->>: Right after that now you shouldn't be able to find a compression.
>> Tara Balakrishnan: Right now all we do is send the entire image over. But,
yeah, I've thought about just if the file size gets too big just sending the points
maybe that you have drawn and redrawing it on the other end.
>>: Just for fun, can I suggest something to you. There's this thing called
[inaudible] where you look at the pixels. If it didn't change much, you may want
to have fun just playing with implementing that yourself. Like I want to send a
pixel, that's the value of another one, send a small code, didn't change, where
say put ten black pixels here or put ten white pixels here, something like that a
little code like that. It's amazing how a simple code like that can be reduced quite
a bit for the kinds of drawings that you have in mind.
>> Tara Balakrishnan: That's actually why you see a little bit of a lag. It's not like
it's sent instantaneously, because the message is so big.
>>: How the groups, once the conversation goes, someone joins the group or
leave the group?
>> Tara Balakrishnan: Can I -- yes, so to stop receiving any messages, you
have to delete your endpoint from the group. Or just delete the endpoint. And by
closing the application, you're no longer receiving any messages, because when
you close the application I delete your endpoint.
>>: But a new endpoint can join this group ->> Tara Balakrishnan: A new endpoint can join the same group. They won't
receive any messages in the past; they'll only receive messages from the
moment in which you join a group.
>>: How do you join a group?
>> Tara Balakrishnan: So what I have on the screen is the -- you can type in a
group ID and join the group or else I'm going to implement a rendezvous a bit
more. And so make it a lot easier for you tell your friend name and have them
type in that name and join that group instead of typing a long list of numbers.
So what I do is every time I run the application I'm actually creating a new group
in case you want to create a new chat experience.
>>: Kind of like Google hang out, that could be the only one in the group but then
other people ->> Tara Balakrishnan: Yeah.
>>: What about getting this in the marketplace? Is that something you want to
do?
>> Tara Balakrishnan: Yeah, sure. That would be nice.
>>: If we can make it happen.
>>: Testing.
[laughter]
>>: I would be interested in hearing what the group thinks. It's functional
[inaudible], some really strong legs under it. But ->>: I guess the plans would be for beta, right, that would be amazing, transfer it.
>>: Right now the services are offered as a research service without cost. We're
covering the costs on that. There's no guarantee in service.
>>: But in terms of ->>: It would just work.
>>: We don't want people going crazy because ->>: Yeah.
>> Tara Balakrishnan: I was talking to a couple of people yesterday and they
said they're probably going to limit the size of messages that you can send. But
right now everything's free, though.
>>: So does Hawaii put a limitation on how many endpoints can be assigned to a
particular group so that it would be a functional limitation on how many people
could join a group?
>> Tara Balakrishnan: Any number of people can ->>: [inaudible].
>>: Don't go ride a bike.
[laughter]
>>: That would be pretty unique for anybody to develop
[laughter]
>>: But I think these are good questions.
>>: These are learnings coming out from things like this which are [inaudible].
>>: I've downloaded work with younger kids, and I think you'd be really interested
in this, particularly if you could do sort of the co-editing where I can modify a
picture you sent, a feature that you talked about.
One of the challenges probably is those younger kids, although they're starting to
get phones, they typically don't have Windows Phones. But I can see that
demographic being really interested in it. I had some videos of them doing
realtime drawings, share drawing, a face of a similar to the flavor to what you're
talking about here.
>>: The picture app didn't show.
>> Tara Balakrishnan: That's a real time drawing.
>>: That's right up that alley, where while you're drawing it the real time the
person sees it, no reverse drawing.
>>: You try to guess ->> Tara Balakrishnan: If both people draw, you can actually have like
collaborative drawing. So when you draw on your phone it appears on the other
phone. But simultaneously someone else is drawing on their phone and it
appears on your phone.
>>: So you use Hawaii for that?
>> Tara Balakrishnan: Yes, I used service for that as well.
>>: One of the ideas put forward for that project was almost a Turing test concept
to see if you could fake -- if the person on the other side was a robot or real
[inaudible] see who is drawing the picture. I thought that was a cool idea. That
came from the [inaudible].
>>: The other thing that we saw kids do, again, this would be a little more
complex feature, but if you utilized the camera on the phone, then you can sort of
also project an image that they can -- that the kids did in our study they went
under the cameras one kid's face was there and the other one drew features on
the face where they put their arm there and drew cats and the other person drew
tattoos.
>> Tara Balakrishnan: There definitely would be a lot of fun to play with. So is
there any other questions? Thank you all for coming.
>>: With the Pictionary app, does it just send basically the lines that have been
drawn?
>> Tara Balakrishnan: Yes, it sends -- it sends some different information.
Sends the line or sends actually two points. It sends the color or thickness, stuff
like that, and then on the other end it redraws the line based on the two points
that I send. Great, thank you all.
[applause]