>> Jaime Puente: Okay, good morning everyone. Thanks for coming and joining us via online. With
great pleasure I would like to introduce Betsy DiSalvo and Blair MacIntyre both Professors in the School
of Interactive Computing, Georgia Tech. Betsy leads the Culture and Technology Lab, which focuses on
research studying culture values and how they impact technology use, learning a production. Betsy
received a PhD in Human Center Computer from Georgia Tech. Blair leads the Mentoring Environments
Lab which focuses understanding the potential of augmenting reality technology. Blair received a PhD
from Columbia University.
I think Betsy will start presenting, so please, and then Blair. Please join me welcoming Betsy and Blair.
[applause]
>> Betsy DiSalvo: Hi thank you. We’re going to be talking a little about how we’ve been working to
integrate Studio Pedagogy in Computer Science classes. Particularly with Projection AR is a project that
we’re going to be addressing here.
But I want to give you some background on myself and on this project. Well, I have a PhD in Computing.
My undergraduate degree is in Studio Arts with an emphasis in Ceramics. My baseline for kind of
understanding a learning experience or an expert learning experience comes from the studio
experience. Where everybody would have objects out all the time in different phases of the process
that they were working on.
This public work space really encouraged student to talk to each other. To ask questions about what
each other was doing. Also there was a lot of practices within the studio that talked about sort casual
kind of pin-up interactions where students would talk about work in progress or formal critique process.
Sometimes you know when we’re presenting this we talk about it as being this, it sounds very warm and
fuzzy. But in reality studio arts practice or design based practice these critiques in this kind of continual
critical thinking and reflection on your work in a public space can be pretty tough. It’s actually pretty
tough to take. Students learn how to take criticism. They learn how to deliver criticism. They also
develop a pretty thick skin about it.
I think these are all qualities that we might want to see a little bit more inside of our Computer Science
classroom. Just to give you some idea about, I think this is an interesting picture because it shows you a
little bit about how these studios work. Sure they have pieces out, everybody’s head down working on
what they have.
But if you notice there’s mirrors behind every wheel and one of the reasons that you do that inside of a
ceramic studio is that so you can see exactly the techniques someone is using from various angles, right.
This is you know something we don’t think about in studio arts is how much we focus on technique in
addition to, and technical skills in addition to sort of the theoretical ideals, or the artistic vision, right.
>>: [inaudible]
>> Betsy DiSalvo: What?
>>: [indiscernible]
>>: Once you…
>> Betsy DiSalvo: That could be because he’s going to go do glaze mixing after this. Or he’s going to;
actually they look dark so he’s probably been working with a Raku fire. They work with fire a lot. I like
fire a lot too, so thus I was a ceramics major.
Okay, alright, so you know a computer science classroom. In a typical computer science classroom is
actually quite different than this. While inside computer science classrooms we often use project based
work, or design based learning interventions. One of the ways that this is usually implemented inside of
a computer science classroom is that students work solo throughout the semester. Then at the big
midterm or the final they present their work. That’s the only time they get this sort of interactive
experience within the classroom.
You know this isn’t actually I think this is one of the nicer computer science classrooms I could get a
picture of, quite honestly. It’s pretty colorful and nice. I’m not trying to give you the worst case
scenario here. But one of the things that does happen often inside of computer science classrooms at
the undergraduate level is that we have people that feel like outsiders.
This is actually a current Microsoft employee who is an African American student at Georgia Tech,
several years ago that I worked with. He told me, “Me and some of my black friends were talking about
the guys in CS. Some of them have been programming since they were eight. We can’t compete with
that. Now, the only thing that I have been doing since I was eight is playing basketball. I would own
them on the court. I mean it wouldn’t be fair, they just would stand there and I would dominate. It is
sort of like that in Computer Science”.
His first year in computer science he felt like this all the time, like everyone was like dominating in the
classroom. He was so far behind that he would never catch up. Because so many of the students
coming into computer science have had experience, previous experience programming, taking robotics
camps, other things like that. They come into the program knowing quite a bit and being quite
confident in their skills.
One of the things that this has done is it set up defensive environments often inside of computer
science classrooms. The work of Jane Margolis and Lecia Barker really has outlined some of these
problems. It means that students will often times raise their hand inside of a computer science
classroom to show off what they know. They rarely raise their hands to ask a question because they’re
trying to learn something, right.
The standard there is about kind of showing off and you know presenting sort of this really self
confident attitude inside the classroom. What it does is it sets up a classroom that’s not welcoming,
particularly, not welcoming to people who don’t have this previous experience.
As I think you all probably know the people who you don’t have this previous experience are often times
females and people from underrepresented minority groups in the US. It makes them feel like outsiders
like the quote I showed you before.
It’s also frequently taught as a solitary activity where there’s very little collaboration and reasons for this
are multiple. Teachers do this because they’re afraid of cheating. Sometimes they allow for pair
programming but that’s pretty limited inside CS classrooms, although it’s shown to be quite effective.
But it’s a solitary effort most of the time because they’re afraid of cheating. It’s really difficult to do
assessment when you have multiple students working on a project at the same time. This actually sets
up a whole environment where there’s a fear of collaboration.
It also has setup an environment where students are in competition with each other for grades, right. If
you help somebody else, that might mean that you helped them get a better grade which is going to
hurt your grade in the long run, right. That’s another reason why collaboration is kind of shut down
within the CS classroom.
>>: [indiscernible] to other engineering?
>> Betsy DiSalvo: Oh, it’s come in to other engineering programs. Actually that thing about preparatory
privilege is more common in computer science than in engineering. Because not everybody walks into
their first engineering classes an undergraduate with like all this engineering experience and kind of
shows off in that way. But in CS classes that happens more frequently, yeah, but the other things
totally.
[laughter]
Although sometimes engineering’s a little bit more conducive for project work. It depends on the type
of engineering we’re talking about.
I’ve been working for a few years to bring some of those studio practices that we talk about into the
computer science classroom. One of the ways I’ve been doing this I’ve been partnering with Gregory
Abowd who teaches mobile and ubiquitous computing at Georgia Tech, as well as HCI, and also a
graduate level prototyping class.
One of the ways that we’ve been doing this is integrating maker base activities into the classroom. The
affordances of this are great. Because these are a visually kind of interesting projects students are
working on. That everybody can see what everyone is doing in the classroom. It really encourages the
students to both share materials and tools, and skills, ask questions from each other. They can get a
good sense where everybody in the class is.
Also one of the things that we’ve seen here and you can see in a studio too, is that students would often
time fail with what they’re doing. Then have to iterate and try again. That productive failure is
something that other people in the classroom were able to see. They were more comfortable trying
new things and failing publically, right, rather than holding everything back until the very last second.
Then showing it when it’s too late to actually improve their learning outcomes in any way.
Some of the things that we’ve been focusing on this is some theoretical constructs from the learning
sciences. Specifically, I’ve been looking at the community of learners this establishes. This is more than
just a classroom of students working together. It’s really about thinking about a community of learners
means that when I walk into a classroom I can act as a student, a teacher, a collaborator, someone who
brings different people together in terms of networking. Or maybe I’m that person within a community
of learners that helps you know you take that thing that you’ve just learned in class and match it to
something in the outside world. It’s taking on a variety of different roles that a learner or a community
of learners takes on rather than just being the student who is absorbing the knowledge within the
classroom.
We have noticed a lot of this happening within these maker oriented learning environments. We think
that it’s very encouraging. The students have actually reported back to us that they like the fact that
these classes aren’t just them talking to the other people in their project group. But it’s also them
talking to people across project groups. That’s been beneficial for them to learn a greater breadth of
information in the class.
Another thing that we’ve been looking at is specifically how students are able to reflect on their own
learning process and reflect upon their own projects. To think about this in a metacognitive way, what
have I learned? What else do I need to learn, like what am I missing here? That metacognitive process
is kind of a, you know sort of like the building block for all learning that happens. It’s almost impossible
to learn something unless you can recognize the fact that you need to learn it, right.
The reflection is taking place through a lot of our critique sessions, having these sort of iterative
critiques as they go on. Both casual and a little bit more formal ones has been very useful for the
students to start to learn to reflect on a regular basis, in their work and to see this in other students as
well.
I will say that the critiques have been a real challenge. I’m going to talk about that a little bit later on as
we get into the AR Studio environment, as well. Then kind of a last piece and I don’t have as much data
on this. We just did our comparative study, I, last week during finals with a standard HCI class, a
standard like traditionally taught HCI class. We had done it last, in the fall semester with our more
experimental class with the maker based learning.
What we’ve done is give them both the same design question. What we’re looking at is to see how
much they transfer in that they’ve learned from other classes into this class, because part of our theory
here is by exposing them to these maker activities. This greater variety of, sort of, ways to integrate
ideals in computing into their own projects they’re going to start drawing on outside resources more.
Start being able to make those unusual connections maybe increase innovation perhaps might be a way
to put it.
I glanced at them, like I rifled through all the data of course. I think that we’re seeing that there is a
greater transfer from outside. But one of the things that we may have lost with this is some of the
central principles in terms of teaching HCI may have been lost within the maker oriented learning. That
means that we have to recalibrate what we’re doing within that classroom.
But just to, I think it’s good to note that kind of thing to say this isn’t a silver bullet, right. For every gain
there may be a loss in relationship to that. You have to make choices.
As I mentioned there’s two practices from the studio model that we’ve been trying to integrate. One of
them is Pin-ups. We think of Pin-ups as very casual sort of interactions that can happen on a daily basis.
It’s that stuff that’s just you know out that we can talk about while it’s in progress.
The other is Meet-ups. This is students who are working together in the studio. They get to look at each
other’s work and it’s the casual Meet-up that happens. It’s not something that is pushed by the faculty
members. It’s something that the students are doing on their own. These are the two things that we’re
sort of looking for specifically within these spaces, trying to integrate into the spaces.
You know, but unfortunately maker based activities don’t work for a whole lot of computer science
classes, right. There’s a lot of things that this doesn’t apply to. Because it just simply isn’t possible to
teach the concepts we want with these physical prototyping.
That kind of brings us to the problem. How can we make visible the process of learning within
computation? How can we present this within other classrooms in our computer science curriculum?
Then how can we expose the hidden digital work that the students are doing, so that they can start
having these types of interaction?
I’m going to hand it over to Blair for a minute.
>> Blair MacIntyre: Yeah, thank you. I just want to talk briefly about the technology setup, what we’ve
done, and how we got there. Then I’m going to hand it back over to Betsy to talk about the sort of
results of this first semester.
As Jaime introduced at the beginning, I’m Blair MacIntyre. I’ve been doing Augmented Reality for a long
time. But notable for this project I spent a summer at MSR working with Andy and Banko, and the
RoomAlive Team when they did this project. This is an image from one of the papers.
When I returned to Georgia Tech and this stuff got published I was allowed to talk about it. I gave a talk
on, short talk on the work. Mark Guzdial who’s a professor at the Georgia Tech in CS education sort of
made this connection between what Betsy was trying to do. What Betsy was doing with the maker stuff
and her goals with Stewart Pedagogy and a potential use of augmented reality. To solve this problem
that Betsy just highlighted, this issue of how do we expose hidden digital work in a learning activity?
The three of us started talking. But before I go on, so if you’re not, for those not familiar with
RoomAlive this is the video from the WIST paper. This might be the one on the MSR website. The idea
with RoomAlive was to use projection AR. To use, essentially take a room, use a bunch of connect
sensors so that we can map out both the static physical structure. But also the action and interaction in
the space and be able to, and then use projectors, calibrate the whole thing. So that you can project
everywhere, tell what people are doing, and create these kind of experiences. This is part of a long
ongoing thread of research that folks here have been doing.
For us the good, the key part was that they open sourced a bunch of the underlying technology. So that
we could take advantage of it and get our students to try to build it. We setup a space similar to the one
that they illustrate in this video. We have a bunch of projectors on the ceiling and have calibrated the
space so that we can project on all, eventually all the surfaces.
Okay, so, to sort of rephrase the questions that Betsy asked at the end of when she stopped talking. The
question that we really wanted to pose with this project was. Can we create sort of a studio learning
experience for STEM in general? But for, in our case computer, intro computer science, such that we
can expose the work in progress of the student, so just as Betsy described ceramics.
The idea, that students can go and see what people are doing not just the finished piece but watch how
they are working. What they’re doing when there’s a failure which can be kind of a dramatic event if it
sort of collapses in on itself. Either offer, I don’t know, it could be, could offer assistance, could learn
from things like, oh that happened to me, how did you fix it you know, and so on. Can we take that and
create that in a computer science class?
Betsy will talk more about the class. We, but essentially we wanted to apply it to an Intro Computer
Science class. That we have a tech that’s very visual where we, called Media Computation where we use
imagery, use very visual media to teach computer science concepts because we thought it would map
well.
Okay, so this idea of using AR for, using augmented reality to display information around us in our
periphery that might be useful is not new. In fact, it’s kind of the central idea behind a lot of what
makes augmented reality exciting to people. This idea of projecting information around the rooms is
also not new. We actually had a project. I pulled one of the projects we’d done years ago and Georgia
Tech, at the turn of the century, love saying that. Where what we wanted to do is project information
all around that had been gleaned from someone’s work on a computer.
What you see in this picture are sort of image montages that represent different tasks people have been
working on that were snapped automatically from the desktop as they were working. The long range
vision of this project was that by showing you the past work that you’ve done. We could both help you
remain, regain state when you wanted to resume a task. Be reminded of the task that might not be
finished, kind of like the messy desk, which is why we have these sort of messy things. You could do
things like annotate them and integrate other stuff.
This and other projects over the years, people have envisioned this idea of having your hidden digital
world visible. One of the things, that wasn’t so common which I thought was really interesting with the
idea of applying RoomAlive to this educational activity. Is the sort of passive nature of the augmented
reality, right? I think if you talk to a bunch of people about augmented reality education they’re mostly
thinking about I want to use augmented reality to teach you how to do something. Right, I’m going to
use it for Math, or I’m going to use it to teach you how to repair something.
Here the activities that the students are engaged in are completely divorced from the augmented reality
in the focal task, right. They’re programming. They’re manipulating pictures. The AR’s being used to
create this pedagogical environment, this pedagogically beneficial environment that we hope will
change and improve the educational environment.
We used RoomAlive as the basis, the RoomAlive software as the basis for being able to display
everywhere. We’re not yet leveraging some of the more advanced capabilities of the system which I’ll
refer to later. Then we wanted, but to do this we had to be able to capture the students work without
them doing anything. We didn’t want the, as soon as, if we asked the student to select a picture and
share it. That would have defeated the whole purpose because then they would have no motivation to
share it. They would, we wouldn’t be seeing this work in progress.
We took advantage, the system, the class uses Python so we took advantage of this web based Python
programming environment from Virginia Tech called Pythy. Pythy runs on a server. We setup a Pythy
server for the lab, for this project and so we modified it. Every time in this Python environment an
image gets shown we write it out to the database, grab a copy of the image, grab a copy of the code,
save all kinds of other metadata that we might need. Then the software that’s driving the projectors for
RoomAlive is just watching this database using a remote SQL connection. Every time a new entry
appears for one of the students in the class we just grab a copy of that image which is accessible on the
web server and pop it up.
Right, so relatively simple. But it allowed us to do this sort of very passive background environment.
The technical, physical setup was five projectors, ten eighty P high resolution projectors, relatively wide
field of view setup in this space. This is one, an image of the reconstruction that was created from the
RoomAlive calibration. We have a relatively large room. There’s desks in the center that aren’t showing
up here because we didn’t calibrate down that low. Three walls, the door is over here. These are
actually windows that we covered up because obviously that would create, the light would create
problems. We actually then could project there as well.
Here’s another view of the same space. In this space we can project here, here, and here. Then actually
kind of over here there’s a normal projector that the instructor uses to project their slides. In the, you’ll
see some images of the space. You’ll see students sitting largely here looking that way because they’re
looking when the instructors talking at them and seeing sort of in their periphery a lot of this other
content.
Okay the content is rendered relatively, this a snapshot of sort of just the virtual content in the three D
world. Each student has a space with their most recent image, when it was updated. Now, we’re just
using their initials. At the time when we first did this we used whatever name they registered in the
Pythy server with, so everybody has accounts.
Using that we get something like this, right, students sitting around working, and content spread around
the walls behind them, very large. So they can really see what’s going on. When there’s changes,
they’re very visible because it changes immediately. This let us create this environment where the
students are working and seeing the work in progress constantly around them in the lab.
>>: [inaudible], just the one next to me that I converse with. I actually don’t see his work.
>> Blair MacIntyre: Well, so we, his work maybe anywhere in the room.
>>: I see.
>> Blair MacIntyre: Right, so there’s a lot of technical questions. We had lots of fantasies about things
we would want to do when we started this work, right. You know I presented the RoomAlive stuff which
was very in some ways sophisticated. In the video you see people running around and touching. It’s like
we should be able to know where people are and all this.
We had ideas early on of should this person’s work be behind them, right? Or should we maybe put it
somewhere on the desk? How can we tell? Because we wanted to get it working and be robust we
didn’t want to rely on some form of tracking of the twelve to fifteen students at once. Knowing which
one was which and all this. These are all ideas that we want to explore which we’ll return to at the end.
I’m going to hand it back to Betsy to talk about the results.
>> Betsy DiSalvo: Yeah, so what we did this semester is looked at two different classroom conditions.
One is the traditional model with an instructor projecting with a normal projector in a classroom.
Students working at their own laptops as you would have normally done. The other one is a studio
model which Blair just described using the AR technology. We both have the instructor and the students
projecting.
We did observations and interviews this spring in both of these conditions. These were in recitations for
our Undergraduate Media Computation class. This is an entry level CS course for non majors.
Traditionally it’s taught as a three hundred person class and a lecture. Then once a week they meet for
recitation or a lab session.
The first condition, the studio condition that we were looking at there was twelve students, two
instructors. One of those instructors was actually Mark Guzdial who wrote the book, quite literally on
media computation and was a person who kind of invented this whole thing, and came up with the
entire class. That was really well taught.
The other condition was the traditional recitation with fifty students, two instructors who are both
undergraduate TA’s. I’m telling you this because I am not comparing these two things. I know they are
not equal, right. We’re looking at this as much more of a qualitative discovery process to understand
what the differences are in these kinds of classes.
>>: What is the subject of the courses is something that has more visuals…
>> Betsy DiSalvo: Yeah, so media computation is about using computation to manipulate media. About
half of that course is based off visual media. Some of it is based off of sound. What are some other
aspects of it?
>> Blair MacIntyre: Web…
>> Betsy DiSalvo: Web type stuff, yeah.
>> Blair MacIntyre: [indiscernible]…
>> Betsy DiSalvo: There’s a lot of visual elements to the class that you can project. That’s actually an
interesting question. We’ll talk a little bit about that later like what can you project and what classes?
Yeah, we did observations and interviews with the students. Actually, our semester ended last week, at
the end of last week, so actually there might be finals today, I don’t know. We’re not done with our
interviews. We certainly haven’t done a deep analysis of this data. But we have some preliminary
findings and it’s really promising.
First of all the thing that we have looked at quite closely is our observation notes. Within the studio
conditions students notice and commented on their peers work. At least once, usually actually multiple
times in every single one of the classes when this was happening. In contrast in the traditional model
the students never commented on each other’s work. Unless, the instructor specifically asked them to
work together in peer groups or present something that they found out, unless they were prompted to.
There was none of this spontaneous interaction happening in the traditional model. This is one of the
things that we were looking for.
We also noticed the students gained inspiration from each other. This picture of Donald Trump well this
is a kind of amusing anecdote. One of the students manipulated this to turn him green. I think he
eventually put little ears on him so he looked like Shrek. The other students thought this was very
amusing and wanted to do the same thing with other images. From that they rift off each other’s work.
They basically remixed each other’s computational code. They were able to come up with different
ideas. This is exactly you kind of want to see happening in a studio environment as well, this inspiration.
One of the other things that we saw is that students sought help based upon the projections that were
in the room. This is a great quote from one of our interviews. The student liked it a lot. But what she
also noticed is that when people had completed the task and she wasn’t done, or she was struggling
with something. She saw that and then she was able to ask those people for help, right. Then when she
saw one of her friends perhaps struggling, not completing the task she was able to reach out to them
and offer help to those students. That rarely actually happens in computer science classrooms. This we
felt was a very encouraging sort of behavior happening in the classroom, as well.
Another thing that we didn’t anticipate make sense in retrospect is a few students have told us that they
could measure their progress within this classroom. If they could see that everybody else had
completed the next task right within the lab, yeah go ahead.
>>: Question on the visuals here. The white rectangle what is that?
>> Blair MacIntyre: It’s a window…
>> Betsy DiSalvo: That’s a window.
>>: Oh.
>> Blair MacIntyre: [indiscernible]
>> Betsy DiSalvo: Yeah, sorry.
>>: [indiscernible]
>> Betsy DiSalvo: Yeah.
>>: Physical you are.
>> Betsy DiSalvo: Yeah, so they could see when they were falling behind. Then they could actually reach
out to the TA or the instructor to ask for help, right. That’s not something that happens often inside of
CS classroom because you’re working solo. You don’t see how far behind you are from the rest of the
class or how far ahead you are.
>>: I’d like…
>>: I have a question.
>> Betsy DiSalvo: Sure.
>>: Are you able to see activity live on each one of these panels, is that how it works…
>> Betsy DiSalvo: Every time they run their program…
>>: Monitor like you know you see everyone’s on this one. Or like…
>> Blair MacIntyre: You know it’s one per person.
>> Betsy DiSalvo: One per person.
>>: Right, but are you also showing their update in real time as they…
>> Blair MacIntyre: Yeah.
>>: Yeah, okay got it.
>> Blair MacIntyre: [indiscernible]
>> Betsy DiSalvo: Every time they run their code it updates the image, yeah.
>>: Okay and are you also able to just focus on one? Could you select one and just everyone looks at it
or?
>> Betsy DiSalvo: I would really like to do that.
>>: Okay.
>> Betsy DiSalvo: Yes, we cannot do that yet.
>>: Yeah.
>> Betsy DiSalvo: But this is a class of twelve so we were able to do that in some ways just by walking
over to it and say look at this, right. That’s a lot of what happened. Even if stuff might be projected
behind me if somebody else noticed it I’d just turn around. You know, so that was okay in this particular
environment, right.
>>: It’s like a regular class, true that this has much more visibility but do you have their, also if someone
would have put in the Trump image on his laptop with his neighbors or someone behind you will look at
it and say, oh, that’s cool and so on.
>> Betsy DiSalvo: It might, yeah and that might, we didn’t observe it in the traditional classrooms that
we saw in this particular case. I can certainly see that happening but it would be a small group. It’d
actually be kind of discouraged within the classroom to have that kind of talk going on while the
professor is teaching, right. This is an environment that encourages it rather than discourages it. That’s
part of setting a precedent.
>>: Although even in this environment if a professor right now talks would you see the background
changes and the professor knows that instead of listening to him someone is doing something?
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: Absolutely.
>> Betsy DiSalvo: Yeah.
>>: Okay.
>> Betsy DiSalvo: That would happen too. Knowing what your students are doing as a professor is
actually a good thing, not a bad thing.
[laughter]
Yeah, so Mark who was our instructor also found the whole process really helpful for him. He’s an
expert teacher. He’s been teaching for many years in this classroom for many years. But some of the
things that he felt were unique about this is that it was a chance for him to let the class be student lead
rather than instructor lead. Engaging students in active learning inside the classroom is actually
something almost all institutions are trying to do more of right now.
But it’s really difficult for professors to do that. They don’t have much experience with active learning.
They’re use to standing on the stage and presenting their slides and not having students interact with
each other during the classroom. In some ways this acts as a forcing function so the students start
leading part of the learning process which you know could be very beneficial.
Implications for this, is that the teachers may need to explicitly provide permission to the students that
they can interact. These students have been well trained to be good students. They know not to talk to
their neighbor or interrupt during class. We need to let them have the permission to do that. You need
to give them explicit instructions to them that that’s available. Maybe set a precedent for it as well,
model that behavior.
Training is really needed for the teachers to try to integrate this into their classroom and into their
classroom practices. The TAs taught a couple of the sessions within the studio classroom when Mark
was traveling. The things that we observed inside that classroom where the TAs actually did not point at
anybody’s images that were up on the wall. Or engage the students in anyway. The students were still
doing it. They were doing it despite the TA trying to present their slides and only present their slides.
But the TA probably needed to be given some guidance and instruction on how to lead a class with this
technology. That’s another challenge that we kind of face.
Finally, students need help to learn how to critique. They didn’t really know how to talk about each
other’s work. Computer science instructors aren’t necessarily well versed on how to lead a critique
themselves. This is something that we’re struggling with in the maker oriented learning, as well.
There’s a few techniques that we’re use to address this. It is just a matter of training and getting people
use to this way of looking at each other’s work critically, commenting on it in a productive way, and sort
of making that a cycle to help each other both learn and iterate on their projects so that they’re better.
>>: I have a question [indiscernible].
>> Betsy DiSalvo: Sure.
>>: I mean this is kind of a generalization. But it’s sort of inferred that a lot of folks who are in CS are
very introverted by nature. They don’t want to share. Or they don’t want to get up in front of the class.
>> Blair MacIntyre: Yeah.
>>: The thing about the critique model is that you’re very exposed to that, right. You’re like oh, my
gosh, everyone’s looking at me. Have you noticed any interesting transitions from sort of typical
introvert behavior to more sort of like getting comfortable, having your work exposed in real time,
getting up and talking about have you seen any interesting developments around that?
>> Betsy DiSalvo: The students are I have more experience with this in the maker oriented learning
environments. The students are pretty comfortable working in their teams. Then getting up and
presenting their work. There’s occasionally a student in the class that’s kind of terrified of presenting.
That’s actually in all classes I think, yeah.
>>: I actually [indiscernible] to death. What if someone wants to explore new thing but is afraid that in
the middle or maybe that will not be successful it will be presented to everybody?
>> Betsy DiSalvo: Well, this is one of the things that the studio model actually encourages quite a lot.
We call it productive failure that you’re allowed to try new things and fail publicly because everybody’s
doing it. You’ve seen most of the class fail publicly in some way. It’s okay for you to go ahead and do
that, as well.
Often times when students are presenting in their computer science classrooms, project based classes.
They only want to present a success. They won’t do something risky, right. It may encourage sort of
risk, trying new things and risky behavior, learning new sorts of activities.
>> Blair MacIntyre: The other half of your question and the reason you had some stuff in the CS class is
this is non-major class.
>>: Oh, okay.
>> Blair MacIntyre: None of the students in that class are CS majors.
>>: Oh, okay.
>> Blair MacIntyre: Some of them will be in the future.
>>: Right, okay, oh, okay.
>> Betsy DiSalvo: We do have, gained a few students from this [indiscernible] media class into CS.
>>: Oh, okay.
>>: I guess the question I had also is how much of it is class wide critique where you’re standing up
talking about someone versus the individuals going hey how did you do that? Hey, I you know that
more individual which could be considered critique. But that helping critique, so I’m wondering is there
a big fifty or is it more…
>> Betsy DiSalvo: Yeah, so the big, kind of formal, stand up and present your work, get a critique is
already happening in a lot of computer science classrooms. What we’re trying to implement is a more
casual, we call them Pin-ups.
>>: Yeah.
>> Betsy DiSalvo: It might be a small subset of the class. Or the whole class that kind of does this on a
more you know intermediate basis rather than at the final project. But then the other thing they
identify is we call them Meet-ups which is when the students kind of facilitate this for themselves.
That’s that very casual interaction.
>>: My question is kind of what you were talking about [indiscernible]. If the, it’s more of an issue on
the introverts on the large class versus the two to three type you know it’s a small group, because you’re
more apt to ask somebody else for help. Or, hey do you solve this in a, work in a safe zone, rather than
that.
>> Betsy DiSalvo: Yeah, so and I think that that actually doesn’t happen.
>>: Yeah.
>> Betsy DiSalvo: That students are like hey, how did you do that? Then they talk to each other. That
might address some of those introvert issues. Yeah, but we haven’t had a huge problem with the
introvert issue. There’s those some students but we haven’t had a huge problem in the maker based
learning environments which I was a little bit surprised by frankly. They have a hard time doing the
critiques because either the students tend to be just super aggressive and like difficult. Or they won’t
say anything bad at all, right. Only I think it’s cute you know what I mean. We need to help them be
more critical in that, right.
>>: I would say that specifically in this media.
>> Betsy DiSalvo: Yeah.
>>: There’s things that when you show them at one point they have a great impact. But if you see them
in the middle you will spoil the surprise.
>> Blair MacIntyre: What do you mean?
>>: I might do a poster like that where I, when you finish you will see what I meant. It will have a strong
impact. But if you see it in the middle on the list you might guess what’s going to happen and…
>> Betsy DiSalvo: You think it’ll be a spoiler.
>>: Or in the worst side you might be too much critical because you don’t know where it’s going.
>> Betsy DiSalvo: I think that that is typically the way people in computer science think about this. It’s
very different than the way that people in the maybe studio arts or design think about this process. I
understand that but I’m trying to actually change that, right.
>>: Yeah, no…
[laughter]
You had an example that says I’m drawing a…
>> Betsy DiSalvo: Yeah.
>>: A cartoon or charactures…
>> Betsy DiSalvo: Yeah, so it’s kind of like a…
>>: It works when it’s finished.
>> Betsy DiSalvo: Yeah, well let’s…
>>: It won’t work in the middle.
>> Betsy DiSalvo: If you think about it in terms of writing a novel or a story, right.
>>: Yeah.
>> Betsy DiSalvo: People who write novels or stories workshop them frequently where you’re
presenting a chapter or outlines, or subsets of that story. You’re workshopping it with other people who
are giving you iterative feedback on it. It helps you think more critically about what you’re writing, may
change the direction for it. But even though you have this big idea of what this is going to be at the end.
I guess what we’re proposing with this work is this kind of workshopping or iterative feedback is going to
be more productive for you. Than you having your brilliant ideal presented at the end of the semester,
right.
>>: Yeah.
>> Betsy DiSalvo: Because the goal is for you to learn something not to be the shining star in the
classroom, right.
>>: Right.
>> Betsy DiSalvo: Those maybe different goals, right. Go ahead.
>>: I think one of the very interesting things is kind of almost the real time nature of the presentation
which is not necessarily. It might be the case for the physical things like if you’re doing clay or pottery.
But if they’re preparing their posters or something like that in more traditional sense it might not be so
immediate, right. You would have these Pin-ups but again on a daily cadence rather than an instant
cadence.
I’m wondering what your thoughts are about would observe similar needs or would there be a similar
benefit if you literally for maker spaces just had a camera above every station? Just like real time share
that or do you just like physically being able to glance over or see what they’re doing is enough?
Because in some sense could you apply the same technique regardless of the fact that this is you know
Python and scripts, and image manipulations in the background. You know whatever we’re doing we
are writing but you’re just magnifying my writing and exposing everything on the large. You know for
everything in real time like this concept of real time sharing to the public is that something that is a core
benefit here? I don’t know like what are your thoughts?
>> Betsy DiSalvo: I totally, the other future work.
[laughter]
Other courses and where else could we apply this in other group environments like I had never thought
about a writing environment for this. But maybe that would be, I mean poetry writing I actually think if
we were posting everything on the wall as we’re writing poetry in real time that could be fascinating.
Actually, maybe perhaps quite beneficial, right, and I would think so [indiscernible].
>>: Since its concise where would you’re doing non-visual…
>> Betsy DiSalvo: Yeah, no, and I mean that’s…
>>: That’s actually…
>> Betsy DiSalvo: This isn’t going to be the answer for every classroom, right. It’s going to be the
answer for certain classrooms just like the physical computing is the answer in some classrooms, right.
>>: I think I can do it in Math too.
>> Betsy DiSalvo: Yeah, you could totally do it in math, yeah.
>>: It doesn’t matter. But the thing that it reminds me of is that you’re always intrigued by is a bulletin
board where when you walk by and something will catch eye and you’ll dig in more deep on the text,
right. It’s not that you always have to have a large visual. It may just be there’s some [indiscernible],
some conscious picks up a few words. Then you’re going to read the rest of it. There’s something
about, I actually like this where it’s on the wall. It could, it would be great to actually see it with text.
You know could you imagine it when you guys do paper writing or something else like that. Doing it that
way where people could just quickly kind of give impact on it.
>>: You can extract key sentences…
>>: Yeah.
>> Betsy DiSalvo: Yeah, its stuff that exists in the periphery like in studio space you have all sorts of stuff
on your shelf, or on your easel, or whatever it is. Even drawing spread around your table. It’s that
periphery that’s happening other people can walk by and see it. Hopefully will be that moment of
serendipity where like what I’m doing and what you are doing kind of overlaps and makes a moment.
>>: [indiscernible] courses in those. But it occurred to me that Hackathons would be a great place
because it’s a limited time. You know short amount. Maybe the data science ones or those have ones
where you’re doing it over twenty-four hours, right.
>> Betsy DiSalvo: That could be cool.
>>: Getting the insight and the…
>>: Right.
>> Betsy DiSalvo: We can actually do that. They have hackathons in our building sometimes. We could
put them in that one group in that room and see what…
>>: We should do that.
>>: [indiscernible]
>>: That’s actually better than the slide that you showed for the make because for the make it seems
like everybody was doing the same project.
>> Betsy DiSalvo: Oh…
>>: At least on that slide.
>> Betsy DiSalvo: It was just a picture I got.
[laughter]
>>: Yeah.
>> Betsy DiSalvo: It was a nice picture that’s…
>>: I think it’s much more helpful. But which one is doing it differently…
>> Betsy DiSalvo: Everyone’s doing different thinks, yeah.
>>: Then ideas could flow.
>> Betsy DiSalvo: You had question.
>>: You haven’t really talked about the fact that you’re now making more attention demands on the
students, right.
>> Betsy DiSalvo: Yeah.
>>: That somehow they, I’m supposed to be doing my little project here on my PC. Then I’m also
supposed to be like looking around seeing what other people are doing.
>> Betsy DiSalvo: Yeah.
>>: I’m wondering like did you find that people kind of shut down and sort of watching the wall or did
they ignore the wall and sort of try to get work done or?
>> Betsy DiSalvo: Our, we have a pretty small sample size right now.
>>: Yeah.
>> Betsy DiSalvo: In our interviews and observations so far we haven’t seen that. I haven’t dug through
all the interviews actually. They haven’t finished collecting them all by the time I left town. We may see
some of that. I wouldn’t be surprised by it.
On the other hand in the idea of treating this like periphery information rather than central information
for the class which is still happening on the projector that the professor’s putting forward. It’s the same
thing that might happen in any sort of studio environment. I mean it’s a lot of heads down work. I
mean in that first picture I showed you with the people throwing pots. They weren’t looking around the
room they were heads down.
But when you get up to you know take a break that’s when you notice other things going on, right. I
mean we all work in those kinds of environments on a regular basis. Is it going to be too much for the
students? I mean that would be a great question to look at and ask. Yeah?
>> Blair MacIntyre: In the project, the [indiscernible] project that I showed a picture of that we did back
in two thousand. We actually ran studies with a Psychology Professor on change awareness in
peripheral displays. We had you know how fast can we change these things. How fast can we change
the contrast so that you won’t actually notice things in your peripheral vision? That stuff never got
published because they moved buildings and the machines got stolen.
>>: Was of the professor?
>> Blair MacIntyre: Greg Corso, he’s retired now.
>>: Okay.
>> Blair MacIntyre: But in, and we talked about other designs for this at different points, right. You
could imagine if we, if every time you changed a picture it tweeted, right. You had something that
looked like a Twitter window and there’s this like constant motion on the side of your desktop. That
would be horrible, maybe. Because it would be impossible to ignore, bang, bang, bang.
>>: Right.
>> Blair MacIntyre: But if it’s hopefully up in the periphery it won’t be…
>>: Right now it just fades in or it’s just a…
>> Blair MacIntyre: They’re just switching.
>>: They just appeared.
>> Blair MacIntyre: They’re just switching. This is like level zero, right.
>> Betsy DiSalvo: Yeah.
>>: You see the students whenever something switches…
>> Betsy DiSalvo: No, not really. I mean they might put theirs up and then look around the room to say,
oh mines up. Oh, look what everybody else did, right. Like that seems to be more the moment when
they run their program they look for where it appears on the wall. Then they might look around the
room to see where everybody else is.
>>: Is say if I work under my displays in a fixed position.
>> Blair MacIntyre: Yeah, we fixed those twelve students and we just fixed them, they were the same
place every…
>>: How aware are they about what they present?
>> Betsy DiSalvo: About what they present?
>>: Yeah.
>> Betsy DiSalvo: They were pretty, I mean…
>>: They must be because…
>> Betsy DiSalvo: They were aware.
>>: It was on their screen.
>> Brian MacIntyre: Yeah.
>> Betsy DiSalvo: Yeah, they were aware of what they were presenting.
>>: Did they comment? I mean sometimes you know oh, it went out to the public.
>> Betsy DiSalvo: Well they certainly looked up to see if it did, right, so that happened. I know that they
were checking to see if it went out to the public. I was worried that some of the students would be
embarrassed or concerned with this. Now, they did get to opt in to the class. They chose to take this
particular; they didn’t really know what it was. But they opted in to the class because it was perhaps
held on a different side of campus on the right time of night for them.
>>: Yeah…
>> Betsy DiSalvo: I think that’s usually the motivation…
>>: [indiscernible]…
>> Betsy DiSalvo: They didn’t know what technology we were using. But it was a smaller class to they
may have liked that. The, but we told them what it was about in the first day of class. They could opt
out, none of them did, but they could have. It maybe that we got students who were comfortable with
this more than the average student classroom would be, yeah.
>>: Do you think that there is a danger that, not a danger that’s a pretty heavy word. But do you think
there’s a tendency for copy. You know like you see everything up there and you’re like oh, that’s
supposed to be what I’m doing. Like, then you sort of sacrifice your own direction in a way because you
might feel collective pressure to do what everyone is doing. Like, how do you manage that? Like there’s
learning and learning skill sets, and getting that in your tool belt so you can get your new stuff, right.
But how do you sort of balance the line between that and then sort of muting out like individual
creativity or generating more homogeny in output?
>> Betsy DiSalvo: Yeah, so I think that, I’m actually addressing this with another project that I’m working
on. It’s…
>>: We know it’s really early, so I mean I know it’s like early.
>> Betsy DiSalvo: No, but modeling what kind of the output is in a design is actually quite challenging.
Because anytime you present a model for the students they’re immediately going to think that’s the
right way to do it, right. What we’re doing to address this is actually bringing out multiple models for
the students to look at, rather than just the faculty members or the instructors. Like this is the best way
you know to do it.
>>: Yeah.
>> Betsy DiSalvo: Here’s a good example, we want to give them good, bad, medium, just different
examples. In some ways I think this already facilitates that more so than what we’re doing, right. Right
now we’re giving them one model of how this is done. Now they’re getting twelve, right.
>>: But if you do, say you ask for people to draw something, okay the same object. If they don’t see
what other people are doing and you compared those drawings of the same object. While, if you
actually show how people are going is there going to be a less of the spread will be smaller when people
are affected from start by other people.
>> Betsy DiSalvo: That’s a really good question. I’m not sure. I actually think it depends a lot on the
environment and kind of the cultural norms in the environment. If you were to give a drawing
assignment likely in a lot of art courses, you would see students actively looking at each other to try and
do something different than their neighbor, right.
>>: Okay.
>> Betsy DiSalvo: Inside a computer science class I’m not sure. That’s actually a very interesting
question, right.
>>: Do you have any examples of how people use the system in a way that you didn’t anticipate? Like
do you have hacks or…
>> Betsy DiSalvo: No.
>>: I mean because…
>> Betsy DiSalvo: Actually I don’t.
>>: Totally it’s just literally presenting a picture out there.
>> Betsy DiSalvo: Yeah.
>>: Doing it on a projector, right.
>> Betsy DiSalvo: Yeah.
>>: But did they use it to like hey guys let’s go to a bar afterwards?
[laughter]
>> Betsy DiSalvo: This is…
>>: Is there a camera?
>>: Is there any kind of hacking on the system that happens?
>>: The guy, the person who chose the Donald Trump image might not…
>>: Fair enough, or that’s a good example, yeah.
>>: [indiscernible]
>> Betsy DiSalvo: I mean we have…
>>: I’m just curious about…
>> Betsy DiSalvo: Georgia Tech is in the middle of Georgia. I mean we’ve got really conservative
students there. The Donald Trump thing might have been a bit of trolling of the class, right, yeah.
>>: Yeah, he was colored green with ears, wasn’t he?
[laughter]
>> Betsy DiSalvo: Yeah, well the color green, yeah.
>>: Obvious.
>>: But I’m just kind of curious did people use in a forum that wasn’t explicitly demanded by the
instructor. But, not demanded, suggested by the instructor but actually kind of unexpected. Like did
people try to actually make a collage and…
>> Betsy DiSalvo: So…
>>: That’s what I was just suggesting, exactly.
>>: Is there any other…
>> Betsy DiSalvo: We didn’t see that this semester.
>>: [indiscernible]
>> Betsy DiSalvo: I think if we kept teaching in this environment we would eventually, yeah.
>>: [indiscernible]
>>: So would we.
>>: I think it also depends on the subject, right. Subject like, so for instance if you’re doing sort of you
know political events, right, in real time.
>> Betsy DiSalvo: Yeah.
>>: Like you can have people be journalists and like stuff like, right.
>> Betsy DiSalvo: Yeah.
>>: Versus okay now we’re learning how to you know connect a circuit and we’re doing this. Versus,
you’re going to see…
>> Betsy DiSalvo: Different things.
>>: Eventually want to game it for different outlets.
>>: Okay.
>> Betsy DiSalvo: One of the things to keep in mind is we’ve been teaching this class for a long time.
Any sort of crazy manipulation to the media we’ve seen. It’s hard to say that’s unexpected because it’s
been happening for years. Now maybe it hasn’t been happening publically for years but, yeah.
>> Blair MacIntyre: But it, I mean Mark did a lot of work early on with Wiki’s.
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: He was like one of the founder, pioneers of using Wiki’s in an educational context.
In this class students constantly would post their results on the class Wiki. But he would have a page
that’s like hey when you get a collage done post it so everyone can see it. You would get that effect of,
but now people are only posting their finished best work, right. It that same effect would have
happened, right, where people are oh, my god, that’s cool. How do I, you know I need to copy that.
>>: Via connect you use that once you have the tools to do more things.
>> Blair MacIntyre: Yeah.
>>: Like for agrees or disagrees put a V or an X. You see that on the walls, right, or…
>> Betsy DiSalvo: That’s true.
>>: Or suggestions.
>> Betsy DiSalvo: That’s really; I mean this is the next step next semester that I want to take. We have
some proof of concept here that this is working and getting some of the affordances we want for
students. It’s certainly not studied fully in terms of the learning science context. But it’s promising,
right.
The next thing I would really like to do is understand what kind of gestures, what kind of interface both
desktop and physical interface, and what things we can do in the room. Like I want to be able to go up
and you know touch this screen and have something else come out that exposes things for the students
that otherwise wouldn’t be exposed, right. I want to be able to grab maybe a picture over there and
bring it to the front of the classroom and talk about that.
We’re going to be doing some design research and participatory design activities with stakeholders. To
try and figure out what kind of design concepts we want to start building. That’s my next steps.
>> Blair MacIntyre: Yeah.
>> Betsy DiSalvo: That’s what I’ll be doing in the fall with this project.
>>: Also…
>>: Yeah.
>>: Are you also integrating any sort of gamification around this? Like are there, is there like popularity
or, I mean not to you know. But, I mean you know if you have certain skill sets in certain areas you
could be an expert. Like you can have profiles being built around a kind of con that you’re generating.
Like, so I mean would that be something you might?
>>: With stickers.
>> Betsy MacIntyre: I’m not pursuing that.
>>: Really doesn’t have that, okay.
>>: Give me my stickers.
>>: Collecting on a…
>>: Just say it, you can just say it.
>>: [indiscernible]
>> Betsy DiSalvo: Yeah.
>>: [indiscernible] stickers.
>> Betsy DiSalvo: Yeah.
[laughter]
>>: You’re like trying to say…
>>: I am.
>>: Everything but stickers.
>> Betsy DiSalvo: Stickers.
>>: We almost had a messenger. We almost had it.
>>: That’s right.
>> Betsy DiSalvo: It is not something that we’re focused on right now.
>>: Sure.
>> Betsy DiSalvo: I think that it kind of is counter intuitive to the philosophy that we’re thinking about
with the studio pedagogy where it’s not really about these. It’s not about stickers or gold stars or things
like that.
>>: Yeah, [indiscernible]...
>> Betsy DiSalvo: It’s really about building this…
>>: Thinking more in terms of just an incentive to jump in and participate…
>> Betsy DiSalvo: Start participating.
>>: And getting sort of, some sort of accolade or moving like to keep going. Like, hey, like you have like
ten thumbs up on this. Like this is exciting…
>> Betsy DiSalvo: I mean a lot of the studies looking at learning in those kinds of environments they’re
very effective for about six months. Then it’s a drop off as the, there’s a saturation point where people
just kind of, there’s a few people that stick with it forever, maybe you’re one.
[laughter]
>> Blair MacIntyre: We are interested in knowing [indiscernible] of the gestures and other stuff.
>>: Yeah.
>> Blair MacIntyre: Knowing what things people look at. Knowing things, so it wouldn’t necessarily be
that you’d give a thumbs up on an image like…
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: The Trumpy thing.
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: Although you might. This is one of the questions…
>> Betsy DiSalvo: Yeah, I mean we think…
>> Blair MacIntyre: That we’re interested in pursuing. It’s just like yeah do we have a desktop
interface…
>>: But like…
>> Blair MacIntyre: Where you can say that. You might not get a star for being always getting thumbs
up images. But we would like to know that that image was inspirational to people because, maybe we’ll
come back in the later semester and show that same image. As an example when they’re doing the
same recitation. It’s detached potentially from you, so you’re not getting, we don’t want to necessarily
encourage people to show off.
>>: No, or competition.
>>: [indiscernible]
>> Blair MacIntyre: Or compete, but...
>> Betsy DiSalvo: Yeah, or compete.
>>: Yeah.
>> Betsy DiSalvo: I think that there’s some things that we can do with this and particularly using the AR
environment where you know I can show that there’s just a lot of activity around this particular thing.
Maybe it’s more subtle than you know a thumbs up. Maybe it’s more of like a heat map type of thing on
the floor. Or maybe it starts pulsating…
>>: [indiscernible]…
>> Betsy DiSalvo: And everybody’s, yeah.
>>: The right, so you can get for an aggregate…
>> Betsy DiSalvo: It’s the same kind of thing that might happen in other environments that we can take
advantage of with the AR, right.
>>: Yeah.
>>: How about actually copying and pasting from the display? I mean not for to steal an IP in this sense.
But it makes it easier to comment or collaborate this way. I could take the Trump, do something and
show you could do this.
>> Betsy DiSalvo: Yeah and I mean we’re thinking about what does that desktop interface look like for
the students. Do I click on it on the desktop? Do I click on it physically? I don’t have the answer to
those questions. But I think that there’s some really fascinating interactions between, because you need
both interfaces, right. Or maybe the students don’t have any desktop interface for it only in a short, I
don’t know. But like that, there’s a broad range of questions here that we have to start figuring out.
>> Blair MacIntyre: I think it goes back to culture too like you can image a small room. Like this where a
small group of students have been working and they’re comfortable with each other. You could freely
give them the ability to grab that, an image, write on it on the desktop, and post it back. A three
hundred person classroom where there’s a hoard of images around. With a group of computer science
students who don’t know each other and have never been in this environment. It would just be troll
central potentially, right. You don’t know what they would do.
[laughter]
I think who knows.
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: But, so I think that, I’m really excited about Betsy’s idea to sort of step back without
having to implement things. To just do some design based research to figure out what do we need?
What would be useful do we think, right? Then figure out how to implement it and move forward.
Like things like heat maps or you know is there a need to have stuff around the students on the
desktop? Is there a need to sort of display more stuff? Do we want to integrate a surface hub at the
front where we can like mark up things and throw things at the desktop, and so on? It depends we
don’t know yet.
>> Betsy DiSalvo: Yeah, so those are some of the questions. Then…
>> Blair MacIntyre: Yeah and so…
>> Betsy DiSalvo: Blair has some more questions.
>> Blair MacIntyre: This goes into the technology, right. Part, in a sense based on what we, once we
finish these, analyzing the data right now based, after we do some design and figure out what we want
to do. There’s kind of a ton of things we could do, right. That’s everything from like using a lot of the
technologies that these guys have developed, right. Is there some of the stuff that we could integrate
more.
Potentially, there’s obvious simple things, like it would be really nice if the projectors weren’t shining in
your face when you’re lecturing in this room. It would potentially be really nice if as I walked around if
there was dynamism on the walls, and so on, right. If I put a real poster up here because I printed
something out. Maybe the things move away so we’re not over. Just all kinds of things you could do
once you decide how you’re going to use the space.
You know better tools for prototyping these environments. I’m all about tools because I have a CS
degree. The, you know given this technology if we have a new idea that a master student wants to sort
of prototype a new kind of structure for the space, can we make that as easy as possible? As, opposed
to requiring a certain level of technology mastery to build these things.
You know Microsoft is developing a whole set of APIs for augmented reality, right. The holographic APIs
and HoloLens as a technology is sort of this example of really being able to do AR in a real environment.
Now the current version isn’t going to be something that we want these students wearing in the
classroom while they’re programming on their laptops. But it’s just AR, right.
I think there’s a fascinating question of if we displayed the same information on a head mounted display
you know it’s just the same stuff around the walls. But now it’s not sort of public even though I tell all of
you people that you’re all seeing the same thing. Would that change what people do? It might totally
destroy it because deep down inside I’m not necessarily convinced.
But I would then have the option to display things personally. You know I could have like all of my
history and my code up there because it’s my things. But I can’t see your code and your stuff or
whatever, right.
I think there’s a lot of potential for exploring other ways of doing similar stuff. We were talking to one
of the HoloLens Team this morning and talking about running classes on AR, right. Well, you’ve got now
a room with everybody wearing an AR display because they’re programming it, right.
It could be the same thing where now the little AR thing you just ran it’s floating above your head in my
view, right. I see like AR everywhere and see the same kind of stuff. That might be kind of cool. That
particular context would enable it. I think that’s it, is that it?
>> Betsy DiSalvo: Yeah.
>>: You started with kind of motivational or enabling CS students to appreciate more of a critic style
evaluation and critic style classes. Do you feel like this room facility that and do you feel like the
students actually learn more of like they appreciate it and learn the, how to critique better? Because I
kind of completely agree with you I don’t think that CS people are really versed or know.
>> Betsy DiSalvo: They’re just not trained on it, yeah, yeah.
>>: Right, [indiscernible]. I’m wondering, do you feel like the people in there actually who took your
course would be better?
>> Betsy DiSalvo: I can say with this augmented reality studio that that’s probably not the case because
we didn’t really implement special measures to try and make that happen. Within the maker oriented
learning classrooms that we’ve doing for two years you know. We’ve added in sort of tools to help the
students become better at critiques. Also to help the faculty facilitate that better and we have seen
progress in that arena.
The first time that we ran crits like there was three students in a class of fifty that said anything. They
were the only three students during the entire class who said anything. That’s not unusual in a CS class,
right. Then we gave each one of the students a small, a half of a sheet of paper that asked them three
questions and said please, you know think of something good about this you know presentation. Think
of something critical about this presentation and what is one suggestion for the presenters?
Then we asked them give feedback on one of those items for each presentation. We got a lot more
feedback. A lot of it was yeah that’s cool. I like that idea you know because that’s the easiest thing to
give. Or as I mentioned before sometimes there’s like this really critical stuff that was like I think your
idea was stupid, that’s never going to work sort of a response.
>>: [inaudible]
>> Betsy DiSalvo: What?
[laughter]
But as the semester went along using that tool actually students got better at it, right. They saw when it
was useful. They also got to experience, so sometimes the person in the class that is super aggressive
gets the least amount of critique back at them. Because people don’t want to become a target, it’s bit
of a bully kind of situation, right.
>>: Yeah.
>> Betsy DiSalvo: But when they suddenly started getting critiques back. Not that they were particularly
negative they started to maybe see. I hope they saw the function of them. But it did start to change the
culture of the classroom a little bit over the course of a semester.
I think I told you this earlier. We just tried something out this semester that’s been I think more
successful was we gave out cards. It’s a little gamified too. We gave out cards to all the students. They
were asked to contribute one thing based off the prompt of the card which was what is something good
about this presentation? Or the card might have said what is something bad about this presentation?
Or it might have said what something; suggestion or you know, what’s a similar type of project that they
should look it?
Several different prompts for them, we handed them out to the class. Everybody just got one. They
only had to ask one, or use one of these cards during the entire class period. They wrote their name on
it and then turned it into the TAs because we still have to do that with the students.
[laughter]
That was by far the best critique that we’ve had. In this instance it said all the students, three questions
for each person was too much. They were so busy filling out the form that they really couldn’t pay
attention. Thinking of one thing was great. They waited until the end. They were able to think, reflect
for a minute and then fill it out or not fill it out but just ask the questions rather than filling out a form.
They also were interested in trading cards. They’re like I can’t think of anything bad to say. Can I use
your positive card for my negative card? That happened too which is fine. I mean that wasn’t a problem
at all.
It gave me hope that there’s better ways to do this. Maybe we just need to work a little bit harder at it,
right. In this environment I think it’s different. I think we’re going to have to think about how to do
critiques in a different way, right. These aren’t CS majors particularly for this class. But I mean
eventually mostly interested in applying this in CS classes so that we can kind of change that culture.
>>: Did anyone successfully float the CS class? I’ve never heard of it.
>> Betsy DiSalvo: Successfully float the CS class?
>>: Yeah.
>> Betsy DiSalvo: Well…
>>: Because I actually…
>> Betsy DiSalvo: Jim Foley and…
>>: I mean like they watched the lectures…
>> Betsy DiSalvo: Who was the students who did that?
>>: Ledgers out then they come in…
>> Blair MacIntyre: There’s a bunch of people doing that at [indiscernible].
>> Betsy DiSalvo: Yeah.
>>: But that’s, are they, is that like CS101 or is it like CS200?
>> Blair MacIntyre: CS higher, like, so Jim did at a three thousand level, Jim Foley did a three thousand
level…
>>: Was there undergrads?
>> Blair MacIntyre: No, undergrad.
>>: Oh, okay.
>> Blair MacIntyre: Ours are one to four thousand for the four years. The HDI class they did a lot of go
watch this video…
>> Betsy DiSalvo: Yeah and they did a really rigorous study on it showing that the flip classroom was
more successful than the traditional classroom.
>>: One of the interesting to me is I think we have here and sort of putting on an infrastructure in a
particular place on campus, projectors, cameras, what have you. Is it gives them a reason to come in to
the place. I think that works really nicely with the flipping theme…
>> Betsy DiSalvo: With a lot of…
>>: I just haven’t heard that much of that…
>> Betsy DiSalvo: Yeah, with the active classroom…
>>: I’m not tuned into it.
>> Blair MacIntyre: Yeah.
>> Betsy DiSalvo: It’s, in active learning it’s often really about flipping they just don’t call it that.
>> Blair MacIntyre: Yeah.
>> Betsy DiSalvo: Like whenever you ask the students like read this paper or watch this video before
class and then we’re going to discuss it in class rather than me just presenting the same material over
again. I mean that’s a flip classroom just, and it’s been happening long before they gave it that name.
>> Blair MacIntyre: With our OMAT, we have this online master of computer science.
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: That’s gotten a bunch of buzz. I think Rich maybe talking about it later today. The
people who are teaching those grad classes are having to make videos and everything because it’s an
online. It’s structured as a Mooc. It’s not just like watch the live video of the teacher. The same
instructors are then using some of them, or using those videos in their online, in their on campus class to
do some of this flipping.
>>: Computer science is funny almost an exception in this whole process, right, because you use the
very same tools that are used to make flipping possible. Are the very same things that can undo the
whole point of coming into you know sort of water down this kind of experience of sitting around in the
same room. It’s that you look at all the social media technologies and telepresence, all that sort of stuff.
It’s like after awhile it’s like why do I come to campus at all?
>> Blair MacIntyre: Well, I think the, you know my question with all of this like you know blowing up
education and disruption, and on and on. Is always, it’s up to us to figure out ways of making it worth
you spending the money to be there, right. There should be some value in coming to class, this sort of
environment having active learning situations, in class. Where you’re being both guided by experts and
working with your peers is an obvious candidate for that sort of value.
>>: If I ask exactly the opposite. Okay, let’s say that you do have a bar on your display that shows those
posters. It changes really gradually so it not attracts your attention. Would that make the solitary
experience of sitting with the computer to suddenly there’s more interaction. Because I can see
something interesting and I could say something about it. Suddenly I’m not alone online but I have
other people to…
>> Betsy DiSalvo: Yeah, I mean I think there’s definitely potential for that. I think that people have tried
to do that in online environments right now. Maybe not with visual imagery but with you know an
ongoing chat or problems being proposed and things like that. But…
>>: It’s really disrupting those chats.
>> Betsy DiSalvo: Those chats are really disrupting?
>>: They tend to, yeah.
>> Betsy DiSalvo: Yeah.
>> Blair MacIntyre: I mean, so the way I think about this relationship and Mark and Betsy, and I talked
about this. Is that you know if we can show that it’s beneficial in this context, in the studio context and
also if we can understand why it’s beneficial or how it works. I mean Betsy’s antidote about the
questions for the crits, right. This, she learned this by actually doing it, right. If we can show that it
works then potentially we can figure out how to design those online tools. To, because now we’ll know
what it is about this environment that actually facilitated learning?
How can we then design a distributed environment that would potentially have the same benefits
without the negatives? Or as little of them as possible like without the distraction, just showing a
stream of stuff maybe wrong, you know you maybe you know as Betsy said to me once maybe we need
to get them into small sub-groups so that there’s not an overflow of information. Maybe we have to
curate the content so that it…
>> Betsy DiSalvo: I said this to him this morning.
>> Blair MacIntyre: That’s why I remember it.
[laughter]
>> Betsy DiSalvo: A long time ago Betsy said this…
>> Blair MacIntyre: It was a long time ago.
>> Betsy DiSalvo: It was like two hours ago.
>> Blair Macintyre: I think there’s immense potential for this sort of stuff. Yeah, hopefully in a few years
we’ll have…
>>: There’s also an equivalent what’s happening right now in team work in you know with Slack, right.
This idea that your aggregating was everything in exposing almost everything about the team process so
that everybody could keep continuous tabs on what’s going on. Now it doesn’t, I don’t know I don’t
believe it actually is super efficient for very large teams, you know hundreds of people like that. But this
studio style, studio sized groups of tens and twenties is probably really very efficient. I guess one
interesting thing would be to actually consider something like that.
The room as just kind of another dashboard experience or a set of tools like Slack or something like that
where you’re really just exposing the real time nature of the, everybody’s work. But you’re exposing it
at a very public…
>> Blair MacIntyre: Yep.
>>: You know all around encompassing way which is like, more lightweight than installing you know
NASA style like bazillion screens to the space mission track.
>> Blair MacIntyre: I mean in some way the most abstract way to describe this is we’re exposing hidden
digital work, right. We’re using it for education, right, or work in progress. We’re using it for education.
But you could imagine…
>>: Well…
>> Blair MacIntyre: You know if it’s a team doing development you’ve got repos. You know you’ve got
all these online tools. There’s years of work on program visualization on how you visualize changes to
repositories, how you do all this. Yeah, expose it. It’s a design problem then which is interesting.
If you could then start figuring out how to actually capture their work in progress before they check it in
that might even be better. But it’s designed, right. I mean the interesting thing for me and I’ve said this
to a bunch of people over the last few months. It’s not actually possible to do augmented reality, right.
I’ve been doing AR in one form or another since like ninety-one, so I’ve been doing it a long time. But it
was never actually possible to do it. We built systems that worked for that one instance. We demoed
them. We recorded the video. We wrote a paper. Maybe we tested it with N equals five or something.
But now it’s like okay I want to put AR around the walls of this room. We could do it. This is still hard to
setup and so on. But with HoloLens, with all these other technologies that are starting to come out I
want to do collaborative AR. I want to display a medical scan so a team of doctors can talk about it,
right.
Now the question is what do I need to display? Not, oh my god, how do I build something that we can
actually use. I think again what Betsy’s interested in doing with the design questions we can actually
now design and then do as opposed to kind of not. I think it’s kind of cool time to be doing AR.
>>: I think that right now if I look at it as a scale or one side the students present their work one time at
the end. On the other maybe what you show right now is almost continuous. But it also has a minus
from my point of view which I have to split my attention in some level, because I have to be aware of
whatever everywhere else. Maybe in the middle you have things; we can have intervals or some things.
>> Betsy DiSalvo: Yeah…
>>: In between them I could be totally concentrated on what I’m doing.
>> Betsy DiSalvo: Yeah and that maybe what we find. I think these are the things we have to play with,
right.
>> Blair MacIntyre: Yep.
>> Betsy DiSalvo: It may be different for different people.
>> Blair MacIntyre: [indiscernible]
>> Betsy DiSalvo: That’s where like something like the HoloLens might be helpful.
>>: [indiscernible]
>> Blair MacIntyre: Okay.
>> Betsy DiSalvo: [indiscernible]
>>: Do you think your findings may change when increasing your number of kids of twelve to a regular
lecture that would actually be fifty people?
>> Betsy DiSalvo: I suspect they will. I think actually we want to focus on; the capacity issue is a big
issue because computer science classes are not getting smaller. There’s faculty, the number of faculty
aren’t changing and the classes are just getting bigger and bigger every year.
I mean, I think the ideal way to think about implementing this is in a class of fifty to seventy. At least on
the Georgia Tech Campus which is totally different than a typical studio environment, but it’s just, these
are the constraints we have to work in, right. We need to figure ways to bring the best of it we can into
that environment without you know breaking the institution and kind of those resource and stuff that
we have right now. Yes, I mean that’s part of the challenge and part of the constraint that we have is
this capacity issue.
>> Blair MacIntyre: Okay.
>> Jaime Puente: Any questions?
>> Betsy DiSalvo: Thanks, great.
>> Jaime Puente: Okay.
[applause]
>> Betsy DiSalvo: Thank you.
>> Blair MacIntyre: Thank you.