>> Desney Tan: It's my pleasure to welcome James George who is -- I was tempted to introduce
him as a computer scientist turned artist, but I suspect he was artist way before he was a
computer scientist. So he is an artist who happened to have gotten his degree in computer
science is basically what happened. His art pieces, his installation films, his mobile applications
have been exhibited around the world and I think he's going to show us some of that today.
James is currently an artist in residence at the Eyebeam Art and Technology Center. He's also a
fellow at the CMU Studio for Creative Inquiry as well as a faculty member at NYU, so take it
away James.
>> James George: Cool. Thanks Desney. So the title of my talk and the plan was to talk about
computational artists through a virtual lens and I think that will clarify itself quickly. But then
after meeting you all yesterday I figured I might throw in a little more personal data, so the
second title for this talk has become A Computational Artist with a Virtual Camera, so it's these
two things together and this is going to get pretty meta, so be careful. So my name is James
George. I have a Twitter and a website which I encourage you to contact me through, and I
have a hobby of playing with cameras in strange ways, and this is one of my favorite pastimes.
[video begins]
>>: I'm about to set the freefall high score.
>>: Here it comes. [laughter] [video ends]
>> James George: So this was the first test of an application that I developed that was called
FreeFall High Score and the [laughter] concept of this application was who can drop their
phone, or rather suspend their phone in freefall for as long as possible. Using the phone's
accelerometer I could detect the minute it left the Earth's gravitational force or rather joined
with it, and start a timer as well as recording video. Those videos would then, you could tell the
story on how long they had fallen and that would upload to YouTube where it was secretly
tagged with that data and then aggregated onto a website ranking the highest score drops in
this kind of high score page. And the concept here is to take something that we value more
than we should, that maybe we don't even realize the amount of that we have become
attached to these devices and then take an application through that language, the logic of
writing applications and call into question that attachment, and put someone in an
uncomfortable situation but still, you know, taking the box of competition and playing creativity
and maybe instead of looking at your phone, look at the high buildings around you in New York
City and think about how can I use that building to break this high score. So I was really excited
about this application, you know, fervently developed it, submitted to the App Store and was
returned with this. [laughter] and I mean, Apple said in their policy that they wanted people to
make innovative applications that make people think about their lives in new ways. I thought I
was doing that, but they had a different opinion, so the application is available on android, not
as cleanly implemented as Apple, but that's how it goes. But I thought that to make up for this I
would do an event, like an installation, like how am I going to get this out into the world?, How
am I going to find people to set the high scores, so I started a contest. And the contest was
modeled after in high school you have an egg drop contest where to teach kids physics, how do
you protect an egg. Well I was going to do the same thing except I would give people phones
and they could keep the phones if they could find a way to drop them off a building and set a
high score. And I would like to show you the winner of that competition.
[video begins]
>> Tim: Hi. My name is Tim. This is my preserver. It's the iPot and it has a little hole in the
bottom for the camera to peek out of and some daisies on top.
[video ends]
>> James George: So the phone obviously survived and he won the high score. It was 2.7
seconds. And actually someone in Finland recently broke the high score, but they were
disqualified because they put the phone inside of a container so you couldn't see the video, and
one of the requirements for the preserver was that you couldn't obscure the camera's lens.
You had to have proof of the phone falling. So that was an individual project, but I also work on
a lot of collaborative things and this is where this awkward position of being a computer
scientist and an artist comes into play is that oftentimes I'll work with other artists who realize
their ideas or merge my skills with them to create larger scale installations and things with
greater context. So I'm going to show two collaborations that I think will resonate with people
in this lab. The first is called Sniff. Maybe we can get the lights down for just a second. Sorry to
spring that on you. So Sniff is an interactive dog made for street projection and he tries to
understand what you think of him and in doing that he tries to make sure that you are trying to
think what does this person think of me. So it's a way of instilling agency in a virtual creature
and inject it into a public space in a way that makes you question your environment. People
that aren't expecting to encounter art or encounter an artful experience and wonder who put
this there? Why is this here? Why is this dog looking at me? And then slowly start to
understand through the power of gaze and interaction that yes, this creature, this virtual dog is
indeed guessing what I'm trying to do. If you run at it quickly it will jump back. If you hold out
your hand it will wag its tail and sit down. If you move suddenly it will bark at you. If there are
multiple people it will choose the most interesting person based on their gestures and, you
know, different -- a group of people will vie for the attention of the dog. And the idea is that
even though we know something is virtual, if you instill something with animated qualities of a
character, we can suspend disbelief and start to really imagine what does this creature think of
me. This has been exhibited several times. This was the first installation on a Brooklyn street
corner. And just because I'm the tech guy, I'd like to show you the behind the scenes of how
this works and this begins to get -- at some of the conversations we were having yesterday. So
this is Unity 3-D rendering the dog which is choosing hundreds of small animation clips that are
less than a second long, trying to decide which one is best based on the dog's desire, where it
wants to stand, what expression or emotion it wants to express. At the same time there's a
vision system that's feeding it information about who is standing in front of it and these
applications were never meant to communicate with each other necessarily, but they work
over a network protocol called OSC and we designed these little communication systems to
make experiences that are more powerful than either of the applications could achieve by
themselves. So that project led to another collaboration with renowned video, music video
director and experimental artist named Chris Milk where he had a concept for this triptych of
installations that interacts with you and I directed the software for this, so I'll show you a video
of him explaining how that works.
[video begins]
>> Chris Milk: It's a triptych where the three individual panels represent the process of creative
conception. This is something that I struggle with constantly. So the first panel where your
body disintegrates into the birds, that represents that initial moment of conception. It's the
moment of inspiration. It's the lightning in the bottle. It's the purest moment of the idea that
there is, so this essentially represents birth. The second panel is representative of the critical
response, either by your own self-doubt, outside critical forces or just the impossibilities
presented through the process of production. This is what it feels like to have your purest
expression picked apart by 1000 angry beaks. In other words, this panel is bad. The third panel
where you sprout giant wings represents that feeling when somehow you and the idea are able
to transcend that death in the panel before. The idea transforms to the process of abstraction
into something that is larger ultimately than its original. This panel is Transfiguration.
[video ends]
>> James George: So can anybody guess what camera we used to make this? [laughter]. It was
the Kinect and this actually used the Kinect SDK because of its amazing skeleton tracking
abilities, and granted we ran into some trouble because we were pointing it at the back of
people and it was designed for the front and their hair or head would disappear, but all of these
aesthetic concerns that come up with using these devices you solve with design and you solve
with in person molding with the data trying to get it to express something that maybe it wasn't
intended to express. And those silhouettes are derived from depth map, something that was
meant for computers to use but we can use it to make it feel like someone's shadow is
disappearing to tell a story in a space. And along these lines as I've been doing this work,
something has been boiling up in my mind about the means of production of these projects and
I have this curiosity that's come from being so deeply ingrained in working with these cameras
in this state, this interaction and so it's boiled into something I called aesthetic research, or
critical aesthetics, but it doesn't really necessarily have a name except when you're talking
about the specific projects which I'll show you. But I think it all sort of started or what set the
spark was a science fiction author by the name of Bruce Sterling who maybe some of you guys
have read and he was giving the closing keynote for the Vimeo awards, the website Vimeo. He
was speaking on the vernacular video and it was a long rambling talk if any of you guys have
seen him talk, but at one moment he stumbled upon this gem that really clicked with me where
he was talking about the camera of the future. I'm going to play just a clip from that talk to see
the next, the remainder of my talk.
>> Bruce Sterling: Except it's not a lens. It's just a piece of black fabric or glass; it’s an
absorbtive surface. Is simply absorbs every photon that touches it from any angle. And then in
order to take a pic…
>> James George: Ulp, oh no. I'll try it again.
>> Bruce Sterling: Except it's not a lens. It's just a piece of black fabric or glass; it’s an
absorbtive surface. Is simply absorbs every photon that touches it from any angle. And then in
order to take the pic…
>> James George: Oh, it's cut short. So what he says, and I have memorized the quote, is in
order to take a picture I simply send the problem out to the cloud wirelessly. I compute the
photograph after the fact from a computational data set from these cameras that are
perceiving the world from every angle and in every moment of time and then I can photograph
after the fact and photography becomes a data visualization problem of sorts. A month after
the talk this camera came out and there was a lot of buzz around this camera and especially
with its interaction like how it opened up all these new possibilities for people to interact with
computers and speak with computers, but as the open-source drivers became available and
people started to pull the images off of this camera and those sort of streamed onto the
internet, we saw pictures that look like this and this is our friend Kyle. He's a member of the
open frameworks community and he was involved with 3-D scanning research long before the
Kinect came out and so he was of course one of the pioneers to be excited about the opening
of this technology for creative projects. And when I saw this image Bruce's words were in my
mind; this surprised me in reference to what he was saying because this photograph was taken
from a different angle from where that camera was, and you can immediately start to derive all
of the implications of that based upon, you know, his science fiction predictions that more
sensors, higher fidelity, higher quality, and you can start to think of what this new
computational photography paradigm that this camera might be nascently ushering in and I
wanted to explore that even in its nascent form. One of the things that I found the camera
needed augmentation for was the resolution of the color. I felt like it wasn't, the photographic
quality of the color camera onboard wasn't enough to satisfy my needs of documenting things
around. So working with the photographer Alexander Porter we strapped an SLR to the top of
the camera and then I wrote software that would capture images, depth images at the same
time that he would take a photograph just by signaling at each other and me clicking a wireless
mouse. We took it to the subway because we had just heard a radio story of a contract
between Lockheed Martin and the MTA where Lockheed Martin had promised a massively
advanced surveillance system to be installed in the New York subway system. And as happens
in product and software design, the product team sold it before it existed and then it never
actually worked. So it resulted in massive lawsuits and also thousands of unused surveillance
cameras just living dormant in the New York subway system. We sort of used this as an
inspirational anecdote to imagine what seeing through that science fiction surveillance system
would've looked like and so we pulled this data together and we started to combine it and we
pulled, the images we were pulling started to look like this and they felt like this fractured
reality of seeing through the eyes of a machine and to spark imagination all over the place and
people who were seeing it and thinking about this kind of thing, thinking about well, what
sensing technology means. And as we start to have this greater collaboration with computers
and interaction with computers, trying to empathize and trying to understand what it means to
be seen and what the limitations are. So we were playing with that aesthetic territory and
connotating that type of concern, but we were also still interested in this idea of rephotography and in this image set we would generate pairs where each image was, each pairs
of image was the same moment in time, but visualized with different data from different
perspectives. It may be easier to see if the lights were down just a little bit. This is getting at
this notion. We also realized a very interesting problem in our naivety that I couldn't just
literally scoot these images on top of one another. I couldn't just take the SLR image and scoot
it onto the depth map and have a lineup, that there was actually a greater problem going on.
And I was really interested in continuing to refine the aesthetics of this process in combining
these cameras together. Again, the open-source community sort of came to my survival and
we refined this process. We built this. So I have these to pass around actually so you guys can
check them out. We started talking to Kyle who was pictured earlier and Elliott Woods and
various members who had done a lot of like stuff that Andy is doing, this calibration combining
different perspectives and we opened up this whole problem space of how do we take a
camera like this and combine it with this other camera to get, to continue to excavate this idea
of depth cinema? And so we designed these mounts so that we could combine them together
and all of these interesting design problems sort of came out of this one notion and other
people started to get involved. So we eventually created the RGBD toolkit which is an opensource platform for depth filmmakers to begin experimenting with this format, and we're also
doing this while we are creating our project, so it's this shared notion of artists as toolmakers
and making, opening a process to the world as well as exploring it through your ideas. I want to
pause briefly while you pass those around and play for you the first pilot film of the project that
we're working on now that I'm ultimately presenting, which is this idea of computational artists,
of seeing artists through the computational lens. And I'll give a little more background on this.
This project started at the studio for creative inquiry, the calibration system at that prototyping
session was done as an artist in residency program where [inaudible] had invited and the studio
had invited many different people from the community to work together in person and that's
where we finally cracked the code on combining these cameras together. And so these in
person intense sessions are what create these sort of bursts of creativity and ingenuity and I
was so privileged to be a part of that because it is what has led to all of this interesting
research.
>>: [inaudible] conference in 2011 which was a gathering to enable Kinect [inaudible]
supported by Microsoft Research.
>> James George: Yeah, so gratefully indebted, and so my collaborator in the film I'm going to
show was also a fellow at the studio, Jonathan Minard and this is the pilot for the interactive
documentary that we are producing that I'll explain later.
[video begins]
>>: With programming you can do whatever you want. At some point it's, I wouldn't say
biblical, but it's like there's nothing in the beginning or just if you have no history it's just an
empty space and then you can start throwing things in there.
>>: What I want to do, I want to program like I can think, so like 10 years ago I painted a lot and
I want to get the same feeling with code.
>>: There's so many things that you want to do that you could do by hand and it would take
you a long time, but if like programming just increases your capabilities almost exponentially.
>>: I addicted myself and I'm highly addicted to programming. You can't stop me. Sometimes I
forget to eat. Sometimes I forget to sleep and I don't know. Can you turn a pickle back into a
cucumber? Probably not, so I'm here for life. You've got me; it's terminal.
>>: This adding and taking away, that's kind of the design process now. It's, I still feel in control
but I like to be surprised. I think that's the beauty of it, the thing when the system goes beyond
what you naturally understand it to do and sometimes you have something that surprises you
and I always look for that one moment basically.
>>: You get to a point in large visualizations where it's actually too much data to look at as a
flat image and you have to some sort of zooming or, you know, flying through navigation to
focus on some small part of it because, I don't know, because there are so many examples of
this like a huge hairball of connection and it means nothing at some point.
>>: And he is basically explaining how the database works and he says at one point here and
then you go around and there's another [inaudible] center there and another one there and
because this is closer this is one which is matched and it's like he explains everything like that.
>>: Is there an experience, that digital experience that you desire to have?
>>: Oh yes, tele-transportation, yeah, that's, there are many things, yeah, there are many
things okay. There is so much I shouldn't talk about, but yeah tele-transportation because you
know there was a time when they said with videoconferences you want me to meet but they
don't work as well. You still need to be with the people to re-discuss in more [inaudible]. You
can do part of it by videoconferences but it has its limits, so I have to travel and I get so sick of
traveling. I waste so much time. [inaudible] And I cannot work and so I wish teletransportation. Is it possible? Is going to be possible? I really want that.
>>: You could start off with the point cloud of this and then I don't know, at some point when I
say, talking about water or physical systems all of the points could just flush or turn it into
water or something like that just to illustrate the thing that you been talking about.
>>: [inaudible] so many people and then…
>>: It's the thing that comes after a video camera. It's the thing that comes after a camcorder.
It's a Kinect with a digital camera on it. I guess there's a sound microphone and this amazing
blinding light.
>>: Because this footage is rendered in a 3-D environment and you have to look at it from
some perspective. You can browse anywhere in it and you can see just edges of it or the
camera angle can change even though this camera has been pretty static. There's just a lot of
room to play. It's experimentation. You don't know what will work. There's no cinematic
language that has evolved yet, so we are sort of at a point to define that.
>>: In theory you could build the whole universe inside your computer. You just don't have
enough power and then you'll never get there and it would be boring to build the same thing all
over again but in theory you could. It's something that I find kind of stunning.
[video ends]
>> James George: So that film was the, the protagonists are media artists and programmers,
people who consider themselves in this confusing hybrid discipline but powerfully creative
discipline of being both implementors and creators of systems, as well as artists and cultural
questioners. And we continue to build this database of conversations and work, working now
in a way to present them interactively and that's something that I'll explain in a second, but I'd
like to pause and answer a lot of questions that I received yesterday and give a demonstration
of actually how this calibration system works, because they think this audience is particularly
receptive to that explanation, so I'd like to illustrate it. So again, this is the problem space. You
have two powerful pieces of commodity hardware and you want them to work together in a
new way and the problem is that you have two lenses which have different angles and it's a
classic calibration problem where you want to be able to look, to understand the relationship
between these two lenses, so that you can create a mapping between the two different types
of data and understand where they live in the world. So the first challenge is to find the lens
intrinsics. So we need to know exactly the shape empirically based on known data in the world
the field of view, the principal point, all of the things that you need to know about a lens to
model it in code in the distortion coefficient. And we do this for both the Kinect as well as the
SLR image or whatever your external camera is. This is one of the things that was really nice
about the libfreenect because it was, in its first release it had access to the raw IR image so that
we could actually see the checkerboard through the lens of the Kinect and then get the
intrinsics off of that. Which now I hear the Kinect does the case support so I guess we will be
able to support our toolkit on that platform which is really exciting. The second part is a little
more tricky and that's finding the lens extrinsics, and this is the new system that we just
recently developed that's a different way of finding this mapping, so I'll explain it really quickly.
So the extrinsic parameter, again, is the translation and rotation between the two lenses and
the way we find this is we collect a small data set of samples after we have the intrinsics, so we
know roughly what the SLR looks like in its own world and what the Kinect looks like, the depth
camera looks like in its own world. And so we collect a series of pairs off of the Kinect and this
is actually the color image that has been distorted to match the depth image. Most of the time
you do it the other way around, but because we’re really interested in the cinematic quality and
the aesthetics of the depth image, we don't want to touch that data. The distortion destroys it
visually so we actually work to pull the RGB pixels so that they match the depth image and this
is somewhat nontraditional. This is supported with the hack’s libfreenect but actually Kinect for
Windows supports this as well. So given this we actually get a really rich data set of
checkerboard positions floating in 3-D space, which is just super powerful because you can take
by inferring the checkerboard, the corners you can then sample those points in the depth image
so you get a known floating data set with known points. At the same time you take the
simultaneous coincident images with the SLR and load those into the software and so then you
have the same exact rich data set of 2-D image points. And for people who have done
calibration this is a simple solve PMP where you have object points and image points and you
can find extrinsic relationship rotation translation. So this is how we then understand the
mapping of those two lenses. And I'll actually demonstrate this working in code and this is
something I hacked together last night so apologies if it's a little rough. But this is part of the
RGBD toolkit SDK, so this isn't the toolkit itself, but it's code written against this library so it
gives me a few simple things like a timeline that lets me visualize both the depth stream and
the color stream together, so this is an external video file. This isn't taken from the Kinect. It's
an SLR. And I can scrub through this finding key phrases in these massive conversations was a
huge talent. We need to have a visual timeline like Final Cut or Adobe Premier-based editing
system, but built into our environment to understand these nontraditional data formats. So
this is one of the nice things that we've been building into the toolkit. So the first thing I'll do is
just show the depth projection matrix. So this is the intrinsic parameters of the Kinect
Visualizer. And it's a classic pyramid. It's a 640x480. Inside of that we can render our projector
points and draw a wireframe of our subject. She's thinking critically. She might even be
thinking about whether she wants to teleport right now, I think. And we can play this back.
[video begins]
>>: [inaudible] videoconference you want me to meet but they don't work so well. So you
need to be with the people to really discuss and [inaudible]. You can do part of it by
videoconferences, but… [video ends]
>> James George: So now we have a big part of our data which is a really nicely surface
reconstruction. We can do some hole filling here which we have stuff for. But we have our
form and we have it mapped in the studio, but now we have to actually combine our known
extrinsics and we do that using projective texturing. Here is a visualization of the RGB camera
in relationship to the Kinect and this is pretty good, right? There is our Kinect. There is our SLR
and you can kind of imagine the lenses floating a little bit out in front, so that looks about right.
If you look at where they land on the figure, that's about right. So then if we actually visualize
our texture being projected here, you can see it coming off of this [inaudible] and then we can
just project that onto our form. And now we have a richly textured SLR.
[video begins]
>>: [inaudible] I waste so much time in planes [inaudible] and waiting for trains and I cannot
work, and so I wish tele-transportation. Is a possible? Is it going to be possible? I really
wanted. [video ends]
>> James George: And so you start to see the camera language here, like I have interactive
control over viewing this person talk and I want to punch in at certain points I'm curious about
which angle looks more interesting. I pull off actually to see the profile or look for someone in
the eye even if they look away from the camera. And this type of interaction has become
extremely fascinating and we really feel like we're excavating a new type of cinematic language.
I hope that was elucidating. That was the software demo. Didn't go so bad. [applause]
[laughter] And actually I'll stop there if anybody has any questions about that technical stuff
because then I'm going to get back into the conceptual project.
>>: Why are you going back to the debating about whether or not [inaudible] off-line? I do
have a couple questions about the calibration procedure.
>> James George: Okay. Do you want to just talk off-line? Okay.
>>: Yes, we can do it later.
>> James George: I don't want to bore everybody else. Yesterday we were joking about
starting up a calibrators anonymous. It's like drinking club but people are like where's your
fudge factor? And it's like I totally hit it. I hard coded it before the demo but I definitely had a
fudge factor. So yeah, there's always -- it's a voodoo. It feels like it's close though to
electronics programming that I never got into writing software. It's like never is right, but yeah,
I would love to discuss more about how it works, and I feel like people in this room are probably
like you don't have to -- that's way too complicated. But now I want to continue to talk about
clouds which is where we are taking this project. And I want to really introduce this idea of an
infinite conversation and this network that we visualize here is an abstraction of this idea where
each one of these dots is a snippet, kind of like what was in the last video that you saw. And we
have 30 hours of these snippets even after it's been condensed. The things that are potentially
interesting to a lot of different audiences who would have intersection with this world of art
and code together, and we are finding ways of networking them, of saying well, at this point in
the conversation they bring up this subject. And at that point there is all these digressions that
could work as follow-ups. In the same way when you are having a conversation with someone
and you are excited, you have a million things that you want to say and afterwards you realize I
only said a small portion of it. Well, hopefully this system, this generative system will present a
way to explore a conversation where you can exhaust those possibilities or feel like they are
never exhaustible. This traversal is kind of hard to see here, is the idea of one path through this
where you jump from node to node, maybe starting with a search query, saying I am really
interested in how online sharing affects creativity or I'm interested in the dangers of perpetual
novelty, and then the system constructs the story for you and you are at will to follow. You can
watch it or you can take digression. At the moment and this is where it gets pitchy is we have a
kick starter campaign that we launched maybe 48 hours ago to help fund this project and, you
know, I appeal to you on a personal level. If you think it's interesting, we would love your
support and, you know, there’s some cool rewards that we have that I'll show you. But I want
to show to you, to further illustrate the concept and to show you where the next generation of
this research has gone I want to show you the video from that kick starter. Hit the lights.
[video begins]
>>: The idea that when I talk a lot about the fact that our lives can be documented through
data and they are being documented through data…
>>: I just think that it's so much fun to build projects that people can bring their own stories to
and their own perspective.
>>: A real passion for code and impulse to share their inventions unites this community of
open-source programmers toward the 21st century’s pioneers of digital art.
>> James George: So you may be wondering what's up with the way this video looks. Well, it's
filmed using a format we've been experiencing thing with called RGBD. It's video but with
another dimension and it offers new potential for filmmaking. This is the first production of its
kind, a film exploring creative technology while using emerging technology. So Microsoft
released this video game controller called the Xbox Kinect and immediately artists and
designers recognized its potential as a 3-D scanner. The community worked together to make it
available for creative projects. We're both filmmakers who worked with technology and we
wanted to use this device to make movies. Clouds will be the first production to use this
technique. So as we've been developing the film, we've been releasing our code open-source
teaching workshops so that others can get involved. Recently we began to see some amazing
projects come out of this. These are all projects that somebody else made with our opensource software in the last few months.
>>: It's almost like people across the planet are dreaming together. There's this imagination
that's coming about by people interacting with one another and creating this totally different
thought space.
>> James George: We have over 20 hours of edited conversations captured in RGBD and we
want to give you access to all of it, but in a way that can be explored interactively, like
experiencing a documentary in a videogame environment.
>>: I'm interested in that underlying aesthetic of the data. I'm interested in how networks
manifest themselves at kind of multiscale levels. I'm interested in emerging properties. I'm
interested in our brain’s ability to recognize pattern…
>>:… How do you like breakdown the passing of time and let the [inaudible]. I think you could
just like disappear and reappear [inaudible] instantly.
>> James George: In this phase of the project we are building an application to present the
interviews as an infinite conversation. We imagine this working in real time allowing the viewer
to control the camera, flying through space choosing what conceptual threads to follow or who
to watch.
>> James George: So pause it there. [video ends]
>> James George: Because after that we just beg, please [laughter]. So this is again this
visualization of this conversation where we want to present these interviews in this completely
navigable world and we know that we can do it and we just need the time and space and
platform to show it. We plan on distributing this on a Mac and Windows application on these
customized USB drives that we are having laser etched so there's also an art collector sort of
attitude to it or a collectible object that these little capsules will contain this precious data. You
can also download it, but something that we are excited about. These new ideas of transmedia
storytelling and film distribution within media and yeah, so that concludes my presentation.
Thank you for listening. [applause]
>>: Really interesting work. I have a couple of questions. The first one is what are the most
interesting aspects of the RGBD video are actually completely novel ways of thinking about
transitions that I see here in your work? You're basically -- the actual shots of people, they have
a certain novelty factor right now. I don't know exactly how well given the kind of quality you
have right now if you can improve on that, but something that's really fascinating is that you
introduce a completely new language of transition. So far we have cross phase and [inaudible]
silly things, but you are introducing all kinds of results and particles in things like that.
>>: People [inaudible] co-located in the same [inaudible] together.
>>: You can merge particles from one person to the other [inaudible] successfully.
>>: Yeah, so my question is what kind of tools and what are you actually building to support
stuff? Because one of the things is a pipeline that you're showing is actually processing existing
software but your transitions are purely animation. This is stuff that you haven't recorded. It's
kind of some [inaudible] scripting and so there's a complete different part of this that I'm
missing in your presentation.
>> James George: Yeah, I think it's because it's still the part that is nascent and to be explored.
When we do these transitions, again, we work in two modes. We have this toolkit, this user
interface application that's essentially a hat turned Gui turned public turn stuck with it, you
know, and there is no relief cycle system set in place. It's just like publishing this stuff. And
that's actually sort of a condemning middle ground because you are stuck with the default
aesthetics that have been programmed into that interface, so with the kick starter in the next
round and what we've learned from the community is we are taking it two ways. The first way
is building a more robust API so people who are programmers can actually get at the VBO
structure, get at the projective texturing shader, really document how those work so that
people that are working at high level can sort of script these things and make their own things.
Granted that will be a smaller subset of people and for actually disseminating this into culture
and seeing really unique things, we're working on data formats, so we've already done a little
bit of this. Actually when you saw the thing with Aaron Copland [phonetic] he was like a
triangulated form and he was drawing in space, that was done in Cinema 4D and we exported a
bunch of data as OBJ files in sequences and actually with Mya [phonetic] we have been able to
bake the projective texturing into the OBJ file and then export an image sequence along with an
OBJ sequence. So actually our process just becomes a data capture and preprocessing system
to let people who, you know, filmmakers and animators that are more expressive in these
environments really make this stuff come to life, and that's going to be the next, the next year
is going to be full of that. As soon as we release these tools there’s so many people chomping
at the bit to get access to this data in their native habitat basically. So that's where I think will
see more of these interesting transitions. I don't necessarily agree that it's just the transitions
that are interesting though. I think…
>>: No, no I'm saying that it's one of the things that you haven't spent much time talking about.
I agree the other stuff is interesting as well. I'm wondering are you thinking of doing the
opposite? So like right now you are saying that you're trying to export as much of the data as
you can and put them into tools that people are using, but those are not technically
cinematography tools. They are mostly modeling animation tools which is interesting. The
opposite, of course, might be that you actually have prescript animation sequences like the
transitions now in movie editing tools that are applicable to RGBD that you can apply so that
you can have a cross dissolve that is a point [inaudible] cross dissolve that you just apply to your
movie sequence. In which case you don't have to go and invest time to do the… It seems there
is also an opportunity there, not to learn Mya and spend a year getting good about it, but I
should just focus on animating movies in this new media.
>> James George: Yeah, I think that what we've really learned is that you have to play ball with
the ecosystem that exists and I kind of had this bullheaded attitude that I'm going to make my
own final cut for depth data and we got pretty far but making that timeline system was insanely
complicated and I felt like I was losing my like sharpness as far as making art project, because I
spent, you know, like three months making this timeline system. And granted that also has
community benefits and we have been using it in the film. It has interactive implications. So I
think as far as the research strain of publishing this idea and pioneering the thought space of
RGBD as a film format, making it available for creative use in existing 3-D applications is going
to be really where the interesting work happens.
>>: James, your three months spent developing the RGBD timeline editor was made possible at
a residency in Japan, right, at the Yamaguchi Center for arts and music?
>> James George: Yeah, it was actually specifically the timeline. Yeah, I spent three months in
Japan this summer at an arts organization as a technician, so it's actually the opposite. But they
have a contractual agreement that everything you produce there will be published open-source
giving credit to the organization. In fact, I wasn't allowed to push code to anybody's get hub
but theirs, but it had to be public. So it's a really interesting restriction because they are
interested in building credibility within the open-source community as an arts organization that
funds tools, so I spent three months there in kind of like, as a kind of Buddhist programmer in
South Japan eating sushi and doing really delicate GUI code which was -- and I did a few little
art projects too, but that's yeah. But yeah, I do agree with you that this transition idea is one of
the most interesting aspects of it.
>>: Can you speak a little more on the interactive aspects of what you're talking about because
I think I got a little bit confused about what's currently here and what we like and stuff like that.
So the film, the kick starter, that's for finishing up this playback linear sequence with the effects
been generated, right?
>> James George: No the kick starter is for the production of the interactive documentary.
>>: So will I be able to self move the camera [inaudible] audio feeding and [inaudible] effects as
I move?
>> James George: Exactly. Yeah, it's something that we are considering. I think the camera
control will be work within a constraint, like it's kind of like one of these web GL systems where
you are influencing the direction but there is, through narrative tension, but then you still feel
like you have autonomy with head movement. But I think that the most interesting interaction
comes from how it influences the story engine. That you are able to steer it on a trajectory so
that it continues to be interesting, so that's more like each moment these digressions will be
presented to you in the world that the people exist in, and again, these transitions when you
make a transition there's a spatial metaphor that happens where you actually feel the camera
moving from that node to the next node and drawing the conceptual relationship mapped into
space.
>>: One of the things that sounds interesting about this project is that the Kinect data that you
are showing and people see it as new because the data looks kind of weird. So do you think
that if instead of having Kinect data that they had these really bad in general, you had perfect
reconstruction may be people wouldn't be as interested.
>> James George: I think that perfect reconstruction would be so new that it would be way
more mind blowing [laughter] but my response to that is really interesting. I thought about this
a lot because in -- we have this term, embrace the glitch, which means that we are kind of like
celebrating the nascent aesthetic of this and…
>>: [inaudible] add SLR?
>> James George: Great. But then we also embrace…
>>: I've been thinking about this too. I mean it feels like you’re, you’re working out the
aesthetics of the RGDB camera but then you did this other thing where you added this SLR
where now it's like perfect video and I wonder if you can comment about that. It feels like you
went back or hedged…
>> James George: Well, you embrace the glitch, but you don't want to look bad [laughter]. I
mean every moment you are making in person aesthetic judgments about it and I think the…
>>: [inaudible] false medium but you also try to make it bend to your will and look good.
>>: I think maybe I was expecting a slightly different answer, something more like well, it has to
be at least as far as the old media is concerned which is just straight up TV video, it has to be at
least, at least look like that.
>> James George: That is definitely a good answer, yeah.
>>: [inaudible] at least sort of like people don't walk away thinking this is crap.
>> James George: So we are playing in this dangerous territory of aesthetic novelty. Even in
our own research we look back at what we made three months ago and we go oh, that's
terrible, but remembering when we had that breakthrough it's like this is amazing. So we are
constantly like exhausting ourselves with this because it's this research in aesthetic novelty and
I think what's important to realize and how to sort of be safe in that territory is frame your
work in the fact that you are documenting right now and so this, these conversations happened
in the course of a year. At this pivotal juncture of this explosion of the art and technology
scene, a lot of it having to do with the publicity that came out of their role in the Kinect hacking
movement which is something that's become a household, almost a household known
movement which has brought exposure to this community. So documenting these people in
this format right now hearing them talk at this, at their age and at this community, it's, you are
making a document of the moment and so we are hoping that the conceptual answer to this is
that when we look back on this it will feel like a home movie. It will be like a super eight film
where you love the inherent imperfections of the media because it was so appropriate for that
time and really captures that time. So rather than trying to make really slick motion graphics,
something that, you know, only successful because of its aesthetic perfection, we are instead
saying this is the medium of the time and let's capture the voice of right now in that medium
and that will give it a timelessness.
>>: So now it's time to create the future Instagram that takes the 50 year from now and makes
it look like RGBD [laughter].
>> James George: Yeah, it's totally going to be, so on your cell phone in 20 years there will be a
Kinect style filter which takes your depth data, tatters up the edges, puts in artifacts; it puts a
few holes in people's glasses, you know, and yeah, exactly. So…
>>: What are your thoughts about giving more views?
>> James George: We've tried it. The one aesthetic problem with that is that joints and I've
been able to fuse the joints during -- it's really computationally intensive. It quickly makes me
think that I should learn PCL and then I go, oh, I'm just going to go like [laughter]. There's still
an artist side of me that is like a fearfully inept computer programmer that I like hide my code
from people sometimes, so there's a computational problem with that that I haven't been able
to really overcome, but also aesthetically those joints I've never been happy with because as
you get further away from the camera, you get these, the data is like major changes and when
you fuse those I haven't gotten it to look good and so, and also the way we are doing these
shoots, it's so like setting up is so instantaneous and kind of on the go that we really can barely
handle the data flow of having two cameras let alone trying to figure out a situation for having
more.
>>: But do you think that you would want to?
>> James George: I think -- definitely, yeah, we've already done, I've done a few experiments
that I haven't published because they didn't look nice, but it's a direction that I want to go, like I
imagine Bruce Sterling's vision of where you actually just have this full reconstruction and you
can just choose these camera angles and every camera angle looks like a photograph, and that's
fascinated me. Granted, when it gets that perfect it won't be interesting anymore, but that's
like the asymptote that we are working towards, so definitely involves more sensors.
>>: Have you thought about distribution? Is seems that in this age where everything has to be
online and everything has to be on YouTube, the big problem is that these are super data
intensive and I admire your way of distributing them on USB sticks. In some sense it's a little bit
retro right now, but it's completely necessary because there's no real good ways of distributing
this. So have you thought about dissemination beyond kind of the…
>> James George: Yeah, I want to show you a demo that we collaborated with a web GL
programmer and we actually came up with a really hack little format to slip an RGDB, a D
stream into a web M video and we made a zoetrope out of our friend. Let me find this really
quick.
>>: So there are early steps in showing RGDB video in a browser. In theory, in principle it
would be great if clouds generated documentaries within the browser. That be great
[inaudible].
>> James George: It's a fact that like, you know, the web GL is still nascent enough that it freaks
me out if we are going to actually go forward and hey, everybody, fund this documentary that
we are going to publish in web GL, whereas, we have the tools and we have the know-how to
make it work as a rich, you know, desktop application, so I think we will do a follow-up project
to this film and it will have a website and I imagine mixing video content with some web GL
snippets. But that would be an ideal format for distribution.
>>: The only reason that you would do it in web GL is because you want to make it interactive.
>> James George: We want to distribute the three-dimensional thing easily through the
internet.
>>: Okay, it doesn't matter [inaudible] unless you are changing your viewpoint.
>> James George: Yeah, exactly, exactly it’s interactive.
>>: But then so now you are, now there is this line your crossing sort of like what your director
wants, your creative director on this sequence and what the user wants to do and I am
wondering have you thought about, like this is a debate that we are having actually with some
related kinds of projects. We don't know like, there's some effort to say we will just render the
thing to a regular movie, regular sequence and slap it on YouTube and be done with it, but that
totally taking away the interactive component, and then the question is, well as a creative
director you might not want any interaction, so for Clouds in particular, what do you envision?
What is the user going to be doing?
>> James George: I think the important thing is like as a creative director you are an interactive
designer and you present a system that's constrained but has freedom within it and so that the
person feels like they are completing the system by being inside of it. So what that means is
like as -- it's an intuitive interaction so moving the mouse will cause the camera to move in an
intuitive way to like highlight different things so that people can sort of investigate with
curiosity, or even the minor parallax of a shifting camera gives you this feeling of fluidity and
openness, and so that kind of interaction that's constrained that you can still move your head
makes you feel like you are in a world.
>>: [inaudible] in Cloud that you're going to have moments where the camera is completely
not under the user’s control and other moments where it is under the user’s control?
>> James George: Definitely
>>: Back and forth, how is that?
>> James George: I mean it's like a cut sequence in a videogame too. I mean a videogame has a
lot of development language for this like visual cues that show that you are passing; you are
taking control away from the user and showing them this richer experience versus now they
are…
>>: [inaudible] the letterbox [inaudible] letterbox [inaudible]
>> James George: Yeah, like cinematic cues.
>>: Is that weird? It's now a movie.
>> James George: Because this is, so this is Aaron Copland in web GL and he has all of these
little snippets of his interviews.
[video begins]
>> Aaron Copland: My first experimentation with collaboration online was a project called the
Sheep Market. It was basically using Amazon's Mechanical Turk which is kind of like the eBay
for brains. You can pay people and you can get them to do tasks online. So for the project I
thought hey, I'll pay people two cents and I will get them to do this ridiculous activity. I will say
draw a sheep facing to the left and I'll pay you your two cents. And I started collecting
thousands and thousands of…
>> James George: Yeah, this is interactive. I can click, yeah I can go to different guys, different
things and they are all [inaudible] outtakes basically.
>>: Aaron Copland: So the second project that I worked on with -- that's definitely one of the
attributes of storytelling across cultural bounds…
>> James George: And so this has a very slight like camera shifting sort of perspective.
>>: Aaron Copland: Across the planet. We are able to communicate across cultural bounds.
[video ends]
>>: So if you already have this it's a nominal step getting this to a surface [inaudible] that ships
out HDMI [inaudible] $600 [inaudible] projector [inaudible] 3-D after shutter…
>> James George: You mean like stereo 3-D?
>>: Yeah, because you have all of the data and if you have this, is this a [inaudible] I can…
>> James George: Do it. I would love to see it.
>>: [inaudible] if I could grab this because the URL is right there [laughter] [inaudible] as we are
talking [inaudible] the end of the spectrum about [inaudible]. Some clusters you have a
creative director that has a leaning towards the documentary style, the linear [inaudible] then
you're talking fully interactive, but if we are both talking, we are making an assumption on
standard 2-D projections technologies, when 3-D projection technologies are all over the place
had a very -- and dropping in price.
>> James George: I haven't seen it yet. I would love to see it and many people have brought it
up.
>>: If we can talk it three I would be happy to show you some stuff.
>> James George: I would love -- yeah, because these figures would really pop in stereo. They
would just pop.
>>: And the data is all there and so there is no reason for it to be and we could just have them
shake hands and…
>> James George: In fact, this open frameworks direct X, maybe the Kinect for Windows RGBD
toolkit’s release will have direct X supported 3-D, which could be amazing. Yeah, it's something
that it's like, I feel like I need ten of me and I think my, it's about opening my process more. I'm
trying to be as open as possible, but still it's like people want to get involved and I don't have
the bandwidth to like do a knowledge transfer to them through e-mail and then they are busy
and to get this community -- I feel like the community is there and are boiling and I just like,
every morning I get up and I answer two or three RGBD e-mails about people who are trying to
get involved here, and they are having problems with it. And so it's just a matter of finding, you
know, it takes time to build that knowledgebase and do these new experiments, but there's a
whole lot of things that we want to try.
>>: If we could talk later maybe get some [inaudible]
>> James George: Yeah, yeah that would be great. Cool.
>>: Any other questions?
>> James George: I don't know how I'm doing on time. A little bit over.
>>: We should probably pull out of here at about 11:50 or so.
>>: No questions? Lunch.
>> James George: Yeah, thanks everybody. [applause].