>> Desney Tan: It is my pleasure to introduce Golan Levin, who is a
professor out at CMU in computation art, but also in design computer
science and entertainment technology, because of how broadly his work
applies to art, science, technology, culture, so on and so forth, all
things we're trying to do here.
Golan is an impressive guy, renaissance man. Probably more
impressively seems to create renaissance teams wherever he goes. He
currently runs a studio for Creative Inquiry out at CMU, but also been
at the MIT media lab, interval research, Electronica, the list just
goes on and on. Amazingly impressive.
Probably the last thing I'm going to say because I'm standing between
you and a wonderful talk was a citation that I found online from a
recent award he won by Fast Company.
He was named one of 50 designers shaping the future. And the citation
read: If CMU is to technology as Hogwarts is to magic, Golan Levin is
Dumbledor. And it was just amazing.
>> Golan Levin:
too kind.
Thank you.
Thank you so much, Desney.
That's really
I am really glad to be here. You all are one bunch of really smart
Fer's. So I'm a little daunted. Also, when one is preparing talks
like this, you never know who you're going to encounter. So I had
heard that Microsoft engineering was real engineering heavy. Microsoft
Research was like super engineering heavy.
I just came from a presentation I gave to a bunch of engineers, none of
whom at CMU, none of whom had heard of Samuel Becket. And it was
like -- I was like so how many of you know about Samuel Becket. No one
raises their hand. It was like: Oh man. And it was important to my
talk.
I had to kind of do a bunch of backtracking. So it's always hard for
me to initially gauge the audience. Today's talk also I have to say I
have to warn you in advance it's a little bit of a mess. I've actually
been preparing it for a while. But the reason it's a mess is because
about half of it is completely new customized just for you guys.
And it has to do, the first half is basically a talk about me and my
work and sort of my pedagogy and the laboratory I directed, the
projects we're doing there, my own artworks and things like that across
new media. The second half of the talk comes from a discussion I had
with Desney, because I understand from Desney that y'all are thinking
about how to integrate arts-oriented thinking into a research lab
environment.
And it's something I've been thinking about for a long time. Just as
an artist myself, I have about more than 20 years of experience working
as an artist in high tech research laboratory environments, and I also
know a lot of people who also have and have also been thinking about
how to do this. In fact, for example -- and I brought this copy for
you guys -- my laboratory recently put out a book about the history of
like artists in research labs.
So it's the kind of thing, like it's on my mind, it's on the mind of my
friends, and there's a lot of like sort of historical screw ups and
there's also like a lot of like awesome amazing things that have
happened as well.
So with the opportunity to talk to you about arts thinking or artists
in research laboratory environments, I thought it would be kind of fun
to kind of show some new thoughts. But they're kind of still kind of
disorganized.
And one can dig back into the history of this kind of thing to a great
deal of detail. And I haven't found all the cool videos from the 1970s
that I'd like to show you. But you can imagine them. You're probably
Googling them or Binging them in real time.
So I'd like to begin with this just because this kind of introduces
some of my work in a way that talks about a couple of my, two of my
abiding sort of passions and curiosities and interests. Anyone know
what you're looking at here? Blob. Implicit surface. Good.
Yes, you are looking at an open GL rendered surface, but what else?
Anyone got -- yeah.
>>: Should I say it?
>> Golan Levin:
>>: Yeah.
Do you know?
>> Golan Levin: This is my friend James. James, who is that guy.
James is going to be talking tomorrow. And he'll be showing his work
in Kinect-based cinema. So we're here together at Desney's invitation
and also of XBox and entertainment.
So I'll talk about his work.
could be?
But anyone else got a clue what this
>>: Short test.
>> Golan Levin: A little bit, actually. Obviously it's a bit
mysterious. These are -- hey, I'm so glad you made it. These are
hands. But they're a particular set of hands that really means a lot
to me. They're the hands of this guy. I'm just going to switch around
and do the turn off mirroring and actually launch into the
presentation. Play. The hands of this guy. This is Merce Cunningham,
shortly before he died consented himself to be motion captured by some
friends of mine, the open ended group, Mark Downey and some other folks
in the media lab, who have a small crew of arts researchers.
They were basically rigging Merce with motion captures. Merce as you
know basically invented, along with Martha Graham, 20th century modern
dance. There was ballet through 100s of years and Merce and Martha
basically said we can invent a modern language of dance. And by the
time this was recorded, Merce was of such advanced age that he could no
longer dance with his full body. But he could still dance with the
parts of himself that he could dance with which were his hands.
So in this case you can even see the little retro reflective capture
spheres on his hand, Viacon system recording him. And he consented
himself to be recorded this way. Shortly afterwards he passed away.
Open Ended Group invited people to interpret this data. And for me
this was a really great opportunity to think about two abiding
interests of mine. The first being gesture, because I believe the way
we moved through space is something that really helps define our
spirits and helps articulate our unique identities in the world and
space and time.
And also visualization, which is a really interesting problem that I
think also you guys think about a lot is how to take stuff that's kind
of ineffable and make it visible. For me this is kind of a poetic
exercise, too, giving back to Merce Cunningham a body he lost. That
was the idea of taking these particles in some sense and using them to
scaffold a new kind of virtual meat that would become his new body
after he had passed away. So that's -- I show it to you because it's
kind of a quick introduction to gesture, mark making and visualization,
which are a couple of things I really care about.
So today my talk is called new media arts inside and outside the
laboratory. Really honored to be here to present this to you guys.
And it's a talk in two parts. First, as new media arts themselves are
manifest by instance in my own work, pedagogy I'll be talking about, my
organizational activities, and secondly some new thoughts as I
mentioned on arts and research environments. So who is this guy? A
few of you know me. I'm delighted to see some familiar faces here.
I'm an artist. For the past 23 years I've worked as an artist in the
context of various research laboratories, including seven years at the
media lab at MIT. Five years at Paul Alan's former Interval Research.
How many have heard of Interval? That's awesome. Not many people
have. It's an interesting little footnote now.
And I'll show some of my own work shortly but in a nutshell I wanted to
say that I feel that artists, sort of not necessarily proscriptive for
everyone, but it's proscriptive for me. I feel artists obliged to use
the medium of our time, which in my opinion means software and
circuitry as an artistic medium. And to be clear, I write my own
software generally C++ or Java, and I'm interested in exploring the new
forms of aesthetic experience which are made possible by technology.
And also the cultural implications of new technologies and technology
itself. And I generally do this by either inventing or adapting
technological systems. So how many of you are familiar or have heard
of the phrase new media art? Okay. So that's the -- I don't prefer
categories, but when I get categories that's where I'm generally called
and that's the work I do.
I'm an educator, since 2004 I've been at Carnegie Mellon where I've
worked to create bridges across technology and engineering. My home
appointment is in the School of Art, but as Desney mentioned I have
courtesy appointments in design, computer science, entertainment
technology. I'll be showing some of my students' work later in this
talk including some that was supported by Microsoft Research.
I'm also a lab director. I direct a small research laboratory called
the Studio for Creative Inquiry. It's been around for 23 years. I
took it over in 2009. Its mission is to support atypical
anti-disciplinary and interinstitutional research projects at the
intersection of arts, science technology and culture. We support a
wide variety of new media research which I'll show today including some
of which that you'll also see tomorrow in James's presentation, a
forward pointer to James who is a fellow in our lab, who has been
working on Kinect-based cinema. And finally kind of a communist. In
the background of everything I'm going to be showing you today is my
participation in several different communities for commons-based peer
production. I hope that's a phrase you guys recognize, but if not it's
basically hackers out in the world basically grouping together and
working across sometimes very large geographic boundaries on collective
projects involving typically open source code.
If you've heard of, for example, Arduino, it's an example of such a
collective. I don't happen to be involved in that community but I'm
involved in several others like processing and open frameworks that
I'll be talking about. These are widely distributed collectives of
makers who are developing free open source tools for arts engineering.
These are programming environments made for artists by artists that
have enabled tens of thousands of creative people to become software
creators.
So working in the arts and education resources are often very scarce.
Much of my work and that of my students, because we're in the arts and
we don't have a lot of money, depends on the free and open source tools
and technologies. I try to give back to these communities as well by
posting code, by organizing activities that bring the community
together and support that community.
So that's a quick overview. A practice in new media arts. So me now
and some of what I do. Some core concerns are the aesthetics of
interaction of data. I'm interested in basically creating provocative
examples, in particular new kinds of interactions that we can have with
data of all kinds, whether it's data from online or data that we
generate ourselves.
I'm interested also in this idea of expanding our vocabulary of action,
by literally allowing people to discover their own potentials creative
agents in new ways, usually that involve play. So how can people
discover themselves in new ways and how can people become, discover new
voices in themselves for acting in the world.
And those are the kinds of concerns I have. I have some ways I like to
work. Most of the money in the world is down here in the useful
section. And that's fine. That's good. I tend to work over here.
I work there. It means I'm poor. But I work there because actually so
much attention is put on this sort of part of the space that actually
there's a sort of huge undiscovered territory over here that eventually
perhaps migrates leftward and downward, but there's like sort of large
untrammeled territory here and lots of unanswered questions, because
very few people can afford to not look there or can afford to look
there, I guess.
They're all focused on the lower left. So that's one place where I
tend to locate myself. And actually as I'll kind of emphasize again in
this talk, and maybe this is since I had forgotten to add the slides,
this is now the time to consult my handwritten notes and just mention
that there are sort of two trends in new media arts that you'll see
both in my work and that of others, that of sort of the progressive
more visionary and sometimes exploratory and inventive and that of more
the critical, the mirror of society.
Sometimes you can kind of say that's somewhere along here. Is it more
about aesthetics, more about kind of creating quality of life and
creating kind of beautiful experiences or sort of sumptuousness or this
kind of visionary thing that transports you, or is it more about
something that makes you think about the world and yourself in a new
way?
So I just kind of wanted to outline that. Because I think particularly
when we involve artists and research labs, the way that doesn't shake
any boats is to go strictly up here. Right? But a lot of the real
meat of society happens down here. And so it's important to hit both.
>>: Could you say a little something about what interesting means,
because normally I think of that as an insult, like the worst thing you
could say about some art is that it's interesting. Because it's just
so bland.
>> Golan Levin: Oh, okay. All three of these. I think provocative.
All three of these are bland terms anyway. Depending on your
particular bent, beautiful, it's like oh boy. Beautiful. It's like
actually I have friends for whom beautiful is an insult.
Seriously. That's like -- it's like oh great, you're saying it's
meaningless. It's like I have some friends -- I have one friend Marius
Watts, incredible artist, designer. Makes really beautiful objects.
He hates it when someone says it's beautiful because not another
meaningless object. Thanks. He's sort of finally come to love the
beauty and he's like, okay, look, you know what it's beautiful and I'm
proud.
But he's sort of like made peace with the fact he's not doing this like
difficult tactical activist confrontational media, speculative stuff
that really provokes.
But you need both. And I will actually talk a bit more about this,
bookmark for later. The second thing that I couldn't kludge into the
talk at the very last minute was Julian Oliver's critical Engineering
Manifesto, which I wanted to talk about but couldn't -- it was like:
Oh my God it's going to kill them if I add that in, too. So if we get
time, I'll come back to this.
But I think in some ways all these terms are a little bit disparaging,
a little bland. It's actually when we mix them we get useful, get
productive and valuable value. Let's say we get value by mixing these
things.
Also in terms of my own personal mode of practice. So I write my own
software. I write it in C++ and Java, and as I mentioned, and to me
it's a certain pride in sort of DIY. I don't mean DIY like me and my
garage, although I sometimes do. But I actually mean DIY in the sense
I take a certain pride in being an artist who is able to use code. And
that's just my medium. If that's my medium, I should be good at my
craft. So to a certain extent I believe -- and I've been trying to
inculcate in my students a certain kind of arts and engineering
hybridity. They have to be culturally savvy and they have to have good
craft and a good eye but have to make the stuff themselves. And it's
not any good if they're dependent on others, please Mr. Engineer or
Mrs. Engineer would you make this for me. No, they have to be able to
DIY. On the other hand, as you guys know from working in teams, and
it's any kind of substantial project in the world requires, you have to
do it with others. And so a lot of -- frankly, it's more fun. So a
lot of projects involve not working with someone else who is my
technical proxy but, rather, working with my peers who have other kinds
of skills and voices that they bring to the table.
This has to do more with my situation within academia, because it's one
of these questions like how do academics get evaluated. But this is
kind of valuable here, because to talk about because this is basically
making papers. And you know that contain reproducible techniques and
algorithms, pseudo code and other kinds of things. This is about
making sublime examples that go out in the world and change how people
think and change what people think is possible. I tend to work around
this side of the spectrum, although occasionally I write a paper where
I contribute on a paper. And it's just about having a mode of
practice. Like can I create influential examples that sort of awaken
people to possibilities of technology and culture that they were not
previously aware of.
And finally another way I like to work is at the intersection of open
source and pop culture. I work with a posse called the Free Art and
Technology Lab. And what this group tries to do is to create open
source technologies that allow people to sort of mash up culture for
themselves.
So basically these are, we release tools to the public that sometimes
have devastating effects. No. That allow people to sort of take media
back for themselves and adapt the media around them. So we release the
tools and then it's because we're releasing tools that people care
about because they get to insert themselves and inject themselves into
and around media that is meaningful to them.
Okay. So some examples. One way that this kind of liberation can
happen is by allowing people to discover their potentials as creative
actors. I do it through variety of different gesture and audio play
systems and interactive surfaces. This just one example from 2004 -whoops. This is an interactive surface I developed in 2004. It was
just when the sort of surface rear projection, what you might expect
series of flocking animals that form a poetic experience for bar
tables, rear projection. This is 2004. This is not too much of a
surprise for you guys who are producing the surface, the first surface.
But, I mean, so it's just basically how can we sort of enhance quality
of life in terms of a social experience with this kind of augmentation.
Another example is this here, which is a table which presents the
fiction that the table is a musical score. And any objects placed
thereupon become elements or musical notes in the score. So it's got
an infrared camera, tracking the surface, what I call a simple
projected augmented reality, provides an additional layer of
information.
And time goes one direction, pitch goes another. And this kind of
shows that. So it's doing basically kind of a Fourier synthesis.
Gating a bunch of oscillators in relationship to the objects it sees
there. And all of this is sort of virtual projection, indicates
octaves and divisions of time.
If there's a vertical mark, then it sort of opens up all the
oscillators. Get kind of a percussive sound. And it's possible to
make some funky beats and a kind of performance environment. Sort of
like VJing in a way. You're not jockeying other people's videos, but
you're jockeying this kind of visual situation.
Another abiding interest of mine has been visualizing communication and
also visualizing the voice. I'm delighted to see Andy Wilson here
because this project here, which I'm going to have to show, first of
all, it's one of my first projects in 2001 as a professional artist.
But it was concerned with this idea of people's telephones as pixels.
And I noticed you had a paper on that.
And what's happening here, this is going to be dark ->>: [inaudible].
>> Golan Levin: Okay. This is a project, really one of my first
projects as a professional artist, which was a concert performed
entirely on the audience's mobile phones. Basically people would
register their mobile phones beforehand outside the venue, about 200
people in the audience. They would give us their mobile phone number.
And we would generate for them a ticket telling them where to sit. We
would send them a new ring tone to their phone, and then because we
knew their phone number, where they were sitting, what sound they would
make, we would make waves of sound that would cascade across the
audience and make cords dialing people simultaneously and melodies that
would snake throughout the crowd.
And each person's phone would also light up in this kind of atmosphere
become a throng pixel.
[music]
This is our interface for launching the phone calls.
[music]
So we were able to dial up to 60 people simultaneously out of the 200
who were registered into the system. And I forgot to mention there's a
huge projection pointing down at them. So as you would get your phone
call ring, your phone would light up. So basically a big spot of light
from an enormous projector overhead. Each person was kind of an
audiovisual pixel here. The purpose was to visualize this kind of
information sphere that we're constantly within and don't necessarily
always palpably feel. So we're up on stage launching these phone calls
into the crowd with this interface.
Okay. Thank you. Perfect. Another project which was from 2003 with
Zach Lieberman and John LaBarber and Yop Banck [phonetic], a project.
Concerned with visualizing the voice.
In this project here it's trying to present the fiction that speech
cast visible shadows, takes the shouts, the barks, speech and song of
two experimental vocalists who are basically people who make funny
sounds with their mouths for a living and attempts to visualize all of
their activities in this kind of extended cartoon space. So here, for
example, is Yonk Banck, he's an experimental Dutch sound poet. He's
making what he calls a cheek synthesizer. Sounds a bit like this.
Pppffft.
Every single cheek flap there turns into a ball. So kind of rapid
stream of balls that are kind of coming out of his mouth as he does
this.
And here's the basic scenario. You see it's a very large display.
It's in this case a three-screen stereographic projection. Computer
vision system that's identifying where his head is. There's a wireless
mic being analyzed by a sound analysis system and obviously a big
computer graphic simulation that tries to project graphics that are
expressively coupled to speech appear to coincide with the apparent
point of origin. This is one scene in the concert. The concert is 40
minutes long, 15 different scenes that go through a series of changes
and tell an arc or story about these two characters. Here's John La
Barber, you can make out her body underneath this voice sensitive
costume. We had about 15 metaphors by which we could interpret this
world.
In this case here it's pegged to her body via computer vision system.
It's a voice sensitive system that listens to her voice and modulates
its periphery as a result.
Here's a voice sensitive environment. A voice generated environment.
This is kind of another vocal effluvium, this fluid, stem that comes
out of their mouths. Here's one where they're painting with the voice.
Just to give you a quick view of what these look like I'll show two
scenes, this one here.
Now, what I haven't mentioned is that every time he makes a little
sound, it stores that sound in the corresponding ball. When those
balls fall down and hit the bottom edge of the screen they're going to
release those sounds. Creating kind of a chaos. Okay. Now, in
another scene, this one's kind of a like a David Attenboro nature
special and Yonk is speaking in abstract Dutch with kind of abstract
subtitles.
[Noise]
Okay. So as you can see, this is not rocket science, but there's a
great deal of poetics in terms of the considerations happening here,
both in terms of storytelling, if you will, and also in terms of the
visual experience.
After making this, we realized that we were actually able to return
these 15 or so different scenes to the public in interactive form.
Although some of the scenes were really highly tuned for these specific
vocalists and their particular virtuosic techniques. About half of the
scenes can be returned to the public and used in interactive situations
so people wouldn't have to experience it vicariously but could
experience it firsthand.
These were sort of participatory scenarios, these are the children like
three and four-year-old girls in France. And one more time with the
lights.
[Noise].
For me, this is really -- for me, this is actually one of the most
important moments in my talk because what's really happening here he's
more than just kind of having fun with the system. Or discovering what
a system can do. He's actually becoming someone or something else.
He's discovering kind of a new identity himself. In the same way that
one can be transformed as an actor by wearing a mask or a costume.
He's really discovering a voice in himself that he wouldn't have
otherwise. And this ability to discover as I said his own potential as
an actor discover this new voice in himself is a crucial moment for me
that is enabled by interactive media.
[Singing]
He's five. All 5-year-olds learn to sing this song in France. And
then they forget. These are teenagers, they're kind of singing random
notes. The pitch is governed the way the curbs bend, making rising
tones, curve clockwise, making phone tones curved counterclockwise.
I'd like to show another project I did with Yop. Actually leave the
lights low because it's going to be a little bit dark. This project
here is another one concerned with speech visualization. For me it was
a unique opportunity to do something we don't often get the luxury of
doing. Maybe it's an argument for considering sort of high-end content
in a research environment. Here's why. It turns out that Yop is the
world expert in performing this very special poem called the Ersa Nada
by Kirtsch Vitters [phonetic]. The Ersa Nada is a master work of 20th
century abstract concrete poetry. Half hour long poem of purely
strictly patterned nonsense. Took Kirtsch ten years to write. And
probably about half a dozen people in the world that have the thing
memorized, and he's one of them.
So he's able to deliver this really insane nonsense poem that sounds
like just gibberish, very, very fast, and very, very accurately, and
that presents a unique situation if you have something like a speech
recognition system, because obviously if you have a fixed text, you as
computer scientists all know that tracking yourself in it becomes a
whole heck of a lot easier. So this presents a unique situation. We
have basically delivery of a fixed text. Can we deliver subtitles that
are sort of coincident with it, and sort of from intelligent subtitles
that are truly real time, not like some at homeworker in Ohio, typing
several seconds after the fact but actually delivered like really smack
on time.
So the purpose of this concert is again it's half an hour long. It's
to deliver the text of the Ersunada poem precisely in real time in
synchrony with him. It's basically got a kind of score follower and a
syllable detecter connected together listening to him and attempting to
connect to those to propel dynamic typography. Here's the basic scene.
He's delivering this. There's a camera looking at him. The software
turns his image red so I can superimpose text in a sort of clean way.
And I'm here because my speech recognition system and my sort of hacked
together score follower and so forth is 96 percent accurate, which is
great except in half an hour it gives you four percent inaccurate. And
very, very rarely Yop may miss a count. He may do something 13 times
instead of 14, but more often than that the system sort of jumps ahead
or loses its place. And I'm there to kind of keep the lid on things in
case things get out of control. Here's a quick clip of that.
[Noise]
This is live real time.
[Noise]
So we can put the lights up a smidge. Thanks. That gives you an idea
of sort of what I would say the opportunity -- I'll come back to this
later -- to not do a parking lot demo. I'll come back to this idea
later. But it's a phrase from one of my mentors Michael Naimark, who
I'll talk about. When you're in a research environment, you can just
go out to the parking lot and shoot something. Or with maybe
20 percent extra cost, you could actually use really interesting
content that kind of produces gripping results. If you fly out a
little further than your parking lot and collect material that kind of
moves people in a more provocative way. It means the demo can live on
in people's minds and maybe reach more people and also in some sense
present higher quality challenges.
So this is kind of the opportunity presented by this particular
wonderful poem. Some work in full body play is an area I've been
working in for a long time. Actually recently uncovered a video of
myself and Patty Mace's group in 1991, Steve, shows myself interacting
with blobs. I'm certain you must have been there at the time. But
another time perhaps. Here's just some quick experiments in full body
play. This is 2005, me experimenting with a kind of skeletonization.
This is not doing a Kinect-based skeleton but rather a video-based
skeleton using the media axis transform, to think about a way to
anonamyse people and maintain the beauty and articulate quality of
their movements in time and space.
So that is looking at this kind of motion of the human skeleton as
literally computed using a skeletonization algorithm. Here's another
piece I'd like to show. It's called the interstitial fragment
processor. And it depends sort of remembering that in the arts -- my
mother's a painter and she, when I was a small child she said always
look at the negative space. Never made any sense to me. Somebody
nodding his head. But the negative spaces are these shapes that are
formed in and around and in between the body, as shown here.
And when I finally -- when I finally understood maybe at the age of
eight or nine or ten what she meant by negative space, it was really an
interesting awakening for me. And something that stuck with me for a
long time. So here in this project called the interstitial fragment
processor it processes the interstitial fragments. It's just like
that.
It finds them and people can begin to discover the spaces between them
and I want to say almost savor them. And I think that's what starts to
happen here. People even in some sense get to meet each other because
they're savoring the spaces between them.
So it gets really chaotic after a while. There's kind of an
audiovisuality to this, making them physical, making them fall down,
these are folks that don't know each other but they suddenly discover
the space in between them. And they also realize they can do it
without touching each other. They can do it just with the shadow.
This kind of shadow play at the same time as making us aware of things
in the world that we don't normally see.
We forget these shapes are always around us. Okay. Faces and eyes.
How many of you have been to the Exploratorium in San Francisco? It's
one of those life journeys. If you haven't been you have to go.
There's actually a place in Switzerland it turns out it's even better
they make sure nothing is ever broken. That's the Swiss.
When I was a kid I visited the exploratorium and I saw this project
there. It's a simple mix and match game. You guys got it right off
the bat. But you understand what it does. This thing kind of lasted
with me. Always been interested in faces. I think as artists we've
got a lot of, I draw a lot. I draw people and I just have a memory for
faces. So I've been interested in faces for a long time. And in this
case I wanted to be able to do exactly that. Do it with recordings of
people. And particularly with video. So this project creates this
kind of mash up of yourself and your peers.
It's in some sense creating a generative group portrait of people from
the project's locale. Made with Zach Lieberman. This project, each
of -- obviously you're entering video into the system. You get kind of
mixed around with the people who are previous to you in the space. And
it's art made by blinking. This depends on the standard open CV face
tracker, good old Vio Jones knock off built into open CV. Once you
have the unfettered face input, you don't have to put your head on a
special chin rest and so forth, and you can do it unfettered capture.
Once you know where the head is, you know where the eyes are, and you
can use those to do the edits.
So the moment you blink, you create a new set of combinations and you
also finish or terminate the cut you were working on.
Here it is at the San Jose Tech Museum of Innovation, kind of a big
installation there. And here it is in high school in Anchorage. Over
in the library they have a capture station. We made a special
interface for it so you wouldn't have to be out in the cold where all
the faces from the people who were inside the high school are sort of
brought out to put a face on the building. Now, it's really diverse
high school. People of every single color, shape and size. And this
simultaneously sort of brings their identities out to the exterior of
the building while simultaneously kind of anonymasing them and not
making them feel like maybe their privacy is being violated. It's only
been one-third violated.
It seems to be the threshold, I guess, because it's people can
contribute to the system and then this literally puts a face on the
building showing the diversity of the people who are in there.
I mean, how am I doing this? The Viola Jones face tracker. Better
than some of the other face trackers that have come out. You guys have
a good one right now. But this is old technology. None of what I'm
doing is really rocket science. I'm one who picks up the pieces that
fall out of the research labs and say how can I use these in
interesting and provocative new interesting ways to create new kinds of
experiences. Robots are something that is one of the main reasons that
I was attracted to Carnegie Mellon. How many from CMU? Couple.
Cool. CMU was a big attracter to me because of the robotics there.
And I hadn't really done robotics before but it's a real ass kicker of
a discipline. Very comprehensive as you can imagine with software and
hardware. Things fail all the time. And sometimes hard to locate
where. But I was attracted to robotics, and I wanted to come to
Carnegie Mellon to get into robotics and art.
In particular I've been interested in eye contact to try and see if
there were systems that can interact with us using gaze, which is to
say the way we look at people. So this is a robot I made. My first
attempt to break down eye contact into the simplest possible situation.
It's a robot with a single eye that engages people in sort of a variety
of sort of familiar psychosocial eye contact behaviors. Here's some of
the guts. Has just three degrees of freedom, up down, left, right,
blink. And a video showing what that looks like is here.
And so what does it do? It's got a couple of cameras. This is before
the Kinect. It's actually trying to do depth from stereo. And from a
distance it tracks your body. You get closer it finds your head and
looks at you. You get closer it finds your eyes and looks at them. If
you get close enough it's about 18 inches it will attempt to look where
you're looking if you look off to the side.
One thing it does is to get your motor neurons firing it will blink
about half a second after you blink. So this creates kind of a feeling
of empathy. And in terms of the language of the arts, you know,
inverting condition of spectatorship saying who is looking at who.
It's something that really is happening. You do feel like it's
watching you, and that's the feeling I wanted to get.
That one's maybe a bit creepy, this one is a bit more fun. The little
robot. This is the big one. And this is kind of a poetic surveillance
system that I developed using what I describe as a quasi discarded
robot arm from Detroit. In 2008, the car companies were hurting. They
needed cash. And they were getting rid of their robot arms, cheap.
And so we managed to get some robot arms over at Carnegie Mellon that
were basically like dirt cheap, and began to think about using large
robot arms as artistic material. Poetic system big brother half Sesame
Street. It's a robot that tries to present the feeling that it's
constantly surprised to see you.
So it's called double taker. It does double takes. And for me it was
the problem, the true research problem for me apart from the
engineering problem of how the heck do I control this thing which turns
out to be a wicked pain in the butt, apart from that is actually the
problem of real time character animation. Right? Where you're not
making scripts that sort of say, and you don't have any kind of like
foreknowledge of what someone is going to do. It has to be real time
to produce allusion of a character, allusion of life. Well, so it's
looking at you and it kind of is checking you out, looking you up and
down, and you're like what is it, my feet? My hair? And it's kind of
looking at you and it will look away and it will kind of look back like
maybe it missed something the first time. And this is trying to again
produce the experience or the illusion or feeling in us that there's
something in all of us that's worth a second look.
Here's kind of a quick video of that. It shows a bunch of children on
their way to, this is unscripted. They do this. On their way to
Sunday painting class at the Center for the Arts.
So that's just the character it has. It does a couple of other things.
If you bounce up and down, it will sort of mirror that to again get the
sort of feeling of empathy. Okay. Moving on real quick. That's a
quick survey of some of my work a new media artist. What do I teach?
I teach studio art courses in computer science or arts engineering in
which the medium is self-written software but the objective is culture
work, art design, provocation, social critique or innovation in ways we
don't yet have names.
My students create things like information visualizations. This was a
really nice project by a student showing albums that had been sampled
primarily '70s funk and albums that had samples in them, primarily '90s
rap and connections between them so you could find out for any given
album which ones did it sample and any given album in '70s which one
sampled it.
You can actually see this kind of, this reference generation. If you
were in the early to mid '90s you were probably sampling from the early
to mid '70s there. Another visualization made by a student. This is
actually a detail from a much larger visualization of a student who
used a high speed camera and computer technique system to isolate and
record every water drop from a five-minute shower.
Like a thousand frames per second camera, and behind a piece of glass.
And what is the purpose of this? It's pure poetics. But he counted
them all and isolated them all and photographed each one. Another
project, my students work with digital fabrication tools. This student
has written a program to generate cutout patterns for inflatable
structures from any 3-D model such as the Stanford bunny. This is a
cake decorating machine. It's a pick and place for M and Ms on icing.
This is full body interactive work involving experienced design in
generative form. The student project grows trees on your upper edges
if you hold still long enough on the shadow. This is a nice project by
a student.
If we could kill the lights for a second. We'll have a look.
Interactive augmented reality using computer vision in which the
students made a program that interprets a drawing as a control panel.
So you sort of do a -- here, you do a drawing sort of like this. And
enter play mode and interprets it as a control panel. It puts out OSC,
which is a common protocol for signalling between systems and new media
arts, and so you can use this to control synthesizer or anything else
that he likes. So he's got knobs and sliders and switches and so
forth. Just from drawing it. So this came from him from a childhood
experience he had of drawing bogus control panels on cardboard boxes
when he was small.
for big computers.
Sort of pretending that they were control panels
Okay. So this is the kind of work my students do.
My courses often contain students from nine or ten different
departments, including art, music architecture, robotics, HCI, computer
mechanical engineering, every student writes code without exception
regardless of their background. Although I do encourage students to
write code in their environment of choice. So it's a real zoo of
languages. I have everything from like flash to Matlab, kind of in the
same environment.
The arts oriented students learn the skills, the arts engineering
skills that you might call computational thinking in order to implement
their ideas in real working forms. The students from the engineering
and sciences encounter something very different altogether. They're
invited to apply their skills to questions that are driven by their
curiosity, their passion, their senses, even their politics, to
problems which may be more personal, more speculative or more
whimsical, whose solutions are less provable. They're invited to have
a point of view. Some of my students from the science and engineering
have never been asked what would you like to make.
And having this liberty is the rare privilege of being in school. For
some of them it may be the last time they're asked that question for a
while. So I try and get them in contact with this kind of part of
themselves where they have a dialogue with what they're interested in,
because some of them won't be asked that question again.
I wanted to give one actually sort of interesting concrete example of
this. So the Kinect came out in approximately early November 2010. In
February of 2011, my classroom became the first class to my knowledge
in the world that had Kinect hacking assignment. We had 24 students
hacking on the Kinect and making interactive projects with it.
The SDK wasn't out for another six months. So we had to use the open
source arts engineering toolkits which were the only way that my
students could engage with this medium that they were so interested to
play with.
So these are some of the projects, and I am actually going to call one
out in particular, though, because something interesting that happened.
This is a pair of sophomores from the school of art. They're
programming a two-person puppet, kind of behemoth character, where one
person like a Snuffleupagus a horse where one person is the front end
and one person is the back. And so her hands are sort of opening the
jaws of this monster and the other guy is controlling the tail and the
legs back legs and so forth.
It was the first two-person puppet with the Kinect. Look, that's not
earth shattering, you know, solve world peace kind of solution to
anything. But for a pair of sophomores from the school of art who had
only been programming and processing which is a flavor of Java for
about a semester, this was huge. They got 50,000 views online. For an
art student that's a profound thing because prior to this all they had
done was they exhibited their paintings and drawings at the local
school gallery, maybe seen by 50 people, not 50,000.
So it was a totally transformative thing for them to be able to use
skills they barely had in Java programming to be able to engage in
culture make something that 50,000 people would think was interesting.
That was made possible by your wonderful invention, the Kinect, and
also the new media arts community that came together to make it
possible for people to interface that with these open source
programming languages and environments that would allow these kids to
make this kind of thing.
Okay. Some work from my lab. As I mentioned, we support atypical and
anti-disciplinary work at the intersection of arts science technology
and culture. We've run a few conferences among other things concerned
with providing basically bolstering computational thinking. Microsoft
Research. You may know, has invested, thank you, in the Carnegie
Mellon Microsoft Research center for computational thinking which
supported several events I've run that are basically attempting to
examine how people can use passion-driven creativity-driven
visually-driven objectives as a way of bolstering computational
thinking. One of these was the first art and code event programming
events for artists and the rest of us. This is a crowd -- this is not
staged. These are people ranging in ages seven to 70 learning to
program in Java in this case, because they suddenly find there's a
reason for them to do so and they can make these kinds of interactive
visual experiences.
The third art and code conference was held in October of 2011. Stewart
was very kind to come out. This was art and code 3-D. DIY 3-D
visualization sensation, this is the event that was actually going to
lead to the work that James is going to show tomorrow. We wanted to
have a Kinect hacking event and bring together all these people you
surely saw the Kinect hacking phenomenon happen online. We wanted to
bring these people together working on different environments different
objectives get them to talk together.
Now, I said DIY 3-D sensing and visualization. We did talk about other
things like light R scanners and structured light and other kinds of
approaches to doing 3-D scanning. Look, in late 2011, DIY 3-D sensing
and visualization pretty much means Kinect. And we had a lot of
different approaches to how to use the Kinect in kind of interesting
artistic ways. You have a question? Sorry. I thought you raised your
hand. Okay. So this was the world's first Kinect hacking conference
thanks to you guys for support. And it actually had a lot of
interesting outcomes one of which probably the major research outcome
for us was the project that James is going to show you tomorrow. And
I'll show you a fragment of right now.
So some research from the studio. Could I ask you to hit the lights
for a second. This is sort of a forward pointer to what James is going
to be showing. James has been working with Jonathan Menard a trained
documentary in film making on Kinect-based cinema. What it really
allows is for camera work to be something that's done in post
production. It allows you to tell a new story or emphasize aspects of
the story later after the shooting is done by changing camera positions
around a virtual object. Another way of thinking about it is that it
makes video captured people as plastic and as malleable as computer
generated characters from CGI. So suddenly you can have virtual
cameras that can move around these subjects and also serve play with
the data that constitutes the subjects in interesting ways. Here's a
quick clip. I'm not going to ->>: Basically explaining how the database works and say as one point
here and then you go around and there's another, another one there,
another one there, because this is close, this is the one which is
matched. Explains everything like that.
>>: Is there an experience a digital experience you desire to have?
>>: Yes, transportation.
>> Golan Levin: So what you're seeing here is a documentary that James
will talk more about tomorrow. It's called Clouds. In case you don't
have a chance to see it or in case I want to give you a forward pointer
to what it is it's Kinect-based cinema involving interviews with
globally renowned creative coders, globally renowned media artists who
work with code as their material, also curators and critics as well
thinking about new media arts and it's a series of interviews. It's
going to be an interactive three-dimensional generative documentary
that James will be talking about. James will also talk about the
techniques of RGBD cinema which he's developed in order to make this
kind of thing possible. I think you'll agree it doesn't quite look
like anything else that's out there. If you've seen anything else like
it that's out there it's probably because they're using James' open
source toolkit which now has been picked up by a lot of other projects.
So that's one example of work which I'm very proud to have come out of
our lab and proud to have supported in its earliest stages.
James and Jonathan met at the art and code event, Microsoft Research
supported. Another project to kind of give a picture of the diversity
of work that we do at the lab, this is a project called the new artist.
It's a set of robots. And it's concerned with the question of can
robots kind of observe each other and respond to each other. One
robot, this one here, is the performer. It's attempting to ingratiate
itself with the observer. The observer judges the performer and
decides whether it thinks the performer is boring or interesting.
There's your word interesting. In this case, the performer is dancing
around, as you'll see, hopping around, and if it's too repetitive, then
the observer, the audience robot will get bored. If it's too chaotic,
the audience robot will get bored. The audience indicates pleasure or
displeasure by its posture. Sort of about they're sort of in feedback
with one another.
Here's a quick video of that. So there's the performer. There's the
observer. And they're each looking at each other to kind of gauge how
they should respond. This is kind of sweet spot between complete
repetition and complete chaos that the performer's trying to hit. So
it's a completely cybernetic system made entirely with robots. There's
no person in the loop there. And a little bit of progress towards Lego
locomotion as well. He's trying. Okay. See. Look at me. All right.
Lastly, the last project I'll talk about from my lab, we're working
with the robotics institute which has invited us to produce a set of
artworks that are accompanying their mission to the moon. They're
trying to win the Google X prize, $30 million attempt to get to the
moon. And so they invited our lab to produce a set of artworks, one of
which, for example, will be using the rover to draw tracks on the moon.
Some of which will maybe line up anamorphically like so. We hope that
we can produce kind of a drawing program like an applet on the web
where kids can go and have drawings and have the drawings rendered on
the moon. We'll have to innovate some crowdsourcing way of collecting
a million drawings and have people vote on a million drawings and not
only have penises.
[laughter]
You can imagine. All right. Part two. Now this part's a mess. But I
just want to say that as you well know, computers in, I'm going to do
this quick really just a few things I want to say about artists and
research environments. Computers in the '50s through '70s were really
expensive. Dedicated to doing important things. So if an artist
wanted to come along and use a machine like that, it was pretty
difficult. Manford here, still alive in his 70s, he still programs in
Fortran of all things what is renown for doing early sorts of
generative artwork, quasi system you would make the plotter work, to be
able to do this he had to beg and plead his way into using the
mainframe on Friday nights at three in the morning. And especially
because it was the late '60s, and they're like long hair, what the hell
are you doing here, it was a big significant cultural divide. You want
to use my computer for what? Typical thing.
Same with Lillian Schwartz who in the early '70s fought her way into
Bell Labs. She kept showing up, innovating heavily in the area of what
became computer film, computer graphics. To make computer graphics at
the time, it wasn't really a priority for Bell Labs. They were
concerned with more important things.
Nevertheless, a real pioneer in the area of computer graphics and
computer cinema. John Whitney by contrast in the mid '60s was actually
invited to be an artist in residence at IBM. 1966 through 1970, he
created IBM a wide variety of early computer art, sort of generative
and sort of geometric computer art there. He did not do his own
programming. They assigned an engineer to work with him. And as I'll
talk about, one of the -- two of the big sort of models you can have
for artists in a research lab context are the sort of exotic zoo animal
versus sort of free range chicken. In this case he was sort of the
exotic zoo animal, expected to produce these things. This is an exotic
zoo animal amiga commissioned Warhol to be an artist in residence.
The idea here objective here was to bring some cultural cache to this
hardware they were trying to push on the world. And you can find this
video on YouTube. It's pretty funny. Now, one of my mentors is
Michael Naimark, and in collaboration with the architecture machine
group and a variety of others at MIT in the late '70s and early '80s,
he created something called the Aspen movie map. Now, Michael has been
an artist and researcher invented, interested in representing places
for a long time. In addition to his influential work exploring virtual
reality he's known for his advocacy of media art as a stimulus for
technological innovation, having directly helped establish a number of
prominent research labs, including MIT media lab, Atari Cambridge media
lab and Apple multimedia lab, interval research. In the late '70s he
was a fellow at the center for advanced visual studies at MIT and there
he helped create this landmark hypermedia installation allowed visitors
to navigate the roads in Aspen, Colorado.
The movie map was made possible through an artistic abuse of the
world's first laser disk player, namely by taking a device intended for
the storage and playback of movies and instead using it for random
access under interactive control.
That was an artistic idea that he brought to the table and the team
helped create. If you had said to people in 1982 that some day you'd
send cars to every city in the world and record video of every single
street in the world, that would seem ludicrous. But you can see here I
by no means saying they ripped him off I'm actually rather saying this
was an influential project that was shown at a lot of different places
including early SIGGRAPHs, and it initiated a conversation.
I would say it had a cultural influence that many people understand
what it would be like from an interface perspective to be able to walk
around and have a user-driven tour of virtual places.
It may not have seemed possible or even something like you'd want to
have. And in fact there's a fairly interesting story. The Aspen movie
map was funded in part by a federal research grant. In 1980 this
project was given the dubious golden fleece award by U.S. Senator
William Proxmire, a sarcastic recognition he bestowed on projects
he felt were egregious waste of taxpayer money.
So I'm just trying to show an example of how artists in research
environments can help transform the language, the dialogue, the ideas
that are discussed.
How many of you guys know this one on the left? This is Myron
Krueger's video place. Myron Krueger was one of the first artists to
work not only with computer vision but with interactive media,
interactive computing at all. From 1969 to 1974 he developed a project
called video place, basically silhouette-based interactivity.
He's a bitter man. When Sony contacted him, he wanted numerous arts
awards for this, and actually was shown, for example, won the Golden
Ike, the world's largest most significant prize for electronic art in
1990. When Sony contacted him in early 2000s saying look we're
planning on having a camera based gaming system become the eye toy,
we've heard you thought a lot about body-based camera based
interactivity, would you like to consult with us, Myron made a mistake
in my opinion. He said you know consult for you? I'm going to sue
you. And it was one of those things that -- it really presents a
difficult challenge, I think, for new media arts, especially when one
can I think not -- without unjust cause actually say that one's kind of
invented a kind of scope, scope of ideas. It's a problem of the
medium.
Okay. Briefly. Here's some reasons why you might want to have arts in
your research lab. From Stewart Brand in his article Creating
Creating, appeared in issue one of Wired 1992. He provided the
following reasons to have artists discussing his time at the MIT media
lab. He said the deal was very clear. The lab was not there for
artists. The artists were there for the lab. Their job was to
supplement the scientists and engineers in three important ways.
One, they were to be cognitive pioneers, exploring the horizon of the
newly possible. Two, and this is maybe probably the worst reason to
have artists in your research lab, but not a bad one, just the worst
one, we're to ensure that demos were done artfully. That is to say
with presentational craft. It's the least interesting reason let's say
to have artists there. Three, they were there to keep things
culturally relevant, having real artists around was supposed to infect
the place with quality, which it did.
Michael Naimark who made the Aspen movie map I mentioned provided six
reasons in 1999 article which was a speech actually introducing
exhibition of interval research artworks at the San Jose Tech Museum of
Innovation. He described the following reasons why interval integrated
new media arts into their everyday research activities. He said: New
media arts projects, one, provide stimulation and provocation to our
research community adding meaning, entertainment and emotional
resonance to our work. Also they often act as magnets to bring
together unconventional combinations of skills and talents. And three
they can also provide content to test tools and even tools to test
content.
Another set of reasons. New media arts projects are sometimes means
for collecting valuable data. Both through explicit query as well as
observation. May leave researchers down unforeseen paths and result in
discovery and intellectual property. And external deadlines and public
scrutiny which can force for decision-making rigor and completion.
They keep us street smart. Putting on a show as a testbed for new
ideas. A simulation of the real world.
So some reasons from, Leah Buickly, professor at MIT media lab, she
points out that research requires diverse modes of thought. Artists
often think in ways which are concrete rather than abstract. They work
in ways that are improvisational rather than planned. Create things
that are expressive rather than functional. I come from Carnegie
Mellon. I work with engineering computer scientists all the time. As
much as I love my school, the computer science department can sometimes
enforce a monoculture where if you're making things which are
expressive, let's say, rather than functional people are saying why are
you doing that, what's the use? So that already begins to cut off
certain modes of inquiry. And it may be premature to ask that
question. Right? So that's an example of what I mean by sort of
having a diverse set of modes of thought.
John Mida, my thesis advisor, now the president of RISBE, talks about
STEM to STEAM. Trying to integrate the arts into the Stem fields. He
says artists and designers are accustomed to innovating with heavily
constrained resources.
That's kind of a good thing. They're used to dealing with ambiguity
and multi-valent perspectives and they're able to deal with adversity
and volatility, noting that the future world will be warmer, more
resource depleted, more volatile. Mida notes: With greater despair
will come greater hope and greater art. With fewer resources will come
greater ingenuity and greater design. We know the world will be
volatile and uncertain. We know there are no better people to handle
that kind of world than artists and designers.
Lastly, artists can bring critical perspective. And this can be the
hardest to explain and the hardest to see the value in. But artists
can bring criticality to an environment otherwise vulnerable to the
Achilles' heel, the cultural Achilles' heel of techno enthusiasm. They
can help predict shots from the outside. Chat I had with Michael, he
observed this example.
He was sort of saying to himself, if you guys know about the Google
Street View Wi-Fi scandal caused them to have lawsuits in every single
country around the world. Huge PR disaster for them, where they were
collecting information they shouldn't have been from the Wi-Fi. He
said, look, if they were someone culturally savvy on the inside who
also had the trust and the technical respect of the engineers there,
this possibly could have been avoided. He's thinking of himself. He's
someone who really understands about representing places like street
view like situations.
And what he could have done had he been on the inside there if there
were that kind of thinking in place. Well, look, there's two basic
models you have exotic zoo animals like Xerox pair. This was the Xerox
park artist and research, artist in residence program. Artists would
keep the work they would do. Then there was the free range chickens
model I mentioned where the farm owns everything. And here in this
case artists were there to participate in the cultural life. They
weren't artists in residence. They were just researchers just as some
of you happen to be anthropologists or scientists or engineers.
I'm sorry. I've gone long. I'm going to wrap it up there and just say
artists outside the lab are starting to meet their own needs and
they're routing around limitations by hacking your protocols. And
making their own arts engineering environments, processing Arduino open
framework Cinder. These are open frame arts toolkits they developed
for each other to easily sketch and deploy quality interactive systems
these are not commercial products. They're generally fewer than a
dozen people developing each, often even fewer, but there are hundreds
of thousands of other artists and programmers who use these. The
people who work on these environments do so out of love not money.
And I actually wanted to say I think probably one of the biggest
opportunities you guys are missing, if I could give you guys one piece
of like: My God this is such an opportunity you guys are missing here
at Microsoft, like oh my God, it would take approximately one person
halftime to be responsible let's say for making sure that open
frameworks worked well on the Windows platform. There are 100,000
people using this to innovate in high tech ways. If there was somebody
within Microsoft who was lurking on the open frameworks list whose job
it was let's say halftime to do so and to make sure that it was a
pleasure to develop open frameworks applications or processing
applications in the Windows operating system environment, in the
Windows ecosystem, you would be investing in the interest of 100,000
incredibly culturally innovative, culturally sensitive developers,
different than your C# .net developers these are different group of
people. These are people. Yes, question?
>>: The challenge with that -- I agree 100 percent with what you're
saying, is that direct X versus open GL right now. And with Windows 8
it's optimized for direct X which provides features like stereoscopic
out of the box HMI which isn't ->> Golan Levin: But it's an offline discussion we can talk about. But
two quick responses to that. The first is a give and take. I think
that open frameworks environment has no prima facie objection to direct
X. I think to have -- to say oh you're working on Windows instead of
Mac or Linux you have the additional option of having it work well with
direct X. In fact that would be a gift to the open frameworks
community because now we have direct X as an option as well. Right now
there's nobody who is stewarding that. It's not that they reject
direct X it's no one's stepping up to make sure this 100,000 strong ->>: Talk to you off line about this. The open framework forum is a
little more embraceable, DL Cinder came on their forum, saying we would
love direct X port but we haven't had the context.
>> Golan Levin: James has a lot to say about this too. He's deep in
the community as well. There's an opportunity for you guys to make
sure that the Windows ecosystem is like loved by these people. And
currently it's not yet happening. I think it's also a give and take,
too. You guys still make Office for the Mac, for example. It's also
to say, okay, look even if we're, even if we don't have a whole lot of
love for Java we could still help processing out. Even if we don't
have a love for open GL and it wasn't made here, we could still support
that so that people who are working in those environments can have a
good experience in Windows.
Here's an example what creative practice looks like, I don't know how
many of you saw this project, but Jason Serage, this isn't Jason
Serage, he was a computer vision researcher who released a very good
face tracker, kind of a appearance model face tracker out into the
wild. He said basically if you want to use it, here's the code.
McDonald and Castro picked it up and started to make this interesting
application where they said you know since we know where the face is, I
could -- since I know where his face is I could do face substitution.
And I could very easily then substitute Arturo's face for mine. Or I
could substitute, let's see who's next, he's wearing Arturo's face he's
saying. I could be Marilyn.
Now, Kyle -- okay. Busts this out in a day. He busts this out in one
day based, of course, built upon Jason Serage's face tracker released
out in the world using open frameworks, open frameworks powerful gives
artists designers leg up busting out interesting kinds of executables.
He puts this out there. Okay, is this research? The question here is
this is what I mean by cultural innovation. This is the concept that
he puts out there in the wild. You're all smiling because it's good
work, and he gets a million viewers in about, in less than a week. He
gets like 200,000 viewers in a day and a half. From this kind of work.
This is the kind of innovation that represents an approach to hacking
outside of the lab now. This is what I mean by people who are hacking
outside the lab. Something else happens as well. Not only does Kyle
make this fairly provocative demo of what can be done with the Serage
face tracker, he does something else as well. He wraps it up in
another project he calls face OSC. Remember I mentioned OSC the
student who made the control panel who speaks OSC. The OSC is this
open sound control protocol for sending data around. He made it so
that the face data could be sent over OSC. And suddenly a new
ecosystem arose where hundreds of other people were using face OSC to
make their own face applications with processing and open frameworks
and maximus P and super collider and other kinds of ableton and other
kinds of artistic environments. This is now what creative practice
looks like outside the lab. He makes interesting work and he also
makes it possible for other people to make interesting work. This is
what our spaces look like. This is what our laboratory looks like.
This is the open frameworks, ours electronic 2008. Lastly one funny
story. How many of you know this patent? I'm on it. It was from
interval research. Have you guys been following Paul Allen versus the
world? You have been. This was an idea that myself and some other
folks had at interval in 1994 and 1995. It was just the idea that you
could have a kind of agent, software agent that would do Internet
searches in the background while you weren't paying attention. This
seemed kind of revolutionary at the time because usually you'd have to
make an explicit query. I'm searching for such, you would get a
response. This agent making requests in the background, wrote it up.
It's attention manager for occupying the peripheral tension of the
person in the vicinity of a display device. Whatever.
It turns out that as some of you, half of you who have heard of
interval know in some sense nothing came of it. Well, this did. And a
couple of years ago Paul Allen realized it came out of interval. He
realized everyone had been infringing this, and he proceeded to sue
AOL, Apple, eBay, Google, Netflix, Office Max, Office Depot and
YouTube, a large company he's not suing which is pretty awesome, I
suppose, although he conceivably could have what's interesting is that
actually there might be a very, very large set of [inaudible] and a
large amount of revenue might come from that patent. Here's the
interesting thing, everyone on that patent they were not the engineers
and scientists. It was all -- it was the folks from the humanities and
the arts, all the folks who were on that patent were basically -- it
was the journalists, the artists, and a few others who were basically
like culture workers. So I guess the point of my story is that these
can come back to you in unexpected ways, in the $500 million patent
lawsuit that you win several years down the line, maybe.
Anyway, it's just to say that sometimes unexpected things can come from
having these folks around. It's late and I'm so sorry this has gone so
long. Like I said, we had that risk. I'm really grateful to be here.
Thank you so much.
[applause]
I promised you really should see James' talk tomorrow.
an hour and a half.
>>: Same time, same place.
He won't go for
Are you open for a couple of questions?
>> Golan Levin: If anyone has to go you should. I realize it's late.
If anyone doesn't have to go, I'm happy to take questions. Sir?
>>: There's a lot of people [inaudible] [inaudible] so thinking about
why research institutes like [inaudible] artists because of all the
risk you're talking about. But the artist, why not should go to and
come to Microsoft or to whatever. If you have access through open
framework processing, some tools state of the art, why is it in or not?
>> Golan Levin: Kind of a candy store, really. I know artists who
wouldn't. And I know them who wouldn't because they're really
committed to maintaining sort of cultural and financial independence.
They insist on working in open source ways. And they say if I can't
share what I'm doing, if I can't share the code I'm doing with the
world I don't want any part of your system. There are other artists
who recognize that by coming here, they get a chance to have early
access to some incredibly exotic technologies, and explore and be the
cultural pioneers, not the science pioneers necessarily but to be the
cultural pioneers of exploring the cultural possibilities and artistic
possibilities of the high end stuff that you guys are developing. So
that's what appeals to them. It's certainly what appeals to me. Now,
they want to be able to have a chance to explore and play. And also a
chance to participate, because it's not just like, oh, can I use that
thing you made, that was really cool. But also to actually participate
in teams with you folks on determining a technological agenda because
they have something to contribute to that. I hope that's a good
answer. James, do you?
>>: I'll contribute my own thoughts tomorrow, too.
in.
>> Golan Levin:
It will be wrapped
Yeah.
>>: I'm chewing on the notion of artistic and novel. A number of the
things you showed were really cool because you were the first one of
anyone who had done it. And conversely there was a thousand years of
people using oil paints who were not necessarily doing the first cool
thing with oil paint but doing something that touched us or brought new
vision to us or opened our eyes.
>> Golan Levin: That's a great question. For the last 100 years,
there's been a certain string of the arts which is absolutely
preoccupied with continual surprise and continual novelty, has an
Achilles' heel, of course. Nothing gets old faster than novelty
itself.
things.
At the same time it allows you to leapfrog a few different
It means you can quickly become the master of something, because
there's no one else to compete with. It means that you have novelty to
be able to show something in the world and people say wow I've never
seen anything like that, and you immediately get their attention.
And so those are some things which absolutely have appeal. At the same
time, actually Michael Naimark who I mentioned a few times in this
talk, has a great article, First Word Art Last Word Art. Artists who
want to get in the first word.
They want to piss on the territory. They want to basically say aha,
I've claimed this. There's also artists who want to get in the last
word, where you produce such a definitive contribution to an
established forum that people say wow that was an amazing symphony, in
a mature sense. It's the difference between like Brahms. And he uses
like Brahms and Beethoven as examples. Like you get in the first word,
you can sort of define the form. You get in the last word, you can
really make a profound statement and take advantage of all the stuff
that's been figured out. There's different ways of working. And they
have risks.
>>: I guess what I'm thinking the flip side of the Aspen project it was
odd. Today it's Google Street View.
>> Golan Levin: Yeah, in the same way that we forget the identities of
the people who, the artists who made gothic cathedrals. It's like we
forget. Because now we have these cathedrals that's a point made by
Stewart Brand there's a shifting territory.
If you're not concerned with calling yourself an artist but call
yourself a cultural explorer it's a lot easier to handle that.
>>: It felt like things that
technology. So many of them
shadows. It was just one of
of it. But the rest of them
like a big clunky table that
thoughts on what can happen?
were constrained by the display
I was looking at projectors which cast
them. The shadow one which took advantage
they were a pain in the surface. Looked
you couldn't sit under. Do you have any
>> Golan Levin: If you have something cool I can take it. As an
artist you have to deal with the inherent properties of any medium
you're working with. And those properties can be problems. But
hopefully if you're a good artist, good designer, you can make those
not foregrounded, you know.
But your sensitivity to those issues is a testament to your own
understanding of the problem of being an interface designer and making
lovely craft. It's the same way that like for years we had these huge
monitors on our desks. It wasn't that long ago, right? Sumit?
>>: The question similar to Danielle's, but I think it's we encounter
as researchers a lot, too. But when you're out there doing new and
interesting things, often you realize years later that you've made the
influence some technology but it's through some unusual path, and the
law back to you might not be clear, and that could have spiritual
ramifications.
>> Golan Levin:
Yeah, some people are real bitter.
>>: Some are bitter. I'm wondering like when you talk to your students
about -- like this is going to happen to everyone, right, if you're
going to be out there in the world creating new things. And what kind
of advice would you give to people to help them have the positive path.
>> Golan Levin: This is a huge conversation. This is like we're on a
mailing list together where this is like talked about usually like once
a month. It's a huge conversation.
The real winners are people who cannot only get the first word in but
also get the last word at the same time. Let's just take one example.
It might be Sergeant Peppers. There's an album where there was nothing
like it before and nothing like it after. And to have a level of craft
and commitment and purpose and focus and passion and statement where
you create something that people go wow, that was an album that we
won't forget, you know, it sort of defines a new medium and sort of
closes the book on it. And that's really hard to achieve. That's a
real consummate kind of expression you don't see very often. Otherwise
there's plenty of people who get forgotten.
>>: But seems like there are people, when you were describing stories,
there's some people who can kind of, who are able to look 20 years down
the road and I was part of making that happen even if they're not named
per se which seems like a much healthier ->> Golan Levin: May be best to have a Buddhist attitude to everything.
That said, no one wants to get ripped off either.
>>: Exactly.
>> Golan Levin:
I feel so bad it's late.
I'm happy -- please.
>>: You seem a little bit isolated from usefulness to ->> Golan Levin:
That's other people's problem.
>>: I just feel maybe there's a way, actually could glom these three
elements together.
>> Golan Levin: I think that life is short and it's hard to hit all
three at the same time. So I mean, for example, a lot of my work has
been people have said I want to use it with children who have
disabilities. Like this could help people learn how to speak or could
help people who are disabled in terms of giving them -- that's a
real -- actually it's just quality of life again but quality of life
that makes people feel valuable and feel like they can have an effect
in the world. That's useful again. I try and do that. But it's hard
to hit all three things. So James, you ->>: I would add that there's been lightened self interest in making
things interesting beautiful. And is it useful. That's beautiful,
there's the trichotomy you put forth. It doesn't have the dollar sign
attached necessarily. And I think even touches on what we were talking
about before, which is you get value in knowing that you push things in
a direction that you think the world should be and that's your reward,
is seeing, if 20 years down the line you see it productized or turn
into something that doesn't acknowledge your influence on it. It's
like I helped shape this. So it's the Buddhist attitude. But that is
useful.
>> Golan Levin:
much.
[applause]
I have to let you guys get to lunch.
Thank you so