>> Simon Baker: So I'd like to introduce Srinivasa Narasimhan who is visiting from CMU.
Srinivasa has been there now five years, five and a half years. And he's going to be
talking about light transport both vision and graphics applications.
>> Srinivasa Narasimhan: Thanks. Thanks for inviting me. It's great to be here. Okay.
So when you say light transport, this term is used to mean collectively all the propagation
of light trace within a scene, okay? So traditionally and historically in computer vision, we
are focused on a very specific kind of light transport, and that's in a light ray from the light
source straight goes to a scene point and then that gets reflected back to the viewer.
Moreover, you assume that the scene radiants does not change with distance as it
travels from the surface to the viewer. But this is not the end of the story for light
transport, right? So there's many, many mechanisms by which you have transfer of light
from a source to the surface to the viewer.
For example, it could be interreflections, right, so you have light rays bouncing between C
points or it could be subsurface scattering where you have light rays penetrating through
the surface, scattering around and coming out of a different scene point. Or finally you
could have a volumetric medium. So surfaces are not emersed in vacuum, you always
have a medium around it so you're not going to trade the problem as 2 and a half D but a
full 3D problem. Right? So it could be the atmosphere, it could be underwater, it could
be any kind of medium.
So these three different, you know, mechanisms of light transport are what we can call
global component, right? And they can result in very interesting and rich effects on the
scene itself. Right? So -- and it's not as uncommon as vision researchers would want to
believe it is, right? So it's everywhere. So if you are emersed in water, for example, or
bad weather, your skin, it appears translucent, right? So marble, flowers, leaves and
you'll kinds of florids and so on.
So it's all around us, right? So we want to somehow try and understand the different
kinds of light transport mechanisms, try to model, measure, and interpret them for vision
and graphics applications. Okay?
So what I'm going to do in this talk is talk about a few different applications. I'll start by
looking at how to simulate the effects of the atmosphere as light is propagating through
the atmosphere on imagery, right? So for example if you look at images such as these,
you see effects visual manifestations of loss of contrast, dimming and blurring and the
glows around the light sources, right? So at night if you go outside and it's misty, you see
these beautiful glows around the light sources, right?
So here what we want to do is try to simulate these effects for maybe video games or
movies. So once you go and shoot the movie and then you want to add these effects of
fog and haze in real-time. So let's try to understand the computational complexity here.
So in order to simulate these effects, you not only have to look at the direct rays from the
source and the surface but also because you have a medium which has particles, right,
you have light rays getting scattered from different directions and then reaching the
[inaudible]. And this gives the feel of the glow around the light source, right, so the light
raze go somewhere else and then get scattered toward here.
So the same kind of mechanism can happen to a surface as well, so the surface doesn't
get light from just one point in this area but you have this entire volume that is lighting up
the surface. Right? And in this case, it sort of makes the surface look like an overcast
day surface rather than a sunny day surface. Okay? So you have highlights that are
diffused out shadows that are much brighter than usual and so on.
And the final sort of light transport mechanism is this slight blurring of the surface itself.
Okay? So let's try and figure out how to simulate this very simple setup for graphics
applications. So just to understand the computational complexity it looks very simple, but
computationally speaking, let's say this is your virtual screen or the image that you want
to generate and the viewer is looking at a single pixel, and you shoot out a ray out into
the real world, and this is standard retracing, right?
So this ray you can divide it up into small chunks, let's say 50 or so chunks, and then at
each of those you want to find the contributions of the light sources at that point. Okay?
So then what you need to do is just march along these voxels and then compute an
integral along the line of site. So this is what you want to do for finding the scattering
from these light sources. For this viewing ray. Okay?
So then after this integral, you also imagine yourself sitting on the surface and you want
to do the same integration over a 2D set of directions, right? So there's a whole lot of
integrals that you need to compute here. Okay? So if you just do a back of the envelope
calculations if you have a 644 AD image, can we dim the light a little bit? Thanks. That's
good.
So if you have a 644 AD image and you have let's say four light sources and you divide
up your ray into 50 voxels, just an arbitrary number, you get a total of 1.9 trillion
calculations, which is quite a lot because you know, adding fog and haze is sort of
blurring and dimming lighting, but it's -- you know, you're doing this very complex
integrals and as a result you're getting this many computations, right? So our goal is to
somehow make it more tractable in the sense that we want to get exactly the same
solution as before. We don't want to create a trade-off space.
So usual solution would be if you approximate the integral you get lower accuracy but
faster speed, right? Here we won't actually beat that trade-off. And say we want to get
exactly the same solution but at a much faster rate. Okay?
So for this we need to understand the structure of this integral itself. Okay? So let's look
at that integral. And I'm going to just sort of breeze through it quickly. So this is known
as the air light integral and Karl Snider [phonetic] who defined this term because this is
very common, right, this is when you see out doors you see this sort of blueish haze on
mountains and things like that, that's known at air light, okay?
So you have this kind of an integral where let's say you have a ray coming down from the
source to a particle, okay, at a distance D and you write its equation as I node by D
squared which is the falloff from the point light source and you have another falloff that is
due to scattering itself. So most of the light gets scattered away and what reaches is an
exponentially decaying function that depends on distance. Okay? So at this point
something interesting happens, so you have light rays that turn, that scatter, and the
intensity gets multiplied by some kind of a factor, okay? Just like BRDF does for surfaces
here you have what is known as a phase function for particles, okay?
So once that happens, then you have right rays sort of going from that particle towards
the viewer and that again results in another falloff. Okay? So this is sort of the basic
principle of light propagation due to scattering, okay? Now, this is not the only ray that
we need to consider. You need to consider all the other rays that are falling on this line
of sight. Right? So that means then you need to compute this integral over a range of
distances that is from the viewer to the surface. Okay? So this is the final irradiance that
falls on the camera, right?
So now this looks like a simple equation. Now let's try and write these parameters in
terms of the parameters of the problem. What are the parameters of the problem? So
you need to look at how far this scene is -- the viewer is with respect to the point light
source. You need to look at how far the surface is with respect to the viewer. You need
to see this angle. And you need to also have the parameters of scattering that is this
beta which denotes the density of particles. Okay?
So you have these four different quantities on when this integral is written.
>>: [inaudible].
>> Srinivasa Narasimhan: Yes?
>>: Is K one predictor function or it's a family of functions?
>> Srinivasa Narasimhan: So K is -- can be a family of functions like a BRDF.
>>: Okay.
>> Srinivasa Narasimhan: So but you can expand it, for example, using Legendre
polynomials.
>>: Okay.
>> Srinivasa Narasimhan: And you could have any number of coefficients.
>>: Okay.
>> Srinivasa Narasimhan: Okay? So one example versus you know [inaudible] would
be lighting going in all directions equally, you have an isotropic phase function that might
be constant with respect to [inaudible].
>>: Okay. And what are typical things like mist and so on?
>> Srinivasa Narasimhan: So they are predominantly forward scattering, so they are
ellipticals around the point, and then most of it is forward scattering.
>>: Okay.
>> Srinivasa Narasimhan: So you can sort of parameterize that ellipse, the stretch of that
ellipse using one parameter.
>>: Okay.
>> Srinivasa Narasimhan: Okay. So if you write down this equation with the parameters
of the problem, you get really an ugly looking thing. And there is no known analytic
solution to this and that's why in graphics you're trying to numerically integrate this
equation, okay?
So what we did in this work, this was way back in 2005, we tried to sort of factorize this
integral, we studied this integral a little bit more and tried to substitute different kinds of
variables, played tricks with it, and we came up with a nice factorization that said this
integral that depends on four programs can be factored exactly equally so it's no
approximation into a 3D analytic function so that you can extract out some of the
functions out of the integral, and what you are left with is an integral that depends on two
parameters. Okay? So in this integral is precisely the complexity in scattering. Okay?
So once you have that, this looks like this, okay? It looks like, you know, sort of harmless
looking function, so I have a [inaudible] the two parameters, it's an integral of an
exponent of U and something. Okay? So if you just, you know, simulate this, it looks like
a very smooth function in the parameters, okay? So what you can do is you can
precompute this once for all so it didn't depend on the scene. So you just computed once
and stored it in your graphics hardware. Okay? And then what you need to do when you
are rendering is just pick values out of this. Okay? Yeah.
>>: [inaudible] I mean, is there some physical explanation?
>> Srinivasa Narasimhan: So there is a -- and so it's a little bit of weird explanation in
some ways. So it -- it depends on a particular transformation of these kinds of variables
together so one axis depends on the angle with respect to the viewing ray. Okay? And
the other axis depends on what is known as optical thickness, which multiples the density
and the distance together. Okay? So both axes are physical quantities. But what you
need is for any particular scene you just pick the right, you know, pixel from this texture.
So it's purely mathematical that if you substitute, you know, variables here you can sort of
separate it out into things that change outside the integral and things that change within
the integral. So once you have that, then your computation becomes extremely simple.
So this 8-0 is this analytic function that I talked about which can be computed very quickly
and you have to have two lookups of this texture, and you subtract these two things you
get your integral of that line of site. Okay?
So this makes it extremely fast. So you can store this, it's compact so you can store this
in graphics hardware. It's realtime cheap evaluation, right? And you can arbitrarily
change physical parameters, so you can make it interactive, you can change any
parameter you want, right? So all of this districtwise and new lookup like that. Okay?
So I should mention that this is now implemented in NVIDIA graphics card so if you
actually buy a PC with an NVIDIA graphics card you can download their STK and you
can say add fog and haze and it will do it in realtime, for any of your movies and video
games and so on.
So let me show you some of the videos here. Okay. So what I'm going to do is show a
few different videos. The first one is just sort of a special case where you have just a set
of point light sources like lamp posts and you want to see the glows around light sources
as you are changing the amount of fog. Okay?
>>: We start by showing a simple scene with three street lamps. Under clear day
conditions, these are simply point light sources. The legend on the bottom left of the
image indicates the frames per second while that on the bottom right indicates the
density of fog or scattering coefficient data ->> Srinivasa Narasimhan: You'll see these numbers better as the [inaudible].
>>: [inaudible] zero. As we increase the density of fog, glows appear. Artifacts in the
glows in this and subsequent clips are due to video compression.
As we decrease beta, the glows become smaller and as we increase beta they become
larger.
>> Srinivasa Narasimhan: Beta is just the density of fog.
>>: Because of the attenuation of light from further sources the glows become smaller as
we move farther away.
>> Srinivasa Narasimhan: So ->>: Why does [inaudible] blurry [inaudible].
>> Srinivasa Narasimhan: Yeah. This is just a CG scene where there's no depth of field
of fact or anything. So this ->>: I thought the fog would see some blurry effects.
>> Srinivasa Narasimhan: Right. So in this case what we are showing is just from the
light source. We are not changing anything with the scene yet. So that's a good point.
The next step is to try and understand the effect of surface shading, okay? So we looked
at how light rays goes from the source to the viewer, now we need to look at how they go
from the source to the surface. Right? So here again you have this sort of 2D set of
integrals, right? And what we can do for example is you can take a BRDF and
decompose it into let's say diffuse and specular that is always done in vision and then
you can compute precompute tables the texture maps for each of these components.
Okay? So if you have a measured BRDF that has, you know, different components and
spherical harmonics, you can do one for each, right?
So again, you will of a set of tables that you precompute, could be two in this case, and
then you start rerendering again. Okay? So I'll just show you that video.
>>: Our model can also be used to interactively calculate the change in surface shading
due to scattering for the first time. Here we show a vase with both diffuse and specular
components under clear day conditions. First we see a brightening of dark regions at the
bottom as we increase the scattering. Note that our method is fully interactive allowing
light sources, viewpoint and geometry to all be moved arbitrarily. Here we move the light
source seeing the changes in shading.
We now decrease beta to return to clear day conditions. As we increase beta again note
the diffusing of specular highlights due to scattering, followed by the dimming of the
overall intensity due to attenuation.
>> Srinivasa Narasimhan: So that's sort of just showing what happens locally on a
surface, the highlights get wider, they get dimmer, and the shadows become brighter.
Okay? Yeah?
>>: The first video that you showed of a lamp post for the light source. Like one of the
lamb post would include the light source for a certain pixel essentially.
>> Srinivasa Narasimhan: Are you saying ->>: [inaudible] when you have a foggy situation you have these like lines in the fog
where a --
>> Srinivasa Narasimhan: Right. Right. So if you had just a small field of view for
example for the point light source. Right. So that's a much harder problem to deal with.
So here we are assuming it's a blind light source.
>>: Okay. Invisible everywhere.
>> Srinivasa Narasimhan: Yes. So we did some experiments, for example you have -you can come up with simpler approximations to create those shadow lines in fog. But
the traditionally you start with a point light source and then you cut off the angles that
way. So, okay, so the full model looks like this. So you have the first model that looked
at the glows around light sources and you have the surface shading. Okay? So you
have three lookups basically, just in terms of computations for one for -- two for the
BRDF, one for the air light, okay? So if you can back to our original setting where you
had a 644 AD image and four light sources, you have this many lookups and this many
analytic computations where you had this as a number of pixels, number of light sources,
and this is the number of terms that approximates the face function. Okay. So it could
be one or two in general.
So bottom line, this becomes 15 million instead of 1.9 trillion. The nice thing is there is no
reduction in accuracy. It's exactly the same integral solution. Okay? But arrived at much
faster. Okay? So let me show you a video which includes all of this.
>>: We have seen the types of effects our model can produce. We now show how these
effects can be combined in a scene with complex geometry and four light sources.
Again, we start with clear day conditions. Now we increase the scattering coefficient beta
of the medium. As we decrease and increase beta, we clearly see effects like the glows
around light sources and the change in shading on the vase and pillars. We can also see
how the appearance changes as we modify the intensity of individual light sources. Now
we show a comparison to the open GL fog model. That model does not capture effects
like the glows around light sources or the effects of scattering on surface appearance.
We can change our viewpoint and move around the scene interactively. Here we walk
through the corridor in realtime.
>> Srinivasa Narasimhan: So that was sort of an example where you have a complex
scene about a hundred thousand triangles and most of the computations ask for
rendering the scene itself and not for the fog, okay?
So and also we showed several different applications where if you want to do a
precomputered radiance transfer how would you include fog in it? If you had a [inaudible]
map of lighting how would you change the convolution that is there for lighting between
lighting and surface radiance you have fog in between, right? So you could write it as a
convolution between the lighting fog and the surface. Right? So there's several
applications that we showed by just, you know, using this kind of an integral. So this
point let me sort of -- oh, this is one thing that I always like to show that this is the entire
code that is required to do this. And so that makes it very easy to include it in your
shaded code or fragment shader for example and add fog and haze to videos and
realtime. Okay?
Yeah, okay. So basically this was just -- you know, this was done in 2005, and I just like
to show this because this gives you a way of how to computationally efficiently render
things. But there's been a lot of work in trying to render volumetric scattering for example
by Hendrick [phonetic], you know, subsurface scattering and so on. Where
computational complexity may not be the issue but you want to create beautiful looking
pictures. Okay?
So there's been significant progress in that domain, which could be smoke rendering,
wine, and things like that. But the next topic that I'm going to discuss is this issue of how
to get the parameters for these renderings. So for example, what are the optical
properties of wine or smoke, for instance, right? What are the numbers that you need to
input to your algorithm to get those nice pictures?
So typically the accuracy nowadays is limited by the input parameters itself. Just like in
vision, there's been a lot of work in measuring BRDFs, for instance, right? So you want
to get as realistic as possible to the real world surface you want to measure the BRDFs.
So here unfortunately in volumetric scattering there's not much work, okay? So there's
no table that I can go to in physics literature and plug out numbers from there for the stuff
that we are interested in. Okay? So what we did was we tried to come up with a
technique to measure the optical properties of liquids. Very simple setup. So you have a
camera that's looking at a glass tank. So the glass is all antireflection coated so it doesn't
have stray reflections. You have a bulb in this tank that is precalibrated so we know the
radiance of the bulb in all different directions. What you do is you fill this thing with the
liquid that you want, okay?
So we did experiments with lots and lots of things. We went to the local supermarket and
we got, you know, wines, beers, Cokes, everything. Okay. So my student was working
on this, really loved this project because it was 23 liters of four different kinds of wine or
something. So this things hold about 23 liters of the liquid.
Now, one thing to note is that if you fill this thing entirely with let's say milk, you would see
anything. So light transport at a basic level you can think of it as blurring the incident light
field, right? If you blurred it significantly you can't recover the property as well. Okay?
So that's one of the major problems in light transport analysis and it's -- you know, it's
been studied in physics for a long time, so if you look at milk here and you have the light
source there, you don't see the shape of the light source at all. And you were talking
about the blurring. Just completely blurs out. Okay? So estimating things from such an
image is ill-conditioned, and it doesn't have to be unique as well. Okay? So it's a much
harder problem to try to invert this.
So what we tried to do is we wanted to side step this problem completely. And measure
the liquids in a state where light transport is simpler. Okay? And one way to do that is to
actually dilute the liquid. So if you dilute the liquid significantly or enough, you would be
able to still see sort of the shape of the light source. It's not enough blurring there. But
you have enough signals so that you can estimate the properties. Okay? So let me
show you a video here of milk in this tank as I'm increasing the concentration. Okay? So
we have filled this tank with water and we add little by little milk to it. Okay?
So that's -- sorry. That's the video. And you can see as the concentration is increasing
you can see the light source shape even now, but then it becomes more and more
blurred. Okay? So until the point where you can't see anything and everything is
washed out. Okay?
So the idea is to dilute media sufficiently, okay? Now, of course this word sufficiently is
the catch phrase in some ways, right? How much do you dilute? How much do you, you
know, milk you require a lot to dilute. For example, wine doesn't scatter much, so you
don't have to dilute much. Okay? So we came up with the technique to do this. I'm not
going to go into the detail here, but let's for now assume that, you know, you can find out
how much to dilute and then we have this kind of an apparatus, so focus your attention
on this ray diagram that I'm going to show here. And this is a typical image that you
capture, okay?
So if you look at a single scan line, this corresponds to a plane in the 3D box, right? And
if you look at this single point, the brightness of this point is a result of an integral which is
similar to the integral that I talked about in the first part. Okay?
>>: Are you [inaudible].
>> Srinivasa Narasimhan: No. Doesn't have to be point light source. So here we are
just going to try and find the integral due to the rays that are going out from the source.
So it could be done purely numerically. We don't have to use the exact same model that
we used before. So a single intensity is a result of that integral. And if you scan this line,
you're measuring all these different kinds of rays. Okay? You're measuring different ray
lens and the angles of turn. So a single image, now if you scan from top to bottom would
be all these angles, right? So a single image contains a lot of information, contains a
whole range of scattering angles, it contains a large range of [inaudible] of light rays,
right? And you have combinations of these.
So every pixel is giving you that information. Okay? So a single image has redundant
information, okay? And what we showed in this work is that you could -- if sufficiently
diluted, you could recover the parameters of the problem uniquely. Okay? So that was
sort of the contribution of this work, that you could invert single scattering expressions.
So then we went out and we got lots of different kinds of media. So alcoholic beverages,
different kinds of coffees, fancy coffees, simple coffees, different kinds of milks, juices,
soft drinks. We also got detergent and shampoo, anything that we could lay our hands
on. And more seriously, we wanted to study ocean water itself. So this experiment I did
when we went to San Diego during CVPR 2005, I think, and we got in -- we carried this
class tank to the hotel room and we got water there and we tried to measure ->>: Do you have any problems with the water being too clear or any [inaudible] being too
clear?
>> Srinivasa Narasimhan: So I'll show you a space, the full space of the liquids and you
have very clear and where you have murky and really murky and so on, right?
So ->>: But if [inaudible] pretty clear, would it need a really big tank to get kind of that subtly?
>> Srinivasa Narasimhan: Right. Right. Absolutely. But the thing is that -- yeah of
course. The image that you capture is not a single image, right? So you capture a series
of photographs for different exposures, linear lies it, get the best possible intensity
resolution and so on.
So we typically work with 20 bits, effectively 20 bits of resolution. So, you know, these
are some of the images that you get for different kind of media. And you can see the
different amounts of glows. You can see some of them don't have any glow, it's an
attenuation, different colorings and so on. Okay. So here's the set of images for just
scattering media. So, you know, expresso, cappuccino, orange powder, that's Tang,
milk, we know that it scatters.
But I don't know why one would want to find this out, but pink lemonade power also
scatters quite a lot. Okay? So on the other hand, you have images like these where you
don't have much glow, there's not much scattering but just attenuation. Okay? So this
could be the chardonnay wine, the Merlot, the Cokes or the grape juice and things like
that. And ocean water doesn't scatter that much, unless you have a lot of impurities in it.
Okay.
>>: [inaudible] brown sugar or [inaudible].
>> Srinivasa Narasimhan: Right. So there's an alignment problem and you capture a lot
of images. So it's just a issue of high dynamic range alignment.
>>: And you can't compensate for that?
>> Srinivasa Narasimhan: We try to do as much as possible to do it. But you know, that
doesn't turn out to be that much of a problem. So for this one you don't really care about
the exact boundary. If you have pixels here, you are fine.
So if you want to plot all the liquids in this single plot where on one axis you have the
scattering power, on the other axis you have the absorption power. Okay. So we know
that milk's scattered quite a lot, absorb less, right, so you could have -- on the other end,
you could have wines and Cokes that absorb quite a lot but scatter less. You could have
obviously the chardonnay Coors light would not do anything. Sprite would not do
anything. So it's very less absorption, very less scattering. And there's this middle class,
if you will, that has modern amount of both scattering and absorption.
>>: [inaudible].
>> Srinivasa Narasimhan: Yes. So it depends on how many of these spoons you put in
this.
>>: Right. So [inaudible].
>> Srinivasa Narasimhan: Right. So that's normalized for the concentration. So
everything is normalized for the concentration. So the concentrations you can basically
get the parameters out and then you can extend it to whatever concentration you want.
That will be more scattering or less scattering.
>>: Does the orange juice really have that much of a -- it's clearer here than [inaudible]
orange juice actually looking like -- is that just because ->> Srinivasa Narasimhan: So this is an orange powder. We have just put a couple of
spoons in it. So it's not as much as orange juice for example.
So you know, the sort of idea here show the table of numbers that we had in SIGGRAPH
paper where we have the parameters see, the way to read is this you have the
coefficients that are related to extinction or absorption, the coefficients that are released
to scattering and these are parameters of this phase function, okay? And each one is for
R, G and B. Okay? So you have 9 numbers that you need to input to any algorithm.
Okay? And then you can render with any favorite, you know, volumetric rendering
algorithm.
>>: [inaudible] roughly speaking yet it's the [inaudible].
>> Srinivasa Narasimhan: Yes. Yes. So it's just like the BRDF but it's one parameter.
Okay. So most of the phase functions for example, if you see most of them are point 9 or
point 8 or something, so that means that mostly forward scattering and elongated.
>>: [inaudible].
>> Srinivasa Narasimhan: What is that?
>>: [inaudible].
>> Srinivasa Narasimhan: Oh, yeah, I mean it's -- so either it was totally unreliable that
way, the scattering coefficient is zero there. So the scattering coefficient is giving you the
integral of the phase function. It's like the [inaudible] of the surface, okay?
So if the scattering coefficient is very low, that's meaningless in blue.
>>: So Coke and Pepsi pretty much absorb all the [inaudible].
>> Srinivasa Narasimhan: Right. Right.
>> Srinivasa Narasimhan: So, anyway, just to say that the measurements were actually
very accurate in terms of you get most of them with one or two percent error with respect
to the measurement that we have made. So let me show you a few renderings. So the
renderings I'll show you are with cognac glass with the liquid in them and illuminated with
this kind of a kitchen and one [inaudible] map. Okay? So the first couple of them are the
wines. We have the Merlot and the chardonnay, so you have the deep red color, and you
have this sort of see through chardonnay, and you also have to look at the caustics, for
example. So this doesn't have that much of a caustic, although there's this faint red thing
because most of it is absorbed. Okay? So here the caustic is much brighter because
you can see through it. Okay?
So you can do different kinds of liquids. So this is England here, which is brewed in Ohio
and I was for the longest time I thought it was Chinese beer because it was one of my
Chinese under grad students would do this experiment. But of course this doesn't have
the fizz and so on that usually comes from beer or Coke, for example. But you can see
the color of Coke is different from the color of wine for instance. Right? So and this one
is skim milk. Okay? So you can see sort of the soft appears of milk.
Now, just to show you these colors a little bit better, I'm going to render them in sort of an
artificial environment where you have a single light source, very strong light source. And
just focus your attention on the caustics. So here you have the beer, can see this sort of
bright orangeish caustic and you have a very kind of a caustic for Coca-Cola. So
depending on the absorption properties. So the fact that we need to get these absorption
and scattering properties very accurate is that if you actually get them wrong in the
second decimal place milk would appear pink. Because remember what we are doing is
we are sending out a million photons through the volume and scattering them individually.
If you get those scattering things wrong, eventually you'll get everything in a different
color. Okay? So getting those parameters was very crucial and that's sort of the
contribution in this work in that, you know, people could have input some random
parameters but you couldn't tweak them, you have to wait for a day or so to get an image
back. So you can't just do this heuristically. So here's wine and orange powder. So this
is some random concentration of the orange powder. And we did, you know, diluted
strawberry shampoo, and this is detergent and [inaudible] so you can see all of these
nice red and blue caustics.
And finally since you have the parameters you can play games like you can add 75
percent espresso and 25 percent milk to make it light brown coffee, right? So the idea is
to not blend the images but to actually blend the parameters. So that is the physically
correct thing to do. So you're mixing the media. All right. So -- and I don't know why one
would want to do this, but so in this case we are averaging wine and milk, but we are not
assimilating any other chemical processes for curdling and things like that, but just as a
fun thing.
So here I wanted to show smoother transitions. We didn't want to go from water to wine
but the opposite direction. So let's -- so you should see in this video it starts with wine
and it slowly dilutes. So you can see as it dilutes the caustic comes up and it becomes
pure water in this case. So all we are doing is just decreasing the concentration in
parameters.
And then it becomes sort of murky and then it turns towards being milk. So again the
caustic disappears. And then finally it goes from milk to being light brown coffee and
then after all the milk goes away it will become espresso. Now, you can see some of the,
you know, noise and the rendering. So that's because you have to allow it enough time
to get to that level. So, you know, you render these things for two days and then you lose
patience and you show that thing. But you it's a really computationally challenging
problem to do this in realtime. Okay. So just to sort of summarize this idea, we
measured the parameters at these concentrations, so this is diluted wine, this is sort of
murky water looking, this is milk. So that's beer and that's Merlot. And these are the real
concentrations. So once you measure them, then you can increase them. So you can
see the really soft appearance. And this is I think hundred percent fat milk. So we can
see they really take soft appearance of milk. Okay. So here we looked at, you know,
modeling and then measurement. Now I'm sort of switching gears on to try and do
interpretation for vision applications. So during my PhD I worked on this notion of vision
in bad weather, taking images passively and then trying to remove the effects of weather.
So the nice thing to note that is that simple image processing techniques will not work
because the effects of weather depends on the distances. Right. So as a bi-product you
also get 3D measurements from fog. Okay?
So assuming that the fog is uniform. Okay. So here what I'm going to be talking about is
going underwater. So you have active illumination. So either it's at night or underwater
where you don't have natural illumination where you can control your own lying. Okay?
The previous case of passive imaging was that, you know, you're stuck with whatever
you have. The image is completely blurred out due to fog, you cannot do anything about
it, right? So if you have control over the lighting, you should be able to illuminate the
scene in a more intelligent manner so that you can see far ahead. Okay?
So one of the things that you should know is that you can't just use a very bright light
source underwater where it's flood lighting the scene. So you have light rays going
everywhere and everything comes back at you. Okay? So flood lighting is a bad idea.
You can't really see anything there. So this is the reason for that is that you have the
sensor viewing volume and you have the lighting viewing volume that is intersecting here
and that's lighting up, right? So if this becomes larger, your contrast goes down. Okay?
So a recent technique by [inaudible] there was a very nice paper where I'm sure you guys
have all heard about it, this is using high frequency illumination to remove global effects.
To get direct. But that doesn't really work that -- theoretically it's fine, but it didn't do that
well a job, you know, in this case because if you think of spatially varying this lighting, still
the common volume is big okay. So for example I'm going to show you an image where
this is flood lit and just placing, you know, contrast chart and you have some dilute milk in
between and you're capturing the image, everything is just really bad and there's no
contrast, and if you do this direct global separation, the direct image looks like this. It's
really noisy because what you're trying to do is you're changing the on and off pixels,
right, but nothing much has changed in the image. So it's like subtracting two large
numbers. You get noise. Okay? So and the final -- the image we're just -- you know, the
global component would just look like the original image itself, like a flood lit image.
Okay? So what we want to do is to not actually spatially throw out light in all directions
but a better idea is to actually focus light in one sheet and then scan it. Okay?
So in that sense, your viewing volume intersection decreases. Now of course this
increases the time of acquisition for sure. And it also says that if you have noise in your
sensor, you need to have more integration time, right, so that you just have a small piece
of light, right, if you don't have enough power.
So it's a trade-off between how much acquisition time you are willing to sacrifice for
accuracy. So what you can do is so this brings us to the point of using light stripe range
finding underwater, okay? And this is for a long time in underwater imaging people have
been trying to use light striping but have not been successful because of the scattering
effects, right? So if you look at light striping and in clear air, right, so you have a sheet of
light hitting the object, and a then you have this sheet scanned across, right, and you find
the intersection typically it's very easy, you threshold and you find this intersection, you
can reconstruct the surface.
But if you look at what happens in a scattering medium, so the light plane itself lights up
beautifully, okay. So this is the scene that you would -- you know, if you go to a movie
theater and you see the projector beam itself, right? So that is what is happening. So
you have a light plane that is lighting up and then you sort of scan it along, and one of
those light planes look like this. If we try and model the brightnesses along this light
plane using the scattering and then try to find this intersection, you can create a template
to detect the intersections more reliably. Okay. So this is much better, for example, in
comparison to a floodlet image like that, right?
So what you can do is so you can write the equations exactly as in the first couple of
parts, so you look at a single light ray that is going through the medium and you have
attenuation along X and Y and you have the same alpha, right so. In the first part it was
K of alpha, here it's P of alpha. But it doesn't matter. It's exactly the same equation.
And then you have -- so in the phase function you could write it as a one-parameter
function, so P of alpha could be replaced by something like that, where G is your
parameter. Okay. So another thing you need to think about is that this light goes and
hits the surface and you see this bump. And then that radiance gets attenuated by this
distance. Okay?
So these are simple equations for the scattering. And you combine the two. So you have
something that is going like this and then you have a bump. That's what it is. So if you
look at it in terms of a plot, so when there's no scattering, you don't see the light plane
itself, you see a delta function, right? And then as you increase the scattering, you get
those things, right? Now, of course you can use this falloff to find the medium parameter,
exactly like we did in the previous part, okay? So you can do that and 3D by triangulation
or temporal analysis is similar to what you do in clear air, once you detect those
intersections. Okay?
So the medium parameters -- yeah?
>>: I have a question. The image where it says no scattering, why are we seeing
something that looks like flowers to the right of the screen?
>> Srinivasa Narasimhan: Right. So there's some kind of an interreflection, and you
know, it's water, but it's -- it has a little bit of scattering. So ideally, you should see a
black image.
>>: Okay.
>> Srinivasa Narasimhan: So you have let's say this stripe is very bright and it's actually
sending light rays out to another object and there's some interreflection as well. So once
you get these parameters, then you can sort of, you know, create a [inaudible]
appearance of the scene, so you can undo what has been decorated, okay.
So let me show you a video for that.
>>: When you have scattering [inaudible] won't that be more of an issue?
>> Srinivasa Narasimhan: Yes. Interreflections will be an issue. And in this case, we
are sort of ignoring interreflections. We are only looking at scattering in the volume,
okay. Interreflections across the scene points will be an issue and that's my last part of
the talk. Okay.
So here let me just show you a video.
>>: Our light stripe range finding experiments were done using a glass tank filled with
scattering media such as dilute milk, a projector and a video camera. The desired
objects are placed within the medium. The calibration ->> Srinivasa Narasimhan: So I think calibration is quite straightforward, but ->>: When the tank is filled with dilute milk, the adverse effects of scattering are clearly
seen in the floodlet image of the scene. Now we show light stripe scanning in dilute milk.
Notice the light plane itself becomes brightly visible [inaudible] ->> Srinivasa Narasimhan: This video didn't do justice to how beautiful it looks in the real
world and [inaudible].
>>: -- the intersection of the light plane ->> Srinivasa Narasimhan: [inaudible].
>>: -- thresholding. Our algorithm detects the intersections reliably despite this
rescattering and reconstructs the 3D shape of the scene.
>> Srinivasa Narasimhan: And this is, so the idea is not to create 3D reconstruction that
much as it is to remove the effects of scattering. Okay? So as long as you get sort of
approximate depth, you're fine.
>>: In addition it computes the appearance of the scene without any scattering effects as
if seen through clear air. Here we show a harder example with higher milk concentration.
Notice the significant back scattering and loss of contrast in the floodlet image. We now
show the light striped scan. Notice that the intersections of the stripes with the objects
are very dim and are almost invisible towards the end. Even in this case our methods
performed very well due to strong attenuation of the light planes and the dull reflectants
of the green cup the handle of the cup is virtually invisible in the input video and hence is
not detected.
>> Srinivasa Narasimhan: So basically this is trying to show what the limitation is. If you
increase the concentration significantly you don't have any light going to the scene, right?
So that's sort of the limitation. But at least compared to flood lighting it increases the
visibility by at least three times. Okay?
>>: We now show our results for a second scene consisting of a doll ->> Srinivasa Narasimhan: So here's a scene where there's lots of blooming and
[inaudible].
>>: The light striping video of the scene shows complex blooming or blurring effects
[inaudible].
>> Srinivasa Narasimhan: So as long as you can detect this better, then you're fine. So
these are white -- sort of whitish objects so you have a lot of [inaudible].
>>: Again, our algorithm successfully removes the effects of scattering as shown on the
right side. We now show the 3D reconstruction of this scene.
>> Srinivasa Narasimhan: And there's no smoothing or anything done here. So every
pixel is just reported just like that. Okay. So, you know, that was one particular
application. And we showed it as part of the same paper also how to do photometric
stereo, and it turns out instead of using three light sources you need five light source
positions to do photometric stereo of land version surfaces underwater. So that was the
result there.
Okay. And that's that. Yeah. That's the video is done. Okay. So this is just to show an
example of a floodlet image versus what you can get if you're willing to wait until you can
scan. But these scanning things can be done pretty fast, okay.
>>: And when you said photometric stereo that, was with the scanning light or you
actually ->> Srinivasa Narasimhan: Yes. Just distant lighting. Exactly the same situation as
before.
>>: And the flood lighting wasn't disturbing things?
>> Srinivasa Narasimhan: So it goes very far away. So it wasn't really close by and, you
know, shooting out rays in all the directions. We made it a columnated source. Yes.
Yeah, that is much of a toy example. This is actually being used underwater now. So
they use the model and they try to do the mapping of ocean floors and reconstruction and
so on. But photometric stereo is much more of a toy example, more of just a vision
theorist that instead of three you need five, because you have more information through
the medium. And it also depends on the depth.
>>: I'm curious [inaudible] like the laser or anything similar instead of [inaudible].
>> Srinivasa Narasimhan: Again, this could be a laser sheet, so you could have a laser
beam with a cylindrical lens and its [inaudible] sheet. So the light source, in this case we
used the projector, but it could be any other.
>>: [inaudible].
>> Srinivasa Narasimhan: Point would just make it, you know, you have to scan it in 2D
and of course the brightness of a single point would be much lower. The power required
much lower -- would be much lower.
Okay. So final -- just one side note as to how to place the light source and is sensor.
And this is a recent thing that we found out that, you know, it's not just that scanning
sheet of light is important but where to place it. Okay? So if you look at the same old
diagram where you have a common [inaudible] volume, you could place the light source
close by. That increases the volume, right? Now, typically in underwater imaging what
they said was place it as far as possible. But there's also the issue of noise. Right? So
if you place it very far you have attenuation and you have noise. So it turns out that there
is a an optimal solution to where you need to place the light source for a given visible
constraint.
So tell me that you want 20 meter visibility, I'll tell you exactly where to place the camera
and the light source. Okay? So you're just trying to optimize for the best image contrast,
which is contrast times the signal to noise. All right. So this would be, for example, just
an example here you have a glass tank with the dilute milk in it and you have a camera
and the light source, you see pretty much noise, okay. So if you place it in the right
position, without having to do any processing, you get a much better image. Right? So
to try and control light transport before image formation is a very important thing in
underwater imaging. Okay?
So this was with a point light source. But you could do this with a -- you know, a lasering
striping for example. And you could get some kind of a strategy for striping. So it could
involve both translation and rotation. Okay? And you get a much, much better image -this is the acquired image. So it's not processed or anything. So of course you can see
some of the vignetting effects and so on, but this image you can see all of the characters
and the -- you now, the contrast gradings much better. Okay?
>>: [inaudible] in the right place?
>> Srinivasa Narasimhan: No. Turns out not. So depends -- the answer -- the short
answer is it depends on the amount of fog. So this is what I'm talking to Honda, and they
want to actually move the fog light around adaptively. So given ->>: [inaudible].
>> Srinivasa Narasimhan: Huh?
>>: [inaudible].
>> Srinivasa Narasimhan: What is that?
>>: It would actually move on the car, like dynamically.
>> Srinivasa Narasimhan: Yeah. So the lights are just placed statically, right?
>>: Uh-huh.
>> Srinivasa Narasimhan: So the driver is here and depending on the fog, you can
actually move it around to see much better. So there's no imaging involved here. So if
you just tell the -- you know system how much farther you need to look at. Now, of
course you can't do this for all situations because some solutions may turn out to be that
the light source would be outside the car, right? And that you can't do. But it's interesting
to note that, you know, if you can just play around with the positions of the fog lamps you
can see much better.
>>: So is there an explanation about why certain angles are better than others?
>> Srinivasa Narasimhan: So it's not in terms of this actual scattering property, it is
because the fog sort of back scatters quite a bit. So it's better to have an angle, for
example.
>>: Okay.
>> Srinivasa Narasimhan: Right. So instead of having the light source right in front of
you. So most of the light actually comes back to you of the.
>>: So it's better to have an angle?
>> Srinivasa Narasimhan: It's better to have an angle.
>>: [inaudible].
>> Srinivasa Narasimhan: Yes. That's the reason fog lights are actually -- well, the
reason fog lights are lower is because there's not much fog on the ground. So you have
fog layer that is sort of hanging in the air and the light sources are actually below the fog
layers. That's the reason they're placed so low. But the angle, the separation angle, just
like in stereo you have the two angles and you have a smaller value.
>>: Right.
>> Srinivasa Narasimhan: Right. It's exactly that principle. Now, with that assumption
you could place it very far away.
>>: Right.
>> Srinivasa Narasimhan: But that's not going to work because you have attenuation of
light.
>>: But right now the headlights were placed on the outside, right?
>> Srinivasa Narasimhan: Right.
>>: So you're saying further is better, why would you ever want to have them moved
closer.
>> Srinivasa Narasimhan: That's because if you had lighter fog, for example. So the
denser fog you might have to place it slightly away a little bit. Lighter fog. So it depends
on the density of fog.
>>: So under lighter fog you'd have less back scattering?
>> Srinivasa Narasimhan: Right.
>>: But what advantage do you get by moving the light closer the to the viewer?
>> Srinivasa Narasimhan: Because you have more sort of direct illumination of the
scene. Right? So if you had just one lighting that is going this way, you want it straight
ahead, right?
>>: Is it because of shadows or back ->> Srinivasa Narasimhan: So you want to -- so what you want to do is you want to get
most of the light to the scene.
>>: Yes.
>> Srinivasa Narasimhan: Right? So as the ray is going, it's attenuating. Remember.
So you want the shortest possible path to the scene that is closest to you to get the
maximum ->>: This is [inaudible] one over R squared ->> Srinivasa Narasimhan: One over R squared times E to the power minus something.
>>: Okay.
>> Srinivasa Narasimhan: So one over R squared is just without fog.
>>: Right.
>> Srinivasa Narasimhan: The E to the power minus something is with fog.
>>: Okay. So it's moderate for light amounts of scattering you're better off being closer.
>> Srinivasa Narasimhan: Correct.
>>: Okay.
>>: You might actually want to turn off the fog light that's closer to the ->> Srinivasa Narasimhan: Yes, absolutely, absolutely. Or even some vehicles have fog
lamps on top.
>>: [inaudible].
>> Srinivasa Narasimhan: Whether that's such a great idea, I'm not sure. But it's -- it
might be more of a style rather than ->>: [inaudible].
>> Srinivasa Narasimhan: So all of this is about just trying to optimize that expression for
a particular visibility. So if you tell me it is 20 meters, we'll try to place it so that you get
the maximum contrast out of it.
Okay. The final part I'm going to talk about is this is going to the previous parts where
sort of data two or three years ago. This is much more new. Here we are trying to think
about what kind of lighting that you want to use. So previously we looked at a point light
source or just a bulb, for instance, or just a laser sheet, right? So in this case, there's a
lots of applications in vision where you want to use a projector, a programmable light
source for, you know, there's been eight pro cam's workshops already for different
applications and so on.
So one of the issues with using programmable lighting, it's great because you can
program every pixel over space and time, but you have defocus, right, you have shallow
depth of field of projectors. Okay? So this is the first time we sort of studied the interplay
between the defocussed illumination and any global illumination present. Okay?
So you can enable algorithms such as depth from defocus, projector defocus or you can
enable algorithms that just direct global separation in the presence of defocus. Okay?
So let me show you what I'm talking about here. So if you look at this scene, right, so
this is, you know, a very bland looking scene. So you have -- but the materials are
complex. So this is a candle. This is a polyresin bust. This is impure marble. And this is
a V groove that is made of stretched cloth, okay? And this thing is sort of a clay pot.
Okay? That's the only simple material there, okay? So what you do is if you want to
explore defocus of the projector, let's look at just a point here. And you shine a pattern.
Let's say a -- you know, a pattern of stripes across. And the projector sends it -- sends it
out in a defocused manner on to that point, right, depending on the depth of the point.
Yes? So if you shift that pattern across, okay, so you can see that this pattern may be
better focused here than it is here. Okay?
So if you shift that pattern across the final brightness would be a convolution of the input
pattern times the defocus kernel. So this is well known. Right? But if you look at a point
on the candle which has subsurface scattering in it, right, so let's look at a point B, and in
that case you have not only defocus but you have subsurface scattering, so you have
light rays going into the surface and coming out, okay. So you have this additional
complication, if you will.
So in that sense, we showed in this paper that the scene radiance can be written
theoretically as the input light pattern can [inaudible] the defocus blur convolve with this
new global illumination blur kernel. That depends on the intrinsic properties of the scene,
the full 3D the shape, the material properties and so on. Okay? So if you just did that
from defocus by ignoring any of these interreflections or subsurface scattering and so on,
you would not have this stuff, right, you would just have a convolution of two things.
Now you think about for your transform of this you have a product of the coefficients.
And if you know the impact pattern, you can divide it out to get the depth kernel. And
that's how you do depth from defocus using active lighting. Okay? So if you do that, you
get an image like that. Clearly this is wrong because the candle is actually inside the part
and this is the depth of this part is the only thing that is accurate. So everything else is
wrong. Right? On so if you look at inter reflections, their subsurface scattering,
everything else is wrong. Okay?
So what we did was, you know, you have these two things acting together, subsurface,
interreflection with defocus, okay. So at first glance you would think that, you know,
adding another kernel is going to be -- is going to make the problem harder, right,
because there's two kernels, you want to convolve the, how do you separate them out
and so on. But what we found was that if you change, let's say, the focus plane of the
projector, then the defocus blur kernel changes much faster than the global illumination
blur kernel. Because the global illumination depends on the intrinsic properties of the
scene, doesn't depend on where the projector is, right?
So instead of doing this on one plane by shifting a pattern, you shift the pattern, move the
projector plane, do it in another. So what happens then is that this blur kernel remains
the same or nearly constrained. These things change and you can still get the depth out.
Okay?
So the two focal plane positions would be one let's say behind the surface. One would
be in front of the surface. And you compute DFD as you're moving the pattern. At every
pixel. So this doesn't require the scene to have texture or anything. This is pixel by pixel.
Okay? And you get DFD that looks like that. If you compute the photo transform of those
convolutions it's basically a product of three things. And the third thing remains constant
as you're moving the focal plane. Okay?
So you can sort of divide that out, and you get a depth measure that does not depend on
interreflections, subsurface scattering, volumetric scattering, nothing. Okay?
So here's an example. This is the example without to go this processing. So you get a
wrong result. And you can now see sort of the dramatic change. Because now you can
say that this depth is pretty close to this depth. Irrespective of the fact that the
subsurface scattering here is much larger than this.
>>: Darker is closer?
>> Srinivasa Narasimhan: Darker is closer, yes. I forgot to mention -- yes, so darker is
closer. So this is the back wall. You have the statue behind.
Now, of course here we're not trying to improve on the accuracy of depth from defocus.
So there is a particular accuracy associated with depth from defocus and that depends
on the aperture. Right. So what we are saying is that given that limit of accuracy you
can do much better in terms of subsurface scattering.
Here's another example. And what we want to do with this is to try and understand
translucency of materials, okay? So my student went out and got all different kinds of
candles and soaps. They're pretty much aligned at the same depth. But their subsurface
scattering properties are totally different. So you have different kinds of wax, different
kinds of pigments with wax and so on. Okay. Different soaps.
So here's what you get if you do the regular depth estimation, right. So this, for example,
is classified as the back wall, which is wrong, right? In fact, what you can think of this is
that it's not a depth map, but it's a translucency map. It's actually showing translucency
effects. Okay? But in our case you get that, which is much more uniform across the
different the materials.
So what we want to do is collect a lot of the materials and build a database and come up
with translucency factors, just like people did in the BRDFs where they came up with
surface roughness factors. Here you can think of translucency factors. Okay?
>>: Isn't that using just a -- like a [inaudible] aperture [inaudible] solution rather than
doing two different takes, rather than changing the focus?
>> Srinivasa Narasimhan: If you just change your aperture plane, you will not get that. If
you just change your pattern on the aperture plane, that's what you are saying.
>>: [inaudible].
>> Srinivasa Narasimhan: You will not get it. Because you have to change something
that depends on the depth of every scene point.
>>: So you can't make the blur [inaudible]. Okay.
>> Srinivasa Narasimhan: I mean, the only way to dramatically change one kernel or the
other is to move the focal plane.
>>: Okay.
>> Srinivasa Narasimhan: So here's another scene. So you have leaves and flowers,
you know, fruits and milk and so on. So I won't tell you how we get this sort of direct and
global component, so if you look at the global component, you can still, you know, by
factoring out global, you got the defocus, you can factor out the defocus to get the global.
Okay?
So here's an example of how to get the global component, irrespective of defocussing
effects. Okay? So this would not be possible by just applying freeze method. Okay. So
you can see that you know, the global component of milk is very high, the fruits, the
leaves, and so on.
So if you look at maybe a depth map obtained without assuming anything about light
transport you get this. So pretty much everywhere it looks wrong, actually. And whereas
this looks much better. So you have this, the milk at the same depth. This is much
behind and these are much closer to the point. Okay? So here's the final example. So
this is an example where you have something made of [inaudible]. This for example
interreflections here, very strange BRDFs, complex BRDFs. So just focus your attention
on the interreflections here.
So this is the incorrect depth map. So you can clearly see that this is wrong because of
interreflections. So you can see that there. And we're not doing any special processing,
it's just a pixel by pixel, it's just dividing two numbers. And you get that. So in this case,
there's no effect of interreflections either on that or this. Okay? So it's applicable to any
BRDF or BSSRDF. Okay? So at the end maybe -- how much time do I have? Have I
overshot?
>>: [inaudible].
>> Srinivasa Narasimhan: Maybe? One minute?
>>: Yes.
>> Srinivasa Narasimhan: Okay. Fine. Just switched on the light. Okay. So the last
thing I just want to very briefly mention, and this is work with Alusha [phonetic] and
Jean-Francois who recently received a Microsoft fellowship. Here we are trying to
analyze the appearance of the sky, okay? So envision we are interested in looking at,
you know, the scene for the right reason you want the right detectors and segmentation
and all of that. But nobody has really focussed on the sky, which in many cases could be
a big part of the image itself.
So if you look at just this part, you have an idea that the sun is coming from maybe the
right. Okay? So it turns out that if you try and analyze the appearance of the sky, which
is basically because of scattering of sunlight, okay, so you write the model for the
scattering of sunlight that is one already presented in the physics literature so you borrow
it, that depends on the view direction and the sun direction. And just like the
measurement part that I talked about, here you have lots of pixels with different directions
and so on. So you can estimate the parameters of the camera, right, with respect to the
sun.
So you can -- by looking at a sequence of images of the sky, let's say from a web cam,
and the sizes we all sort of done something with a single image, so if you look at a
sequence of images, you can find the [inaudible] elevation and the zoom of the camera
with respect to the lighting. Okay?
So just to show an example, here's a web cam. So you can see that the sun is
somewhere here. And it's going to move as I'm going to play this sequence. Where's the
sequence? Okay. So it's moving like that. So this is a camera looking east. Okay? So
you can see there's lots of clouds and everything, so you still can take advantage of the
blue part of the sky. And here's the estimated illumination with respect to the camera.
So for this, you need to know the timestamps, so you need to know when it was taken. If
you know the GPS, you get absolute orientation of the camera. If you don't know the
GPS, get relative orientation. So the sky is acting like a calibration target here. So here's
another example where the camera's looking west. And so the sun comes from behind
you and then sets in the evening. So you can see that it's set there. And here's the
actual geometry.
We initially when we did this, we thought this is not going to be that accurate. But we find
that it's surprisingly accurate, so it's about, you know, two percent or three percent error
with respect to ground true-up. Ground true-up we defined by doing another kind of
calibration where you're just looking at the sun positions in the field of view. So that
would be a straightforward calibration problem for a 3D to 2D match, you know,
projection. So we compared with that and we got very good results in that sense. And
so we -- we applied this for hundreds of the cams now. And we can do this very well with
single images if the image has enough information. The sky is big and the sun is not
totally behind you, you can do it better.
Okay. With this let me just quickly summarize that, you know, volumetric light transport is
a thing that has been ignored in vision for a long time for historical reasons. But I hope
that I convinced you that this can have implications for a variety of things, vision graphics,
you know, Shanick Engineering, and we did some, you know, research in medical
imaging of endoscopy, as well. And it's always used in astronomy and monumental
monitoring and so on. Thanks a lot.
[applause].