17457
>> Matt Uyttendaele: I'm Matt Uyttendaele, hosting the guys from Kolor today. This is Alexandre
Jenny. And he's going to be talking about a subject near and dear to my heart, panoramic
stitching software. I've always been impressed with the Autopano line of products that these
guys produce. They've been at it for about five years. For a small company, they've really been
able to write a lot of impressive software.
So with that, I often think of software developers as rock stars. I think we all do. And Alexandre
has been in the studio for the last six months producing his latest version of Autopano and now
they're on their world tour. We'd like to welcome them to Redmond.
[applause]
>> Alexandre Jenny: Thank you, Matt, for the introduction. So I'm Alexandre Jenny, the CEO of
Kolor. Kolor is really a small business. Don't expect millions of USD revenue. It's just a small
business. We are actually seven people in this business.
And so we are doing a panorama software for casual photographer and professional. Many field
of applications.
So why am I here in fact? That was the first question I came up with for this presentation. First,
because Matt invited me. Thank you, Matt. I want to thank you for that.
But when I was really thinking about this question, the most obvious response was because I was
used to climbing mountains during the weekends. And, in fact, when you climb mountains, you
just spend a lot of time just going on the peak and then on the peak, when you have your small
camera, you just take a small field of view.
And instantly, when you go on the peak, you have this restriction, and you want the big picture,
the one that just covers the whole field of view, and not the small one.
And that was the start of my quest for the good products to produce panorama pictures. When
you're on the top you have that, with the small camera. But you want that, of course.
So I wanted the big picture. I tried to get it. At this time it was 2002, 2003. I wanted that. And I
spent a full summer in stitching. So full summer, two months, just doing stitching with standard
tools.
I did 163 panorama, each of them were from two to let's say 20 images. And the other time to
produce these images was about 3.5 hours per panorama. Imagine that. I just spent a full
summer just by doing this panorama.
And no way. It's still too complicated. Shouldn't be that complicated. It just seems something
natural to register images to align them.
There should be a solution. And Autopano is the product that I created to satisfy first my needs,
my personal needs. So it was the answer to my own needs.
I wanted simple [inaudible] and should address first the issue that no other product can. What
are these issues? Quick, you don't need to put control points to say this part of the image this
part of the other image. That's just a manual switch. It should be automatic.
And then it should address the color problem. The photometric problem, because most of time
you use camera with everything automatic. So auto exposure, auto color correction, or, sorry,
auto balance, auto wide balance.
So when you go on the mountain that's typically you have snow. The snow is white. But you
always have this little differences between images that say this white is a little more blue. This
white is a little more red, because of auto wide balance. It doesn't work very well in camera. And
you have to address that. And this was really the main problem when you do mountain
panorama.
So we came up with Autopano. It was not a quick one-day job. It took us months. Let's say
many years, in fact, to be able to release the first version.
The key point of Autopano, it's really a strong technical background. We were based -- I think we
were the first auto SIFT licenses from UBCA, so we use sift detection to be able to get really
strong feature detection.
We have this unique color exposure correction that works. In fact, many are doing that
automatically. We have even manual tools to, for example, select the picture which has the true
color information. The one that will in fact propagate everywhere else.
We have auto ghost blender. Of course, this is needed when you have overlapping zones.
There are people that can move between the two images. So you should be able to address this
issue. At this time in 2003 there was nothing that could handle that.
Since the Version 2 we have motorized the head support. So some illustration of that. This is
just a small D-SLR shot. You can see the color correction issue. Auto exposure. I don't know if
this view is just good. But the one on the top in the middle is just reddish and the second behind
is just bluish and you just calibrate that and you get everything right.
Auto ghost that's produced. You get one out of the two. Motor is add support in Version 2 we
support -- you probably heard about the Gigapan panorama head. We support that, so we use
the structure of images to be able to achieve a good panorama even if there are some images
result feature in it.
So full blue sky, clouds moving. Water moving. Right now nothing really works in those areas.
The proper result, we represent that. This is the proper result head. A really good one. More
professional ones are the [inaudible].
So what about software design? It was really at the first time it was really our aim to be able to do
something that everybody can use, because the tools that were available at this time was more in
general tools. If you understood what you are saying about focal lens, control point, optimization.
Angles, you can use them. But photographer just cannot understand that.
And one key point in our product is accessibility. And this really makes the difference. A good
technology is nothing but a good design of the software. And this is the main rule since creating
Autopano and that's why we often get this quote from people: It's simple. So people use it. And
many photographers like our product because they just hit a button and it works. They have
some simple control to be able to adjust.
I will quickly show you a live demo. So you have a two-panel system. On the left you have -let's say like lightning box where you put all the images in it. So here are some images in this
folder from our summer trip here in the U.S.A. you just grab all the images and sort them
according to the exist time.
When you do a panorama, you often do it that way. Walk away that way. Walk away. If you
think about the time lapse, your images are often good in a small time gap. So, for example, this
one, it was taken I think yesterday. Just hit detection. Everything is done. It's ready to be able to
get a good result. And that you get the preview of the panorama.
That's pretty much about it. Cool feature about the Russian tool is GPS support. So we have
dynamic load and you can see even in the preview the full where images resulted in the need of
calculated, result the need to calculate them before.
You can have -- you can see where the individual images are. You have some live, let's say,
layout adjustment mode to be able to get the lines straight. Auto cropping. You need auto crop
so you have the full extent of the panorama. And then you just hit wonder. This image can give a
50 mega pixel images. Yeah. That's one example. You also have projection control. [inaudible].
It doesn't sound really such a narrow high field. These were taken Sunday. This is Seattle. One
good feature about sift is in fact even if I group, have several panoramas in it, it will short that out
automatically, because it can figure out which images go with which one. And that's really good
for work flow efficiency.
Now we can zoom to see. I can still see all the layouts of the images on top of each other. This
was in Vancouver yesterday, much less -- let's say weather. Less nice weather. Okay. Let's go
back to the presentation.
In fact, when we build first that product, we thought it was just a tool for landscape photographer,
as I was a casual photographer. In fact, that's totally false. The field of application we get huge
variety of that.
High resolution photography, of course, cartography, we had people using it in cartography. Real
estate. They use that to be able to automatically, people are just up loading images on the
server. The server just creates panorama on the fly. So instead of having just a small part of a
room, you have the big field of view of the room.
In fact, panorama address this issue of the camera that doesn't have small focal lens. Most of
the time they have 24, 28, 35. But never 10 or 15. And taking just three images can help you
when you don't have the place to put you really far from the subject. And this works really well for
real estate.
Higher resolution painting reproduction. We had that. You probably heard about some project in
the museum in Spain, where they did some gigapixel about that. We have why yours that do that
also for painting restoration. So they just shoot at really high resolution, 100 or 200 images, to
get every detail, every scratch and then the painting go to restoration, and you can have a before
and after architecture fine art.
Movie industry we have digital they're using a lot of our projects on their movies because it's
really work efficient to be able to create HDR spheres, but also a main painting, they have
integrated the product directly into their tool compositing tool.
Let's show you some cool application of that. This was one that was really funny to do. Our
partner is in fact in charge of web cam fields in ski reports. And they have, let's say, 200, 300
web cams, all over the Alps in 50, 80 locations.
And these web cams are motorized. So instead of using them just to make a spot, a life spot on
some part of the ski resort, we now use the controlled motor remotely and we took pictures and
with Autopano server, so the server part of the project, we just create the panorama each
50-minute. Here an example. It's not live from the web. I just down load the field. This is the
same area of a time. So you can do time lapse panorama. But in fact it's three row, 51 columns.
Each 15-minute stitch together and you get that.
Another cool application, gigapixels, you probably heard about all these projects. Yosemite
project has produced a lot of gigapixel one. There are many photographs from New York City,
we did RLM sorting gigapixel two years ago. We have our own project. It's called para 20
gigapixel, but I cannot show you yet because it's not been [inaudible]. We've been unlucky this
year with weather. Since October not a single weekend sun in Paris. Can you imagine that?
We have a dedicated hardware. The first why I know two cameras shooting at the same time.
This really helped us to reduce the time of shooting, because it just took so much time to do
gigapixel 200 images. You take between two and two and a half hour.
So with still camera you reduce it by two. Aiming quite at the same location with some
overlapping, but not much.
This is the view you will have in Paris. So this is something we really care about because most of
the time when you do gigapixel you just care about the technological part, but not care about the
composition of the images.
So we had to find a point, a viewpoint in Paris, that is not really known. So it doesn't really care if
it's not in the picture. Because if we showed that from the Eiffel you cannot see it. Where each
monument are really well located in the images, not in our field of view.
So this is 230 degree field of view. And everything you have all the monuments that you can
think about in Paris.
>>: [inaudible].
>> Alexandre Jenny: Nomart is there. Should be somewhere -- yeah.
And something really cool is that at the viewpoint location you have the Eiffel Tower with invalid -you just not see, low resolution -- invalid golden top that are just aligned. So you have it on the
same spot.
So I hope this picture will be produced in September, because for the moment the weather is
good but it's too hot so you have too much haze and such problem during the summer. So we
will wait again September, October, November, to get that.
This project won photographic awards by SFRC. It's a big French company like Verizon is here.
It's a mobile company. And we did -- even if I didn't mention it yet, we did already get a prize
about that. It's nice. But another cool application, it's something I'll just get on it one month ago.
A customer came to us. This customer was doing sun pass calculation using the software to be
able to predicate some energy, some energy gain by installing solar cells.
So with that we can produce really an accurate estimation of what each solar cells
implementation implant facilities will gain in fact in the third world country. This project was done
by our state weather company. It's called Meteor France and they were using our software to get
the panorama and just added this sun pass.
If I can calculate it and even estimate the date of shooting, if you have the orientation of the
panorama. So let's say this 80 degrees north. By the haze of the sun and the time of shooting
you can say it's May, October, and so on. Some really cool stuff to do with that.
Let's talk a bit about the industry. The cells of DR cellular and the cameras worldwide. In fact,
digital really hits the mass. And when we can see in 2004 there were let's say two million D-SLR
sold each years. In 2008, it's quite done. So five times more DSLRs sold. This year the
estimation will be around 11.
This is just for D-SLR. So already some good cameras. For standard cameras, camera
compact, compact one, last year it was 122 million. And it doesn't count the one that are building
in mobile. So with the mobile one, just let's say 200, 400. I just really -- and if only one percent of
D-SLR could be interested in creating panorama, the projection market for us, it's 100,000 users
a year.
So for a niche market it's still -- really in this stitching area, two markets, distinguished markets.
First the photography market. Fine art photography. They need high details. They were doing
that with medium from a system by using really complicated technique. Now they're doing that by
stitching just by multiplying the number of shut.
Landscape and natural photography, you just need the high field of view. So that's stitching is
really -- you have many customers in that area. The scripted photography, the stitching allows
you to modify the focal point. So you can have really extended deep of field by doing that. So
that's really good.
What these people need, they need high resolution panorama to produce several hundred
megapixels or more. That's really one of the goals. The resolution -- the sharpness in the
images when they print that. So you can just go to see the exhibition painting and see everything
is really sharp at every edges.
And they need the ability to control every aspect of the panorama. So color, layout, distortion.
Straight lines need to be straight. The second big part of this industry is where. That's obvious.
The goal is to create virtual tools. Everyone that is want to market their house, their domain, their
ski resort, hotel and so on can use such techniques.
Their needs are different. It has to be really stupid simple. Because real estate agent just most
of time doesn't know anything about photography. If you have to ask him, hey, just change that
to have a manual exposure. Oh, yeah, just not possible.
Even the need of a software. You have to install that software to be able to stitch. No, it doesn't
work. And it needs to be quick. That doesn't really care about really high quality. They need -- I
want my field of view and that's it. I have my own twine in this industry. What I saw recently is
that imaging hardware like the Gigapan of the Merlin head gives now access to gigapixel images
to everyone. And I really share that the number of pixels will increase because of that hardware
and because the camera themselves are continuing to increase the resolution.
Standard imaging. This often shows that the sector has reached a certain point of maturity. For
example, the papywizard XML specification, this is an official specification that these motorized
heads just recorded while they shoot. And there are some other manufacturers that know are
using this specification as a standard one. So I hope soon every kind of motorized head will have
this specification so we can guarantee the stitch afterwards.
There are still some later data that should be stored cleverly. Mainly the field of view and the
projection of -- there is already some kind of such specification like the KML or you can also have
that in the deep zoom system where you can specify the field of view.
But there is no real open standard meta data about that. And nothing directly stored in XF. This
is something that seems to me missing for the moment.
There is one open question. We have many of these images. And the only website that you can
use to store them is gigapan.org and then you have the layer on Google Earth, but there's
nothing really about it. I think there's really the need of another website that can do that in a
better way and integrated, for example, in other like Virtual Earth as a layer of gigapixels. So
that's, let's say, something for you.
Flash is actually the standard for virtual visits. There was the QTVR technology but it's just dead
because Apple just doesn't support it anymore. And every tool that high resolution tool that is
created on the web on flash, this is really now the answer. If you want to do a virtual visit today
the answer is flash.
Tomorrow, it could be some new standard via hearing on that some 3-D standard that may
emerge in the web. For the moment, nothing is really clear. But with 3-D directly integrated into
the motor, virtual tool will use that of course and we're allowed to create really good new stuff.
My point is research. Teaching is most of time researcher have this, let's say, trying to say
stitching self-problem. No. There are really some open questions in those areas.
For example, we know how to shoot specially really well. So [inaudible] access no problem and
stitch that together. But if we think at a third dimension, let's say deeper field. Let's say HDR is
working. Let's say focal managing. So you mean that at each position you use different focals.
What is the best way to achieve the stitch?
Should you do deeper focused stacking before, after? Should you do HDR before, after? What
is the work flow? And for the moment no one really looked at which one is the better.
This is another trend that will come soon, I think. Because of this hardware that is now available,
you can have a feedback loop. The feedback loop is in fact at each step of the shoot. We can
have an image shut and we can get it back on the computer and you can do content analysis on
that images. Then you can decide, A, is it unfocused? Is it not. Does it need HDR, so on. And
you can image in by shooting a panorama by making a decision at each step of the panorama.
The hardware now can -- we are quite there to be able to do that. So there will be some good
ideas that may come out of that new feature in fact.
GPU computing on small device. Stitching for the moment to be able to make that on small
device. It's really hard to do, because the devices are not peripheral enough. But if you have
GPU stitching, here is something that does work well on GPU. You can have, with the mobile,
you just paint panorama around. You don't have to shoot to press a button. You just have light
painting of panorama. That's something that will come when the power and the GPU will be
really powerful enough to have live tracking and live panorama compositing on small devices. So
I think this will come and it's really a more easy way to create panorama than doing that on any
application actually in mobile. You shoot at each position. You have a blending life [inaudible] it's
not that easy. If you do that and everything in this painting you see old and so on. That seems
for me the next thing in the mobile panorama generation.
We are still a small company, but we are also doing RID research. This is -- we don't publish so it
doesn't -- we don't have that. But we still have some good internal research. I just picked on that
because of the scientist here.
We use mainly a shift as the feature point detector. And shift is good but it has some lacks. The
area allocating on blobs. And the blobs often don't really have the same accuracy as ages. So
we were looking at the better system to be able to do several kind of features, combine them
together to get a better detectivity and get better accurate results. And this is the results we have
at quite at the end of the study.
We used -- we compile Autopano 2 standard detection, Autopano 2 high detection. This is the
record position measurement. And we found that using Harry's Laplace [phonetic] detector
closer, log detector, combined together get better results that the standard detection.
So we hope that the same feature compared to the standard will get something like 95 percent in
the work load and 95 percent precision compared to the high detection quality.
So I never really heard about any people using several kind of features to be able to do that. And
the wizard we have, it's just a good way of -- something about straight line. We fully inspired
ourselves from the locally adapt projection panorama distortion paper that was published in this
year's SIGRAF and we already have quite an implementation of that.
I don't know why you decided to produce this paper, but it's really something that fine art
photographer needs, absolutely. And this is something that he's really -- we are going to put that
in our software for sure.
Kolor papywizard project. I'll take five minutes to present that. This is a standard piece of
hardware that you can buy. It's a telescope. We just unplug everything. It's a telescope
motorized head. So you can get the L bracket and the motorized tripod. It's 200 USD. So really
cheap one. And the open source project was to cover that head that was designed for the
telescope, to shoot panorama. Panorama, in fact, the Gigapan, you can say I can do gigapixel
with Gigapan. Gigapan, the main lack of this project, don't support D-SLR. You can do gigapixel
with just a small compact camera. The quality of the lens is just -- is really not the wide market to
address that.
So this one really you can put a heavy sheer on it and it's really cool. So what do you need to do
gigapixel now with this open source project. You need to buy the normal head. To create a small
cable to have the shutter release and combine this little module. That's German company that
does that. Just plug it -- that's touch screen LCD. You put it here. And you have every control
here, just select -- we'll do some live. Select focal lens.
I will take a grid. So I select the left bottom position. The top, wide position. And say shoot. So
nothing more complicated than that. This project has another part.
As the hardware is now really known, everything has been disassembled and documented. They
came up with that part. Just a standard piece of electronics. It costs to produce $25. Not much
more. It covers the signal from the camera into standard serial signal. And then you have a
bluetooth, serial to bluetooth converter.
And with that you can remote control the camera instead of having this touch LCD system.
And the papywizard is in fact the open source home control software that was [inaudible] and it's
a pattern tools. You just have everything inside to be able to do that on mobile directly and
remote control that.
So you create a preset. Let's say I will shoot that way on papywizard you hit one and the head
just does that.
So there are a lot of development done with this papywizard tool because it's open source. So
many people can put different needs inside. Some adds, this add is supported. New ones are
going to be supported into this project. Time lapse you can do.
And it's with that tool that the feedback loop is going to be created. So that tool will be able to
decide at each position what to do with images. May decide to do something. And remote
control, they add on that.
So papywizard generates XML file. With this XML file the Autopano Pro just analyze that and you
have quite a guaranteed stitch because we don't have feature point because of sky. We just rely
on the location recorded into the XML file.
Something else. We did last year a book of the best images -- the best -- the best images the
user provided to us. We have four free books here if anyone is interested. The Pano Book 2010
is really to continue this kind of marketing to be able to show the people, hey, you can do
panorama. Look at these images produced with our tool.
Just grab the software and try. So it's a beautiful hard cover book. It shows what can be done
with panorama technology. And it was a collaborative project because it's like a dictionary.
Several authors. Not only one.
This year the images will be selected by a jury of professional panoramaists and here's the icon
of the older one. And really for us it's also a good way that the people that use panorama can
show their works worldwide, because this book is shipped worldwide through our website. So
that's really a good marketing for them also.
Why did I talk about that. Because I wanted to show you some of the panorama done last year
by the contributor of that book. Oh, that's not a panorama. Oops. This is the panorama. Of
course. First the panorama is to address that wide field.
Here are some examples. In fact, when we think about panorama, we often think about ratio in
the images. It's something that is really near and large. So this one, is it really a panorama?
The brochure is just two for one. It's not really that wide in ratio.
But it has a really huge field of view. Hmm, but what about that? The ratio 1-1, but 360 field of
view. And this one, hmm, this is called little worm kind of images.
Vertical panorama. Often people think that panorama is something horizontal. No. Think about
that. It's often cool to do that. I usually do maybe say vanity vertical panorama where I shoot my
feet and then go to the scene. So I can say my feet were there.
Some strange composition where a reason is just a two, it's two straight lines that don't cover
themselves. So the background one is under a bridge. And the top is a church, in fact. And that.
This is a panorama so many images. But also stitched into the color of that axis.
So huge dynamic range. Let's go back to a little semantic. The definition I found on Wikipedia, in
most general case senses a panorama is any wide angle view of a fixed space. Ah. Here we
have a new class of images with [inaudible] because of the number of pixels, or enlarged field of
view. Or augmented dynamic range. And of course no common ratio. Everything can go from
one over one to one over 10, 20, and only one common fact. They were produced using several
sources. Question, how can we [inaudible] attend Map HDR image? Is this still an image? Is
this something?
I think that what we saw with produced images from several sources, it's a new kind of image. It's
not something that we know for the moment. We don't have any word to describe this image. It's
not a panorama because it's not just about field of view. There is something to address there.
My conclusion: Panorama shooting, it's a long journey during these five or six years in the
business. It was first a hobby. It began for me a real business.
And it has a huge potential in many areas, and we discovered that every day, even now. Some
notes. We have a special price on Autopano Pro and giga for Microsoft people. It's directly on
the website.
You use this discount code. [laughter] easy to remember, okay? Thank you.
[applause]
>> Matt Uyttendaele: Thanks. We have some time for questions.
>>: One of the applications you say you can use this software for high resolution.
>> Alexandre Jenny: Yes.
>>: Is there any way you get high resolution is by having a long focal length, panning and
stitching it together or do you have the ability of taking the same picture by essentially the same
focal lane over and over again and then [inaudible].
>> Alexandre Jenny: If you mean that if you could do super resolution? No, we don't do super
resolution. In fact, super resolution, even the best algorithm now just can you get twice short time
the quality. But you never get a time the precision. But if you shoot panorama you just move far,
get longer focal and you have the quality. Because you really have a picture that is straight from
the camera and create less distort.
So stitching you don't have enough resolution, shoot more. That's easy.
>>: Are you supporting the Autopano Pro spherical panorama 360 by 180?
>> Alexandre Jenny: Yes, of course. We do, yeah, we support Fish Eye of course to do that
because it's very -- it's very easy to do with Fish Eye lens of any kind.
>>: So you throw the Fish Eyes and create the ->> Alexandre Jenny: Exactly. Yeah, exactly. Uh-huh. In fact, this is the main use with the
digital -- it's to create -- you heard last year the Speed Racer movie. They had to produce 10,000
sphericals and they used our software with it because of detectivivity to create all the spheres it
was around eight terabites of panorama to create.
>>: You said earlier that there aren't very many sites that host panos. There are a number of
companies who tried to make a business out of commercial hosting panoramas for the tourism
industry, xeriscape and 360 cities. And what are your thoughts about them either the technology
they've used or about their business.
>> Alexandre Jenny: About the technology they are using, for example, for 360 Cities, they're
using in fact the same technology that we are using for our virtual tool software. It's flash-based
technology called Caro Panel. A really good one there. The tool they're using is great but it's not
their own technology.
The business behind that, I'm less sure for the moment. So hosting is like Flickr. Flickr you only
pay if you really want to have huge space. I don't think that professional photographer will pay
just for that.
I know -- I have talked to them. But I'm not sure about the business, because it's still a niche
market. I hope that it will be used by many mass people, but, no, it's still a niche. So don't forget
that.
You won't have 10,000 users on such websites, that's for sure.
>>: When you have large images is it possible to save them in multi-resolution tiled format?
>> Alexandre Jenny: For the moment, no. We use for the moment PSV format, either from
delayed largest, 300,000 wide. So that's -- and then we have tools that use these PSB file to
convert them into the hierarchy. For example, in HDTV2 format or soon we'll put [inaudible] and
that's it. But we prefer for the output of Autopano Pro to keep a single file and that's why we even
introduced in version 2 a new format, so color whole format just a single header with every data
put on a single line but allows you to break this 300,000 limit that we know we will break for the
Paris project. So we introduced that.
Then the tools there will just use the images to cut them into pieces and to sell and to have the
hierarchy to be able to display that on the web.
>>: So one of the things I really like about panorama software requires people to upgrade their
PCs because you can really take advantage of multi-core and GPU. And can you talk a little bit
about how you're taking advantage of a core's GPU's.
>> Alexandre Jenny: This is, since the start, we put our effort to do administrating into a
wandering part. Wandering is the fact you have several images. You blend them together to get
the full images. So this works since the first version. For version two we wanted to push that
further and we converted into multi-core the detection part. So the detection is the step where
you analyze all the single images. You're trying to find matching between them. You register
them and you create the preview that you can see in the software.
So quite each stages of these algorithms have been administrated in several layers. In fact, the
algorithm itself have been coded multi-sorted, but it also uses optimized map library that are
themselves using [inaudible] and about that we passed several Intel tested and benchmarks and
we normally use our software as a multi-core sample, some multi-core benchmark, let's say,
showcase, I will say. Sorry.
>> Matt Uyttendaele: Okay. Any other questions? Let's thank Alexandre one more time.
[applause]