Image Mix and Match
• Internet images
– Colorization
– Most similar image searching
– Collage
• Object insertion
• Data-driven approach for robust similarity measure
– Cross domain(Photo, Photo with different lighting, Painting)
– No domain specific treatments
• 異種画像に利用できる画像の類似度計算法
• Idea
– Detect unique region of the target image (comparing to the other)
– Place high weight on the unique regions
– 与えられた画像のどの特徴が，Web上の膨大な量の画像に対してユニーク
かを学習
– ユニークな画像特徴に重みを置く
• Image feature vector (画像特徴ベクトル)
– Intensity histogram, Gradient Magnitude Histogram, HoG, SIFT
– 画像間距離は，この特徴ベクトル間の距離として計算する事が多い
Intensity histogram
HoG, Histogram of oriented gradient， 4k-5k dimension
• Linear support vector machine (see Pattern recognition textbook)
•
Given d-dimensional feature vectors belongs to class A and B, xi∊A, yi∊B xi, yi∊Rd
yi
xi
Find the maximum-margin hyperplane
that divides xi∊A, yi∊B
•
この絵は２Ｄだけど本当は特徴ベクトルと同じだけの
次元，5000次元とか
Goal
• Given a image Ip
– Detect unique parts of feature vector of Ip comparing to the others
– Place high weight on the unique regions
Ip
1) Compute feature vectors of Ip xp and the other
internet images xi
2) Compute hiperplane by linear SVM
3) Project feature vectors of all images onto the
normal of the hiperplane
Other images on Web
このnormalがPCAの軸のような役割になる
• Image colorization from internet image
• Gray scale画像の色付けを，Web上の画像を参照して行う
– Combination of many techniques
• Internet image search, foreground segmentation, suitable
image filtering, Image similarity measure, graph-based
color transfer, selection UI for weight tuning
• Input: Image + text label (e.g. rooster)
• Procedure
– Input: grayscale image with foreground segmentation & text label
– Search images from internet
• Google image search / Flickr
• Automatic fore ground extraction
• Filter similar images for back/fore ground(Ad hoc energy function
intensity,
texture, density of SIFT)
– Color transfer
• Graph based color transfer method
– Maintain the neighborhood consistency
• Compute with different weighting values
 The user can select one of them
– Output: Colored images with different reference images and different
weighting values
Color transfterの計算時に，gray画像と参照画像の両方とも，微小領域に分割し，近傍に矛盾がないようなcolor transferを計
算している．例えば蝶の羽などでは，黄色の隣にはオレンジ色が来る事は無いなど，良い結果を生んでいる．
• Arcimboldo-style collage generation
– Input : Source image and text label for searching element image cutouts
– Output: collage consists from internet images
• Giuseppe Arcimboldo 1527-1593
• Itary
• Collage like drawing
– Each element is recognizable (elements are taken from a same theme)
– The assembly of the elements resembles something
Two problems
• Best fitting cutout search
1) Search images from internet
2) Cutout foreground image by saliency
detection and GrabCut
3) Distance metric between
- Hole in the target image
- Cutout image from internet
• Input image segmentation
1) Mean shift clustering
Compute modes in color & space feature space
エッジ保存フィルタを連続して書けるような物
2) Marge & split strange local regions
-Mean shift clustering generates local regions that
not match any element cutout image from internet
-Trim such regions by Ad hoc iteration
Color distance term / Shape distance term
With best fitting affine transformation
Semantic aware segmentation is difficult…
• New tool for inserting objects into Photographs
– geometry and light estimation with user’s guide
• Less user interaction
– Deal with interior light and exterior lights (e.g. sun light from window)
– ある点がすごい新しいとかではなく，他と比較して全体的なパフォーマンスが
上がっている感じ
• Inputs
– Single image
– User annotation (geometry, light source position)
– 3D model that will be inserted into the Image
Overview of the system
• Geometry estimation
– Previous work + user’s correction, user interface to add other geometry
•
Light source estimation
– Next page
• Object insertion
– Add object into 3D scene and render it with estimated parameter
Interior light estimation
=
+
1) Decompose input image into Albedo and direct light image
2) User points the position of the light
3) Automatically adjust light parameter (position & RGB)
!!Full automatic light source detection from single image is very difficult!!
Exterior light estimation
1)The user marks boundary of the source and projection of the exterior light
2) The system automatically computes mask and direction
• 今年のSIGGRAPHに
• Image = Albedo + Direct
• Image = Albedo + Indirect + Direct という分解をする論文あった
Results