Single image upscaling
1. large
2. realistic
3. faithful
4. fast
Previous work
parametric image model
example based
noisy
result
Freeman et al. 2002
Fattal 2007
Shan et al 2008
Sun et al. 2008
generic looking edges
Glasner et al. 2009
New approach: locale example based
local self similarity
corner step edge
line step edge
too few
locality non smooth shading
examples
small upscaling ratios
4/5
Increase exemplar
quality and
size
1/2
maintain search locality
new non-dyadic filter bank
Local self-examples upscaling
frequency
content
interpolated image
low pass
original image
high pass
Local self-examples upscaling
For each patch:
frequency
content
Search a local area for
best example
low pass
Add to
interpolated
image
interpolated image
Take
corresponding
patch
high pass
Local self-examples upscaling
Repeat for all patches, to fill the high frequencies
frequency
content
interpolated image
low pass
high pass
Local self similarity
cropped ≈ downscaled
Local self similarity
Patches in original image can matched locally with ones in downscaled version
Local examples are enough
query
db
image
local
4.0
2.9
3.55
1.6
1.05
1.05
2.7
2.05
2.05
3.3
6.5
image database
full image
2.96
5.61
best matches
3.06
5.61
Visual assessment – external, exact NN, local
Large external
example database
external database
Searching the
entire image
global search
Searching local
regions in image
local search
Comparison of example search methods
Need for non-dyadic scalings
large ratios
small ratios
mixed ratios
Dyadic filters
1:2
full
frequency
content
higher
half
lower half
dyadic filter bank
Non-dyadic filter bank
4:5
full
frequency
content
1:2
small scaling
ratios
better
examples
higher
part
lower part
non-dyadic filter bank
Non-dyadic filters: downscaling
example for the 2:3 ratio:
dyadic case:
1. convolve with one filter
2. subsample by 2
1. convolve with 2 filters
2. subsample each by 3
Non-dyadic filters: upscaling
example for the 2:3 ratio:
dyadic case:
1. zero upsample by 1
2. convolve with 1 filter
1. zero upsample by 2
2. convolve with 2 filters
3. sum
Use of the filters in upscaling
Upscaling using
inverse scaling
filters
Smoothing by
downscaling
and upscaling
1. Uniform stretch
brightness
When interpolating, smooth areas come from input
Uniformly spaced grids should remain uniform
255
0
grid coordinates
2. Consistency
The interpolated image, if downscaled should be equal to the input.
Formally,
↓ ↑ 𝑰
upsample
=𝑰
downsample
Previous methods achieve consistency by solving large linear systems to
achieve this property
3. PSF modeling
Difference between
point spread functions
Large image - small
camera point spread
function
Small image - large
camera point spread
function
frequency
4. Low frequency span
When upsampling
don’t add new
frequencies
Upsampling filter
should be low-pass
original
interpolated
5. Singularities preservation
≈
similar
amount of blur
blurred Image
interpolated image
Real time video upsampling on GPU
Search and filter-banks
are both local operations
main GPU
memory
GPU cores
NTSC to full HD
@ 24 fps
Bicubic x3
(zoomed in)
Ours X3
(zoomed in)
Bicubic x3
Ours X3
Genuine Fractals™ x4
Ours X4
Glasner et al. 2009 x4
Ours X4
Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn