Pragmatic approaches to image
resampling
Tom McGlynn
NASA/GSFC
Outline
 Why do we resample?
 Isn’t resampling a solved problem?
 Some approaches to resampling.
 A new algorithm for exact-area
resampling using clipping.
 How should astronomers choose
resampling algorithms?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Why do we need to resample?

Display – transform image into ‘standard’ form



Undo warps and distortions
Transform to standard frame
Resizing




Magnification and minification

Rotation

Similar operations as in display, but final frame may not be ‘standard’

Building sky region and all sky images

Dither additions, image differencing, speckle analysis
Image comparison – transform one image to match another
Mosaicking
Image arithmetic

Can have most extreme requirements on accuracy of reconstruction but
often involves very similar images.
Different problems levy different requirements for accuracy and
robustness. Many resampling problems involve substantial
changes in geometry.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Resampling examples
SkyView transforms the
EGRET all sky map in Galactic
coordinates to Equatorial
coordinates.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
WMAP data must be transformed from HEALpix formats to
human friendly ones.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Transforming SOHO/EIT
images to Cartesian
coordinates. And that’s
before handling differential
rotation…
On Resampling of Solar Images
Solar Physics, 2003,
C.E. DeForest
Tom McGlynn
IPAM Workshop
Jan 27, 2004
SkyView dynamically mosaics
surveys data for display or image
comparison
Tom McGlynn
IPAM Workshop
Jan 27, 2004
SkyMorph image addition.
Three co-added
NEAT images
taken at 20
minute intervals.
(1999 discovery
images for
asteroid 9460)
DSS image
(1950)
Adding NEAT images for asteroid discovery using
SkyMorph. Other survey datasets can also be
searched for pre-discovery
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Classical image resampling
1. Image reconstruction
The continuous image is regenerated by some
method, often by interpolating between the grid
point by using an interpolation kernel.
2. Sampling
The new grid is constructed by sampling the
reconstructed function appropriately.
3. Filters and transformations
Intermediate filters and transformations may be
applied at various points in the process.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Ideal Reconstruction
 A band-limited function can be
exactly reconstructed by convolution
with the appropriate kernel.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Shannon says…By
hypothesis f…
…has a Fourier transform, F
bandlimited to 1/2T.
-1/(2T)
Pixelate: Multiply by
Comb function) with
separation of T…
1/(2T)
… means convolve the Fourier transform with the
Fourier Transform of Comb, but this is just another
Comb function with separation of 1/T which just
relplicates the Fourier Transform.
5T
… but that’s equivalent to
convolving in space with
Fourier transform of
boxcar, sinc(). So we just
need to use sinc
interpolation to get back
original image.
So mutiplying by a boxcar gets back
original F …
Solved problem?
 So sinc function is the optimal filter.
Just reconstruct the image using sinc
interpolation and resample as
needed!
Tom McGlynn
IPAM Workshop
Jan 27, 2004
But… real images are finite
 Chips typically have 1-4K pixels with
larger detectors using arrays of chips.
 Edge effects need to be considered …
 Copying image, image reversal, constant
value, 0 beyond edge
 How do we handle missing pixels?
 Typical CD song has ~100K samples
before we begin hearing the music.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
…Is the image well sampled?


Is sampling rate adequate? How do we
tell?
What can cause high frequencies?



Noise and photon statistics
Features
Processing


Filtering and compression – flat can
mean high frequencies.
The sky

1 Gigapixel Nyquist-sampled image for
0.001” resolution covers only 16”.
 Do users prefilter images?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
…The sky isn’t a plane
 Proper basis for sphere is spherical
harmonics.
 CMB studies use HEALPix pixels to make this
easier.
 Are HTM grids going to be used?
 Derivation of sinc function kernel assumes
rectangular grid. Other functions are
appropriate for other samplings.
 What are the effects of ignoring curvature?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
…Point versus Area resampling
 Pixels have finite extent.
 In 2-d case the relationship between the
original and resampling pixels can be
complex.
 1-D and 2-D resampling share ‘calculus’ but 2-D
geometry is more complex.
 If input and output pixels are similar we can
presume the function being reconstructed is the
flux convolved with the pixel mask.
 How do we handle resampling pixels with
complex shapes?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
What makes resampling hard?
 Calculus
 Undersampling
 Noise, features, missing pixels
 Constraints on the output range (avoidance of
negative values, integer valued functions)
 Geometry
 Differing projections and coordinate systems
and orientations.
 Scale changes
 Non rectangular pixels.
 Variations over the image
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Alternatives
 Image regeneration
 Is the image a derived product?
Pixels added?
Models?
 Retake the image
 Adapt requirements to minimize
projection issues
 Often not feasible or desirable.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Some reconstruction kernels
The background
image is
reconstructed by
convolution with
the given kernel.
The sinc function is
the ‘optimal’
reconstruction
kernel for a well
sampled image.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Nearest Neighbor
Universally panned…
 Bad geometric distortions
 Suspect photometry
 Poor for magnification
but even the least elegant techniques may have
their place.
 Fast
 Histogram of values can be preserved.
 Sometimes exactly invertible.
 Integers stay integers



Classifications (does vegetation+water=city)
Photon counting algorithms

Man-made boundaries
 Constellation maps
Retention of discontinuities
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Bilinear Interpolation
 Fast
 Better at preserving astrometry and
photometry
But
 Smooths and blurs the image
Tom McGlynn
IPAM Workshop
Jan 27, 2004
‘Higher order’ interpolation
 Best job reproducing well sampled images
 Limits blurring
But…
 Sometimes slower
 Can be sensitive to ‘features’ or undersampling.
- negative values when resampling.


Harder to handle missing pixels
Gazillions of choices


Interpolating or approximating
Polynomial approximations







Splines
Local support polynomials
Truncated sinc
Windowed sinc: Lanczos, Hamming, …
Image support for calculation of kernel
Image support for resampling calculations
Very high order methods (e.g., exact sinc) can be very slow
Tom McGlynn
IPAM Workshop
Jan 27, 2004
More choices:
Interpolation versus Redistribution
 Interpolation
 Output flux at a pixel is computed as a sum of
weighted input pixels using reconstruction kernel.
 Tends to assure differential properties of image
(continuity, derivatives)
 Redistribution (Drizzle, exact area sampling)
 Input pixel’s flux is distributed over output pixels
using redistribution kernel.
 Can ensure integral properties of image (flux or flat
field).
 Easy to handle missing pixels or other discontinuities
 Global integrals can be conserved to the limits of
arithmetic precision
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Exact Area Resampling
Calculate the
resampling pixels
as the weighted
averages of the
pixels they cover
(or weighted sums
for extensive data)
weighting
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Approaches



Delaunay Triangulation
 Using suitable collection of points one finds a set of
triangles where each triangle belongs to just one input and
output pixel.
 Hideously slow…
Girard’s theorem (Montage: Berriman and Good)
 Work on the celestial sphere and use Girard’s theorem to
calculate area – a `practical’ example of using parallel
transport of vectors.
 Very, very slow.
Clipping
 How much of the resampling pixels can you view through
the window of the input pixels?
 Comparable speed to other high-order methods.
 Equivalent to algorithm used within Drizzle?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Exact area resampling using
clipping.
For each resampling
pixel…
First find all the pixels
that may overlap the
resampling pixel, by
looking at the range of
the resampling pixel
corners. Now for each
candidate original pixel…
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Clip the resampling pixel on each
edge.
1
2
3
Tom McGlynn
Clip the
resampling
pixel by each of
four clipping
boundaries.
(SutherlandHodgman
algorithm but the
extensive clipping
literature suggests
more efficient
approaches, e.g.,
Liang-Barsky).
4
IPAM Workshop
Jan 27, 2004
Clipping by a infinite line
v3
c2
For i = 1 to n
v4
v2
if vi is inside
c1
Inside
v1
if vi-1 is outside
Outside
Input
(v1,v2,v3,v4)
Output
emit crossing point (e.g., c2) and vi
else if vi is outside
emit crossing point (e.g., c1)
else
emit vi
(v1,c1,c2,v3,v4)
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Triangulating the overlap
D
C
The area of the overlap
polygon ABCDE is easily
computed as the sum of the
triangles ABC, ACD, ADE
E
A
Tom McGlynn
B
IPAM Workshop
Jan 27, 2004
Normalization
 Intensive images
 Track total flux and overlapped area for
each pixel and use ratio for pixel value.
 Preserves flat fielding
 Extensive images
 Just add flux.
 Preserves total flux
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Features



Convexity of pixels allows simplification of algorithm (since
convex regions clip convex regions to convex regions).
Clipping on rectangular grid is especially easy but either grid
could be triangular or hexagonal or even discontiguous.
Don’t have to make clipping window the same as pixel
boundaries.




If clipping window is smaller we get Drizzle-like algorithm
If clipping window is larger than pixels we have a variable box-car
along with resampling.
Easy to accommodate ‘bad’ pixels or regions.
Symmetry between resampling and original grid

Can resample in the `convenient’ direction if transformation is
easier one way or the other.
Not the most ‘accurate’ algorithm, but it can be extremely robust.
“Best worst-case resampler”
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Possible clipping ‘kernels’
Exact area
sampling
Drizzle
redistribution
Boxcar smoothing
with resampling
Step pyramid
kernel with
different fraction
of flux in each
box
Adjusting the size of the rectangle changes how the
algorithm considers the flux in the pixel to be
distributed.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
How can astronomers decide which
resampling algorithm to use?
 Accuracy
 Does resampling affect astronomical measurements?
 Detection, Astrometry, Photometry, Morphology,
Resolution
 Cost (CPU, memory)
 Complexity
 Coding and comprehension costs
 Robustness
 Point versus area resampling
 How well does it work when in hard resampling
situations?
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Need for a roadmap
Issues:
Undersampling
Pixel distortions
Spherical geometry
Edges
Prefiltering
Bad pixel regions
Compression
…
Tom McGlynn
Goals: Mosaicking, Resizing,
Displaying, Subtracting, Dithering,
Comparing, Undistorting, Projecting
Users
Traits:
Accuracy
Speed
Comprehensibility
Availability
Robustness
…
IPAM Workshop
Jan 27, 2004
Methods:
Nearest neighbor
Linear interpolation
Splines
Polynomial kernels
Gaussian
Sinc
Hamming
Lanczos
Exact area
…
Analysis in other regimes
Quantitative Comparison of SincApproximating Kernels for Medical Image
Interpolation
Erik H. W. Meijering, Wiro J. Niessen, Josien P. W.
Pluim, Max A. Viergever
Round-robin resampling – a sequence of
resamplings leading back to the original image.
Good for determining the ‘best’, but not for
assessing the cost
• Truncated sinc resamplers are the worst.
• Variety of higher order resamplers do pretty
well.
• But the images don’t look much like typical
astronomical data)
Tom McGlynn
IPAM Workshop
Jan 27, 2004
…
Image Reconstruction by Convolution with
Symmetrical Piecewise nth-Order
Polynomial Kernels
Erik H. W. Meijering, Karel J. Zuiderveld, Max A. Viergever
IEEE Transactions on Image Processing, vol. 8, no. 2, February 1999, pp.
192, 201.
•Going beyond third order kernels doesn’t seem to buy one anything.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Comparing image resamplers via a
model of the human vision
system
Richard Harvey, Stephen King,
Richard Aldridge
and J. Andrew Banghamh
A big driver in
commercial applications
is rendering of text.
Even astronomical references often compare samplers in
qualitative terms.
Features in Lanczos
resampling.
NN
LI
Lcz2
Lcz3
Lcz4
SWarp v2:0 User's
guide
E. Bertin
Moire patterns in linearly interpolated
resampled images
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Measuring accuracy quantitatively




Single resampling
Astronomicalish images
How does noise affect resampling?
Two resampling scenarios:


Small Pixel: Rotation, final pixel size/original pixel size=1.1
Big Pixel: Rotation, final pixel size/original pixel size=2.5
 Use Sextractor to estimate parameters of original and
resampled objects.
 Model image with 100 gaussian objects and variable
noise
 Use DSS image for reality check
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Test Images
0: 0 (Number: Noise)
15: 0.2
5: 0.006
10: 0.03
20: 1.0
North Pole image
DSS image
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Samplers Tested
 Interpolation algorithms




NN – Nearest Neighbor
LI – Linear Interpolation
LCZn – A Lanczos n-lobe interpolator (n=3,4)
SPn – An n’th order spline (n=3,4)
 Redistribution algorithms
 CL – Clipping exact area
 CL0.5 – Clipping using a window of half the size
of the pixel (similar to Drizzle)
 MN –Montage exact area algorithm, resampling
done on celestial sphere.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Small Pixel Detection
200
Boxes give the results
for the real image
looking at the first six
samplers listed. The
dashed line gives the
measurement in the
original unresampled
image.
180
CL
CL0.5
LCZ3
LI
MN
NN
Orig
Real
Lcz4
Sp3
Sp4
160
Sources Detected
Did we detect
all of the
objects
detected in the
unresampled
data?
140
120
100
80
60
40
20
0
0
5
10
15
Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
20
25
Small Pixel Astrometry
1
0
5
10
15
0.01
0.001
0.0001
Noise Sample
Tom McGlynn
20
25
CL
CL0.5
LCZ3
LI
MN
NN
Orig
Real
Lcz4
Sp3
Sp4
0.1
Offset in pixels
What was the
average offset of
the resampled
image from the
measurement in
the original image?
IPAM Workshop
Jan 27, 2004
Small Pixel Photometry
0
5
10
15
0.01
0.001
0.0001
Noise Sample
Tom McGlynn
20
25
CL
CL0.5
LCZ3
LI
MN
NN
Orig
Real
Lcz4
Sp3
Sp4
0.1
Offset in pixels
How much did
the flux change
when we
resampled?
1
IPAM Workshop
Jan 27, 2004
Small Pixel Morphology
1
The image modeled
circular gaussians.
What is the average
axis ratio (a-b)/a
measured in the
resampled data?
Note that graphs
compare with model
and not measured
values.
Non-circularity (a-b)/a
0
5
10
15
25
CL
CL0.5
LCZ3
LI
MN
NN
Orig
Real
Lcz4
Sp3
Sp4
0.1
0.01
0.001
Noise Sample
Tom McGlynn
20
IPAM Workshop
Jan 27, 2004
Small Pixel Blurring
0.006
0.004
Blurring (1 pixel = 0.01)
Is the resampled
image blurred by
the resampling?
How much larger is
the it?
0.002
0
0
5
10
15
-0.002
-0.004
-0.006
-0.008
-0.01
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
20
25
CL
CL0.5
LCZ3
LI
MN
NN
Orig
Real
Lcz4
Sp3
Sp4
Big Pixel Detections
120
100
Number of objects found
80
CL
CL0.5
LCZ3
LCZ4
60
LI
NN
SP3
SP4
Orig
40
20
0
0
1
2
3
4
5
6
7
8
9
10
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
11
12
13
14
15
16
17
Big Pixel Astrometry
1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
0.1
CL
Change in flux
CL0.5
LCZ3
LCZ4
0.01
LI
NN
SP3
SP4
Orig
0.001
0.0001
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Big Pixel Photometry
10
1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Change in flux
CL
CL0.5
0.1
LCZ3
LCZ4
LI
NN
SP3
0.01
SP4
Orig
0.001
0.0001
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Big Pixel Morphology
1
0
1
2
3
4
5
6
7
8
9
10
11
0.1
12
13
14
15
16
17
CL
CL0.5
Non-circularity
LCZ3
LCZ4
LI
NN
SP3
SP4
Orig
0.01
0.001
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Big Pixel Blurring
0.03
0.025
Blurring (1 output pixel = .025)
0.02
CL
0.015
CL0.5
LCZ3
LCZ4
0.01
LI
NN
SP3
SP4
0.005
Orig
0
0
1
2
3
4
5
6
7
8
9
10
11
-0.005
-0.01
Noise Sample
Tom McGlynn
IPAM Workshop
Jan 27, 2004
12
13
14
15
16
17
18
Timing
1000.0
Tom McGlynn
10.0
IPAM Workshop
Jan 27, 2004
(L
cz
3)
ta
ge
on
M
Sw
ar
p
Cu
b
ic
Sp
l
in
e
s5
s4
La
nc
zo
s3
Java
La
nc
zo
La
nc
zo
Cl
ip
nt
er
p
1.0
Li
(Times are for
complete process
including I/O and
coordinate
transformations.
600 MHz
processor)
100.0
NN
Time to
resample a
500x500
sample in a
700x700
grid.
C
Complexity
 Low order methods are trivial to program and relatively
easy to test.
 Some higher order kernel interpolation methods are only
modestly more complex. The initial computation of
splines is more challenging, but splines are very easy to
evaluate
 Previous exact area sampling techniques were complex
and time prohibitive. Redistribution methods based on
sample kernels can be comparable to interpolation
methods. Clipping algorithm is relatively straightforward
to understand but use a more algorithmic and less
functional paradigm.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Robustness

Discontinuities, features, holes are harder to handle directly in higher
order methods

Prefiltering or image patching.

Supersampling (averaging multiple sample points within a single
resampling pixel) is an easy step towards area resampling for
point resampling methods, but requires understanding how many
samples should be made within each pixel.

Magnification of well-sampled images best done with high order
samplers

High order samplers perform poorly on minified images and where
pixel shapes are very different.
Need a quantitative measurement of robustness.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Class 1 Resampling
Resampling pixels are similar in most respects to
input pixels: translations, small distortions and scale
changes.
• Use high order techniques, e.g., Lanczos or spline
methods when data is well sampled. The first is
easier to program, the second is faster. Limited return
from going beyond cubic spline or Lanczos 3.
• Drizzle approach can limit blurring effects of
redistribution while accommodating features or
undersampling.
• Noise induced errors typically outweigh sampling
errors except for blurring.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Class 2 Resampling
Resampling pixels substantially different from
original pixels, but more or less constant over
image.
• For small resampling pixels (magnification)
point sampling techniques should work with well
behaved images.
• When minifying, supersampling or inherently
adaptive techniques, e.g., exact area or other
redistribution techniques are best.
• Sampling errors can easily dominate errors due
to noise.
• Balance accuracy and robustness.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Class 3 Resampling
Resampling pixels vary significantly over
image. Big holes or features. Nonrectangular, non-contiguous grids?
• Use adaptive resampling techniques.
Exact area sampling is a good bet.
• Use best robust techniques.
• Objects are going to have substantial
distortions in original or sampled grid.
Tom McGlynn
IPAM Workshop
Jan 27, 2004
Concluding thoughts
 No single best resampling technique.


Probably a small suite algorithms can serve for a wide
variety of situations.
Can algorithms self-select?
 Too empirical
 Can robustness be defined quantitatively?
 Can we predict accuracy of resamplers?
 Lots more to consider:
 Extended objects

What is the interplay of resampling with other elements
of processing, e.g., compresion?
Tom McGlynn
IPAM Workshop
Jan 27, 2004