Creating Better Thumbnails
Chris Waclawik
Project Motivation
• Thumbnails used to
quickly select a specific
a specific image from a
set (when lacking
appropriate metadata),
but…
• Larger sets require
smaller, less detailed
thumbnails.
Thumbnail effectiveness
Can you find the picture with the dog
in it?
Thumbnail effectiveness
Can you find the picture with the dog
in it?
Thumbnail effectiveness
Can you find the picture with the dog
in it?
Problem Statement
• How can thumbnails be made more effective?
Show only the salient parts
Two subproblems
• How do we determine the salient portions of
the image?
• Knowing the above, how should we crop the
image?
Determining Image Saliency
• F. Liu, M. Gleicher, Region enhanced scaleinvariant saliency detection, in: Proceedings of
IEEE International Conference on Multimedia
& Expo (ICME), 2006, pp. 1477-1480.
• Two-part method:
– Scale-invariance makes this method suitable for
determining salient features of many sizes.
– Region-enhancement aids in making logical crops
later on.
Scale-invariant Saliency
• Image transformed to perceptually uniform color space
(Lu*v*).
– Difference in color values proportional to perceived
difference.
• Gaussian pyramid created and a contrast map is
calculated for each level.
– The contrast of each pixel is the sum of its distances from
its neighbors (in Lu*v* space).
– Pixels closer to the center are given more weight.
• The contrast maps are summed to find scale invariant
saliency.
• Algorithm implemented in Matlab.
Region-enhanced Saliency
• Regions extracted from the image using any
preferred method.
– Mean shift method used in paper (and in this
project).
– C++ code from EDISON, Matlab wrapper function
from shawnlankton.com.
• Scale-invariant saliency is averaged over each
region.
Saliency-based Cropping
• B. Suh, H. Ling, B. B. Bederson, and D. W.
Jacobs, "Automatic Thumbnail Cropping and
Its Effectiveness," in 16th Annual ACM
Symposium on User Interface Software and
Technology. 2003, pp. 95-104.
• What is the smallest crop that exceeds a
specified saliency threshold (proportion of the
total image saliency)?
– Can be calculated by brute force, but inefficient
A greedy approach
• User specifies a target output size and saliency
threshold.
• Select the most salient region.
• Until we reach our threshold:
– Expand selection to include next most salient region.
– Adjust selection to match aspect ratio of output size.
• Not guaranteed optimal crop
– But much faster
– Guaranteed to include whole regions
Demonstration
• Bounding box in red, correct aspect ratio in
green
Determining Saliency Threshold
Automatically
Saliency vs. Area
1
0.9
0.8
Area (percent of total)
• Find the point of
diminishing returns:
when does increasing
the crop area not
become worth it?
• One simple method:
select the image coming
after the biggest
increase in saliency.
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Saliency (percent of total)
0.8
0.9
1
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Results
Original
Thumbnail
Further possible improvements
• Usability testing
– Test users on how quickly they can locate a
specified image in a set given a table of
thumbnails.
• Tweak region parameters
• Saliency threshold selection
– Try out more sophisticated metrics
• Basic feature recognition
– Face detection