Parker Dunlap
11/15/2013


Semantic image analysis techniques can
automatically detect high level content of
images
Lack of intuitive visualization and analysis
techniques


Allow users to effectively browse/search in
large databases
Allow analysts to evaluate their annotation
process through interactive visual exploration

Target search
◦ User knows exactly what they want, a precise image

Search by association
◦ Find interesting things related to certain image

Category search
◦ Retrieve images that are representative of a certain
class


Semantic contents of images are more useful
for image exploration than low level features
but in most large scale image collections
(internet) semantics are usually not described
This has given rise to techniques that enable
automatic annotation of images according to
their semantic concepts



Contains semantic image classification
process that automatically annotates large
image collections
Contains coordinated visualization techniques
that allow interactive exploration
Contains visualization techniques that allow
analysts to evaluate and monitor annotation
process
1.
2.
3.
Annotation Engine
Image Browsing Interface
Visual Image Analysis



Abstract Image content by detecting
underlying salient objects (distinguishable
regions)
Associate salient objects with corresponding
semantic objects according to their
perceptual properties
Keywords for semantic objects are used to
annotate the image
Highlighted regions are salient objects detected and associated
with
semantic object “sand field”


Goal is to bridge the gap between low-level
visual features and high-level semantic
concepts
Annotation engine has set of predefined
salient objects and functions to detect them
from images
◦ Uses techniques like image segmentation and SVM
classifiers

Annotation engine assigns a semantic
concept to the data based on semantic
content
◦ Sand, Field, Water → Seaworld
◦ Flowers, Trees → Garden

Image overview using MDS
◦ Use the annotations to calculate distance matrix
and input into MDS algorithm
 Distance between each pair of images in the content
space
◦ Algorithm outputs a 2D position for each image
based on similarity with other images



Maps image miniatures onto the screen based
on their content similarities
Similar images placed closer to each other
Goal of MDS is to map some high dimensional
data into lower dimension (in our case 2D)
◦ To learn more about MDS see MDS Overview



Visually represents the contents of the entire
collection of images
Correlations of different contents and
detailed annotations are displayed
Interactively exploring large datasets with
real time response (high dimensionality)




Block of pixels to represent images contents
Each image is mapped to a pixel whose color
indicates if the image contains/doesn’t
contain the content for that block
Pixel representing the same image is the
same for all blocks
Allows us to observe content of image
collection by scanning labels of the blocks




Can see correlations among the contents
Can also select images to see them
highlighted in the view
Position of the blocks are determined by
similarity with neighboring contents
Pixels are generally created in a spiral
arrangement starting from the center and
moving out

Pixel order can greatly effect the looks of VaR
view

To increase scalability, interface users
miniature versions of images
◦ High res original pictures would increase load times

Load image miniatures as textures objects in
OpenGL
◦ Allows all interactions to be done in real time

To reduce clutter in the MDS overview, the
system provides many interactions
◦
◦
◦
◦
◦
◦
◦
Reordering
Dynamic Scaling
Relocation
Distortion
Showing Original Image
Zoom
Pan

Reordering
◦ Randomizing order of all images allows each frame
to have an equal probability of being visible
◦ User can also explicitly bring certain image to the
front by selecting it

Dynamic Scaling
◦ Interactively reduce image miniature size to reduce
overlap or increase image size to examine detail

Relocation
◦ Manually change position of individual image by
dragging and dropping

Distortion
◦ Enlarge size of certain image(s) while retaining size
of all others

Showing Original Image
◦ Actual image (instead of scaled down image used
by OpenGL) opens at full resolution in new window
◦ Only loaded when requested to save space/time

Zoom/Pan
◦ Zoom in/out and pan left/right

Can use multiple techniques at once to
achieve some goal
◦ Use Dynamic Scaling with zooming in to examine
local details with less clutter

Selection
◦ Interactively select a sample image to see similar
images in display
◦ Can change similarity threshold via a slider to
increase/decrease number of results

Sorting
◦ Images can be sorted by concepts or similarity to
selected image



Inspired by rainfall animation
Correlations between image of interest and
other images are modeled through an
animation
Focus image is on the bottom (ground) and
the other images fall to the ground (rain) at
accelerations related to their similarity





Search for images with/without certain
content
Reduce a selected subset by requiring images
must/not contain certain content
Increase selected subset by adding new
images
All these functions done by clicking on
images while holding certain function key
Offers many similar interactions as MDS as
well



Each image has its visual representations in
both MDS and VaR views
Selected images are highlighted in both views
Can use appropriate view as needed
◦ MDS to select image based on relationship to
sample image
◦ VaR to select image based on content

Common strategy is to start from VaR and
switch to MDS after number of images has
been greatly reduced



We can use the MDS and VaR views to see
how well the annotations of images
correspond to their actual content
Select “red-flower” images from VaR view and
verify using MDS view to see if the images
are actually red flowers
If automatic annotation makes a mistake,
user can manually annotate image to fix it

VaR display also shows the reliability of the
annotation by surrounding it with a colored
frame
◦ Green is safe to use, Yellow means lower reliability
measure

Reliability measure can be determined from
annotation process or manually set up by
analysts


Comparison of SIB to the sequential
thumbnail view from Microsoft Explorer
Modes used in Microsoft Explorer
◦ Random Explorer – images are randomly sorted
◦ Sorted Explorer – images are sorted according to
semantic concepts generated by the classification
process


10 participants from varying fields
Each subject used both Sorted Explorer and
SIB
◦ Random Explorer was only tested on 3 participants
since expected results were so low

Participants given 3 tasks to perform on 2
data sets
◦ 180 second timeout window
1.
2.
3.
Presented with a particular image and asked
to search for it from the 1100 images in the
data set
Asked to find images containing particular
features (sand, water, sky, etc…)
Asked to approximate what proportion of
the images in the dataset contained
particular contents (% that contain
mountains)

Random Explorer
◦ 2/9 trials failed
◦ 81 seconds was average time with 29 seconds
standard deviation

Sorted Explorer

SIB
◦ 2/30 trials failed
◦ 29 seconds was average time with 20 seconds
standard deviation
◦ 6/30 trials failed
◦ 45 seconds was average time with 26 seconds
standard deviation


Failure in SIB was due to inaccuracy in the
annotation process
SIB tended to be slower than Sorted Explorer
because content names could be confusing
◦ This advantage will decrease as the data set grows
because Explorer provides no overview model


Task 2 had similar results to Task 1
Task 3 was where SIB became dominant




Positive feedback for SIB
Enjoyed Search by content feature the most
Enjoyed MDS overview over Windows explorer
to see entire collection of images at once
Suggested side-by-side views, example
image next to blocks in VaR view



Semantic Image Browser was introduced that
attempts to bride information visualization
with automatic image annotation
MDS image layout that groups images based
on semantic similarities
VaR content display to represent large image
collections



Semantic Image Browser: Bridging
Information Visualization with Automated
Intelligent Image Analysis
Value and Relation Display for Interactive
Exploration of High Dimensional Datasets
MDS Overview