The Versus Comparison
Framework
Kenton McHenry, Ph.D.
Research Scientist
National Center for Supercomputing Applications
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
The Problem
• The abundance of file formats is a problem when
preserving electronic records
• Why?
• Will there be software to load the file in the future?
• If not will the specification for the format still exist?
• Was the specification ever available to begin with
(closed/proprietary formats)?
*.pdf (*.prc, *.u3d)
*.ma, *.mb, *.mp
*.k3d
*.w3d
*.vtk,
*.vtp
*.lwo
*.c4d
*.dwg
*.blend
*.iam *.skp
*.max, *.3ds
Available 3D File Formats…
Converting Formats
• In order to preserve content for future use one option is
to convert the file to an open/standardized format that is
likely to be supported for some time.
• Store both this file and the original for provenance
• Ideally with one file format for a particular content type it
will be easy for users to view/use the data.
Converting Formats (continued…)
• How and which format!?
• Fully supporting the many available formats is an enormous
undertaking
• If a file format is closed/proprietary it may be difficult to retrieve
the data directly from the file
• May be possible to reverse engineer and recover some of the
content
• Vendor file formats sometimes store application feature specific
pieces of information that’s not supported in other formats
• Examples include: animations, physics, …
• When converting to a format that doesn’t have a place for
such information we must drop it.
• Information loss…
Converting Formats (continued…)
• There are different ways of storing the 3D content itself
• Faceted:
• Comprised of vertices and faces
• Popular within the graphics community
• Boundary Representation:
• Comprised of vertices, edges, edge loops, and primitive
surfaces
• Popular among CAD users
• Constructive Solid Geometry
• Comprised of boolean operations on primitive volumes
• …
Converting Formats (continued…)
• Translating geometry representation may not be trivial
• B-Rep to Faceted
• Translating involves triangulating the surfaces created from
the bounded primitives (tesselation)
• The resulting sampled surface will suffer from aliasing at high
viewing resolutions
• Can accommodate by performing a finer triangulation (i.e.
more triangles and a larger file)
• Faceted to B-Rep:
• Translating in this direction is non-trivial!
• How does one decide if a group of triangles should be
grouped together as part of some larger primitive (e.g. part of
a cylinder).
Converting Formats (continued…)
Format
Geometry
Appearance
CSG
Faceted
Parametric
3ds
√
√
igs
√
√
lwo
√
√
√
√
√
√
obj
√
√
√
√
√
√
ply
√
√
√
√
√
stp
√
√
wrl
√
√
u3d
√
x3d
√
√
√
B-Rep
Scene
√
√
Color
Material
Texture
Bump
Lights
Views
Trans.
√
√
√
√
√
√
√
√
√
√
√
Groups
√
√
√
√
√
√
√
Animation
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
• How do we measure information loss?
• Which is the best format to use for preservation?
NCSA Polyglot (2009)
• Conversions service based on utilizing any and all
available 3rd party software
• Imposed Code Reuse: Re-attaching a programmable interface to
compiled software.
• Scripted operations within software
• GUI scripting (e.g. AutoHotKey)
• Created a simple workflow referred to as an Input/Output Graph
• Compared files before/after conversion to measure information
loss
• Distributed across multiple machines
• Web access
ISDA File Migration Tools
•
•
•
•
Conversion Software Registry
Software Servers
Polyglot
Versus
Software that can Convert between Formats
• There is a lot of software available, each with its own
unique capabilities
• A lot of it is not free
• It would be expensive to buy a package just to check if it truly is
capable of converting between a desired pair of formats
• How can someone know what software to get for their
needs?
http://isda.ncsa.illinois.edu/NARA/CSR
The Conversion Software Registry
Adobe 3D Reviewer
The Conversion Software Registry
Input/Output Graphs
Adobe 3D Reviewer
Input/Output Graphs
3DS Max
Adobe 3D Reviewer
AutoCAD
Blender
Cinema 4D
K-3D
LightWave 3D
Maya
Wings 3D
Input/Output Graphs
Shortest conversion path
Software Servers
Software Server
• To program against software
• i.e. to write new code that can utilize functionality within arbitrary
software, compiled code, where the source code is probably not
available.
• Imposed Code Reuse (or Software Reuse): The
process of attaching an API like interface to software so
that its functionality can be called within new code.
Software Servers
• Shares the functionality of software over the web
• In contrast to services which share data: ftpd, nfsd, sambad,
httpd
• Similar to services such as: telnetd, sshd, VNC, rdesktop
• The main difference is in the interface:
• Uniform across all software
http://host:8182/software/<Application>/<Task>/<Output Format>/<InputFile>
• Simple
• Widely accessible
• Capable of being programmed against
• Allows any desktop application to become a cloud based
web service*
Software Functionality Sharing
Software Functionality Sharing
Software Functionality Sharing
Software Functionality Sharing
Software Functionality Sharing
Software Functionality Sharing
Software Functionality Sharing
Software
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Excel 2010
Word 2010
Power Point 2010
Publisher 2010
One Note
Access 2007
Wordpad
Notepad
Calculator2010
Internet Explorer 8
Winzip
Photoshop CS5
Adobe Acrobat
ABBYY
•
•
•
•
•
•
•
•
•
•
•
•
•
•
3D Studio Max
Adobe 3D Reviewer
Google Sketchup
Wings 3D
Blender
K3D
Paraview
VTK
Cyberware PLYTool
NIST X3D Tool
Imagemagick
IrfanView
GIMP
Microsfot Paint
Polyglot
• Listens for Software Server broadcasts on the network
• Catalogues available input/output operations and
constructs and I/O-graph
• Identifies conversion paths between input and output
formats
• Carries out CHAINED conversions
Polyglot
Polyglot
Versus
• Java library/framework for comparing file content
• Distributed architecture
• RESTful Web Interface
• http://<host>/versus/comparisons
• dataset1, dataset2
• adapter, extractor, measure
Adapters
Name
Package
Description
Mesh
3D
Audio
Audio
Encapsulation of audio files.
Bytes
Core
Simplest possible representation of data.
PDF
Doc2Learn
Buffered Image
Image
Standard Java representation of image data.
Image Object
Image
Encapsulation of the Im2Learn Image Object.
SIFT GPU
GPU
Load 3D files content into a mesh made up of vertices and polygons
connecting those vertices.
Encapsulation of the Doc2Learn PDF document.
Encapsulation of image data for SIFT Gpu specific processing.
Descriptors
Name
Double Array
Package
Core
MD5 Digest
Core
Three Dimensional Double
Array
Vector
Core
Label Histogram
Doc2Learn
Keypoint
Pixel
Image
Image
Color Layout
Image
Grayscale Histogram
RGB Histogram
Pixel Histogram
Image
Image
Image
MOPS Features
SIFT Features
Fiji
Fiji
SIFT Gpu
Gpu
Harris Corners
OpenCV
Hough Circles
OpenCV
Hough Lines
OpenCV
SURF Features
OpenCV
Core
Description
A single dimensional array containing double
values.
A data integrity structure generated from the raw
data.
A three-dimensional array containing double
values.
A list of generic elements, allows greater storage
flexibility.
A histogram of labels obtained through
Doc2Learn.
Generic container for invariant feature detectors.
Generic type for various image package
descriptors.
A two dimensional grid of sub-images over the
input image.
A one-dimensional grayscale image histogram.
A three-dimensional RGB color histogram.
A multidimensional histogram for a pixel’s
intensity and position.
Invariant feature type used for image stitching.
Popular invariant feature type used for image
comparison and object matching.
Same as SIFT but implemented through Gpu
libraries.
Well-known corner detector used for image
inference, tracking, and recognition.
Circles detected in an image with the Hough
Transform.
Lines detected in an image with the Hough
Transform.
Invariant feature type that can be computed faster
than standard SIFT.
Extractors
Name
Light Field
Package
3D
Description
Surface is represented by silhouettes taken from 3 canonical
positions capturing the surface shape minus any concavities
(i.e. the convex hull).
Statistics
3D
Ignores the surface and focuses on the vertices of a 3D object
returning their mean and standard deviation. Simple, but fast
to compute.
Surface Area
3D
The sum of the area occupied by the polygons making up a
surface. Considers surface and is still fast to compute.
Audio
Audio
Sampling of audio from existing file for histogram usage and
comparison.
MD5
Image Histogram
Core
Doc2Learn
Creation of the MD5 hash from data.
Generates a non-standard color histogram
Line Graphics Histogram
Doc2Learn
Generates a histogram to compare vector graphics found in
documents.
Text Histogram
Doc2Learn
Generates a label histogram based on word frequency.
Array Feature
Image
Generates the three-dimensional double array; a generic
image container.
Color Average Vector Feature
Image
Generates an average RGB color over 9 regions taken from
the image.
Grayscale Histogram
Image
Generates the histogram for grayscale images. Useful for
image comparison.
Pixel Histogram
Image
Generates the multidimensional histogram for feature
matching.
RGB Histogram
Image
Generates the histogram for color images. Useful for image
comparison.
Signature Vector
Image
Feature vector (for an image) containing colorspace
information and pixel position.
MOPS Features
Fiji
Open source implementation for the MOPS detector.
SIFT Features
SIFT Gpu
Harris Corners
Hough Circles
Hough Lines
SURF Features
Fiji
Gpu
OpenCV
OpenCV
OpenCV
OpenCV
Open source implementation of Lowe’s method.
Gpu implementation for the SIFT detector.
Corner detector for images.
Circle detector for images
Line detector for images
Open source implementation for the SIFT detector.
Measures
Name
Package
Chessboard Distance
Core
Dynamic Time Warping
Core
Euclidean Distance
Core
Manhattan Distance
Core
MD5 Hash
Core
Bhattacharyya Distance
Image
Neyman’s χ2
Image
Czekanowski Distance
Image
Histogram Euclidean
Distance
Doc2Lear
n / Image
Histogram Intersection
Doc2Lear
n / Image
KL Divergence
Image
Jeffrey Divergence
Image
Motyka Distance
Image
Normalized Cross
Correlation
Image
Ruzicka Similarity
Image
Sum of Squared
Differences
Image
Tanimoto Distance
Image
Wave Hedges Distance
Image
Fiji /
OpenCV /
Invariant Feature
Comparison
SIFT Gpu
Earth Mover’s Distance
OpenCV
Description
Also known as Chebyshev; the greatest
difference along any coordinate dimension
(between two vectors)
Similarity metric between two (possibly)
varying sequences over time.
Distance between two n-dimensional points in
Euclidean space.
Absolute difference of coordinates of points,
distance between two points measured along
right angled axes.
Binary measure; either equal or not.
Measures the overlap between two probability
distributions.
Tests the goodness of fit between two
distributions. Variant of the standard χ2 test.
Sum of the absolute value of the difference of
two distributions divided by the sum of the two
distributions.
Bin-by-bin comparison using the standard
Euclidean distance. Well known and widely
used.
Sum of the absolute value of the difference of
two distributions, scaled by one-half. Well
known and widely used.
Non-symmetric measure of the difference
between two probability distributions. Well
known measure of entropy.
Symmetric measure of the difference between
two probability distributions.
Sum of the maximum of two distributions
divided by the sum of the two distributions.
Similar to sum of squared differences;
invariant to the magnitude of two points.
Sum of the minimum of two distributions
divided by the sum of the maximum of the two
distributions.
Sum of squared differences between two
arrays, cheaper to computer than Euclidean
distance.
Sum of the difference of the max and the min
of two distributions divided by the sum of the
max.
Sum of the absolute value of the difference
between two distributions divided by their
maximum.
Compares the invariant features between two
images by calculating the pairwise Euclidean
distance and voting for a match using a
predetermined threshold.
Measures the similarity between two
probability distributions. This is the minimum
cost of transforming one distribution to the
other.
Extractor Previews
Measuring Information Loss
We would like to assign
a value to each
conversion edge …
• With a “universal” converter we could convert files from
every format A to every other format B
• Assuming we then had a loader for both format A and
format B we could load and compare the 3D content
independent of how it is stored.
Measuring 3D Information Loss
•
•
•
•
•
•
Adobe 3D Reviewer
Blender
Cyberware PlyTool
K-3D
NIST VRML/X3D
VTK
good… (e.g. 1.0)
not so good… (e.g. 0.1)
Measuring 3D Information Loss
• Data representation
• Meshes
• Loaders
• Use 3D similarity as a means of comparing 3D models
•
•
•
•
Statistics
Surface Area [Brunnermeier, RTI 1999]
Spin Images [Johnson, PAMI 1999]
Light Fields [Chen, Eurographics 2003]
Statistics
• Use the mean and standard deviation of the vertices to
represent the model
• Simple but fast to compute
• Sensitive to size and orientation of the model
Surface Area
• Use the sum of face areas to represent the model
• Also simple and fast to compute
• Sensitive to size, somewhat sensitive to shape. Will
detect loss of faces.
Light Fields [Chen, 2003]
• Compares silhouettes from various viewing angles
around a model.
Light Fields
Light Fields
Light Fields
Light Fields
Light Fields
• Fairly fast to compute
• Sensitive to shape of convex hull, invariant to rigid
transformations
Spin Images [Johnson, 1999]
• 2D histograms of the in plane and out of plane distances
of vertices neighboring a given vertex.
N
q
b
p
a
Spin Images
Spin Images
Spin Images
Spin Images
Spin Images
Spin Images
• Expensive to compute
• Sensitive to relative vertex position, ignores surface,
invariant to rotations and translations
3D Information Loss and 3D File Loaders
• If we were able to load every file format we really
wouldn't need to use software reuse to convert via 3rd
party applications.
• Implement loaders for a small number of formats that will
make up our test data set
• Convert from format A along path to some format B then back to
A again
• Estimate path scores by comparing before/after content and
assigning scores to all edges along path
STP to X3D to STP
A3D Reviewer
STP
Vrml97ToX3d
X3dToVrml97
X3D
WRL
A3D
Reviewer
WRL
STP
Software
Application A
Application B
Application C
Application X
Application Y
Application Z
Application
Wrapper A
Application
Wrapper B
Application
Wrapper C
Application
Wrapper X
Application
Wrapper Y
Application
Wrapper Z
…
Software
Reuse
Software Server
…
Software Server Client
Software Server
Software Server Client
Software Server Client
…
Comparison (Versus)
Polyglot Steward
Polyglot Server
Conversion
(Polyglot)
Polyglot Web Server
Conversion (Polyglot)
…
Software Reuse
Polyglot Client
Polyglot Panel
Web Browser
Closed Source Software
Versus
IOGraph Weights Tool
Comparison
(Versus)
…
Which conversion preserved the most?
• Using the light fields measure:
• Emphasizes shape through silhouettes
• Adobe 3D Reviewer between *.pdf and *.stp (61.67)
• Using the spin image measure:
• Emphasize shape through relative vertex positions
• Adobe 3D Reviewer between *.obj and *.pdf (59.07)
Which is the best format?
Within the context of preservation we can define this as the format that retains on average the
most information when converted to by other formats.
• Using the light fields measure:
• Emphasizes shape through silhouettes
• *.stp (40.73)
• Using the spin image measure:
• Emphasizes shape through relative vertex positions
• *.stl (34.89)
• *.stp being a CAD format has more variability in vertex positions
due to tessellation
Word Spotting in Versus
• Include Word Spotting feature code as extractor in
Versus
• Just another form of content based comparison
• Specific to handwriting of course
• Include general purpose efficient indexing into Versus
• e.g. hierarchical agglomerative clustering
• Not just for Word Spotting
Conclusion
Image Software by
Information Preservation
3D Software by
Information Preservation
1.
2.
3.
4.
5.
ImageMagick
Image Formats by
1. Adobe 3D Reviewer
Adobe Photoshop Information Preservation
2. 3DS Max
3D Formats by
GIMP
3. Maya
Information
Preservation
Microsoft Paint
1. PPM
Video Formats by
4. Blender
…
2. PNG Information Preservation
5. …
1. STL3. GIF
2. STP4. JPG 1. AVIImage Formats by
3D Formats by 3.
FileOBJ
Size
File Size
5. …
2. MOV
4. MAX
3. WMV
1. MP4
1. JPG
5. …
4.
MPG
2. MAX
2. GIF
5.
…
3. PLY
3. PNG
4. OBJ
4. PPM
1,682 formats
5. …
5. …
2,007 applications
*Rankings shown demonstrate the type of output that will be obtained from future work and does NOT represent actual results!
Versus and CBR
• CBIR: Content based image retrieval
• Versus as a content based comparison framework can
serve as the back end for general purpose content
based retrieval
Acknowledgements
• This research was partially supported by a
National Archive and Records Administration
(NARA) supplement to NSF PACI cooperative
agreement CA #SCI-9619019 and by NCSA
Industrial Partners.
• The views and conclusions contained in this document are those
of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the National
Science Foundation, the National Archive and Records
Administration, or the U.S. government.
Imaginations unbound
The ISDA Tools (Free and Open Source)
Image, Spatial, and Data
Analysis Group
http://isda.ncsa.illinois.edu
Kenton McHenry
Rob Kooper
Michal Ondrejcek
Luigi Marini