Programming Languages for
Compressing Graphics
Morgan McGuire
Shriram Krishnamurthi
John F. Hughes
Brown University
{ morgan | sk | jfh}@cs.brown.edu
Encoding Images as Programs
Describing Images (1)
setPixel(0,0,BLUE);
setPixel(1,0,BLUE);
setPixel(2,0,BLUE);
…
setPixel(50,0,WHITE);
…
setPixel(100,0,RED);
setPixel(101,0,RED);
…
The Cost of Bandwidth

Major cost of doing business on the web





Yahoo!: 65,000,000 pages/day
Valve: 1M €/software patch
Image compression is a one-shot activity
Low vs. high bandwidth users
Multiresolution
Describing Images (2)
repeat 50 times
repeat 50 times
repeat 50 times
nextRow();
repeat 50 times
…
setNextPixel(BLUE);
setNextPixel(WHITE);
setNextPixel(RED);
setNextPixel(BLUE);
American Flag
Doesn’t compress as well as the French flag in
the “repeat n times” language.
Adding More Primitives
drawRectangle(BLUE, …);
drawRectangle(RED, …);
drawRectangle(WHITE, …);
drawStar(WHITE, …);
…
Is this enough?
Describing Complex Images
Describing Complex Images



JPEG: language consists
of frequency domain
instructions
GIF: language consists
of setPixel and
dictionary lookup
Preferred format
depends on the image
Observations


Images are programs
Even within one language, many possible
descriptions produce similar images



Lossy compression
Description length depends on language
and image complexity
Best compression when the language
matches the image
The Obvious Compression Scheme

Compress the image in several formats



TGA, GIF, JPG, SVG, SWF
Choose the best
Add a byte to the front of the file specifying
the compression language
Problems with the Obvious Scheme



None of the formats may be particularly good
for our image
Even JPEG tops out around 50:1, has serious
artifacts
Lacking ideal features like:



Multiresolution
Time/Space tradeoff
Introduction of new formats requires new
browser plug-ins
Describing Complex Images
How can we do better than
the obvious approach for
images like this?
Using an Expressive Language
(compose
(vertical-gradient BLUE WHITE)
(polygon DARK-BLUE …)
(polygon BLACK …)
(* (polygon …)
(blur tree-texture)
YELLOW))
Using an Expressive Language


What if we design a really expressive language for
representing images?
Because the “data” is a program the decompressor is
part of the “data”




Each image gets its own custom format
High compression
We have control over multiresolution, perceptual
artifacts
Package for the web as a plug-in

We only need to upgrade the plug-in when the language itself
changes
What’s wrong with this plan?
Example
setPixel Language
Expressive Language
setPixel(0,0,WHITE);
?
(blur (+
setPixel(1,0,WHITE);
(rotate (banana))
setPixel(2,0,WHITE);
(distort (* RED
…
2 Mb
(circle))) …) 2 kb
The Encoding Problem
?
Pierre can’t code!
The Encoding Problem
Photoshop menu
setPixel(0,0,WHITE);
EMACS
(blur (+
setPixel(1,0,WHITE);
(rotate (banana))
setPixel(2,0,WHITE);
(distort (* RED
…
2 Mb
(circle))) …)
2 kb
Where is the “Save as Code” Menu?



It is easy to convert from an image to
programs in the GIF/JPG language
More expressive language = harder
conversion
How much harder?
Much Harder!



Converting the image into a program
that produces it is a search problem
The search space is the space of all
possible programs
This is an infinitely large space
Tempering Expressiveness
“Good” compression languages are
ones where:
 Expressive power is large
 Searching is easy
How do we make searching easy?
Steerable Search Techniques

Genetic Algorithms


Metropolis Search


Inject domain information through fitness function
Inject domain information through transition
probabilities
Simulated Annealing

Inject domain information through gradient
estimation
Perceptual Fitness Function
1 
 40 e(i ) x  e( g ) x  10 ix  g x
m(i, g ) 
100n 

 35 b(i ) x  b( g ) x  15  ix   g x 


e (i ) = edge filter, b (i ) = convolve with Gaussian,
|ix | = color magnitude


Tweaking this is the domain-expert’s job
Perfect fitness function not necessary (or
possible!)
Designing the Language

Desirable language properties for
compression



Expresses many images compactly
There are many programs for which
another, shorter program exists that
produces visually similar output
Desirable search properties


Mutations safety
All programs terminate
The Evolver Language
Scalar
:= real between 0 and 1
Vector
:= Scalar x Scalar x Scalar
Matrix
:= Vector*
Value
:= Matrix | Scalar | Vector
Operator
:= Add | Collage | Blur | Noise | …
Call
:= Operator x Expr*
Expr
:= Call | Scalar | Vector
The Evolver Language





Automatic coercion between Matrix,
Vector, Scalar
Every operator has the same domain
and range
Primitives include stock images and
textures
No looping constructs
No functions!
l: Not the Ultimate Compressor!
l: Not the Ultimate Compressor!

Copying code is sometimes good


Multiple instances of a pattern in an image
often differ slightly
Hard to evolve both definition and
multiple applications
Added Benefits of Search


Artist can save the image immediately
Webmaster uses Evolver toolkit to search for
an equivalent program





It is easy to find a large program quickly
Webmaster lets Evolver continue to run
Upload new, smaller encodings as they are
found
More time = less space = less cost
Multiple constraints: size, time, artifacts,
decompression time
Results
Proof of Concept
TGA 1:1 (128x128)
Proof of Concept
JPG 24:1 (64x64)
Proof of Concept
Evolver 50:1, infinite detail resolution
Proof of Concept
TGA
Evolver
JPG
Lake Matheson
Original
Lake Matheson
Original
Compressed 50:1
Gradient
Original
Compressed
Aspens
Original
Aspens
Original
Compressed 54:1
Maples
Original
Maples Code
Collage(HueShift(HueShift(Min(RockImage(), Collage(Rotate(0.22693111, {0.4339271, -0.060890462, -0.14983156}),
{0.9689341, -0.31166452, 1.0}, EnvironmentMap(Interpolate(Derivative({0.5260445, -0.9943271, -0.83629435}),
FishImage(), 0.22693111), Collage(VSplit(VGradient(), Interpolate(Min(RockImage(), Collage(0.22693111, {0.90638816, 0.3161332, 1.0}, EnvironmentMap({0.3538252, -0.11179591, 0.76402247}, {0.3538252, -0.11179591, 0.76402247}))), 0.75621074, Derivative(HueShift(LowColorNoise(), {-0.60136193, -0.9961748, 0.956824})))), {-0.6025814, -0.5151359, 0.2444776}, MiniBlur(Blur({0.48381335, 0.37744927, 0.18049468})))))), LeafImage()), LeafImage()),
Rotate(Rotate(Interpolate(Max(Rotate(LeafImage(), Cosine({0.13357106, -0.48899084, 0.46273336})), LeafImage()),
{0.26036343, -0.2474052, 0.3318561}, Add(Rotate(Interpolate(Max(LowNoise(), LeafImage()), {0.26036343, -0.2474052,
0.3318561}, Add(Rotate(Interpolate({0.26036343, -0.2474052, 0.3318561}, {0.26036343, -0.2474052, 0.3318561},
Max(Rotate(Blur(Noise()), FrequencyStars()), BitAnd(VSplit(LeafImage(), {-0.3782095, 0.06973941, 0.7708523}),
Collage(Blur({0.19214037, 0.7060751, 0.9632803}), {-0.94875985, 0.9535051, 0.9628181}, {-0.94875985, 0.9535051,
0.9628181})))), FrequencyStars()), Zoom({0.84155905, 0.44450688, -0.6368634}, Interpolate(0.97325927, {0.43938103,
0.8003519, -0.8865588}, FrequencyStars())))), FrequencyStars()), Zoom({1.0, 0.35874906, -0.42753658}, {1.0,
0.35874906, -0.42753658}))), FrequencyStars()), {0.30287892, -0.7879979, 0.756324}),
Distort(Distort(HueShift(Distort(Distort(Distort(HueShift(Distort(Rotate(Distort(Distort(HueShift(Distort(Rotate(Distort(Disto
rt(Blur(Distort(Distort(RockImage(), ArcTangent(ExpandRange(ColorNoise()))),
ArcTangent(ExpandRange(ColorNoise())))), ArcTangent(ExpandRange(ColorNoise()))), ArcTangent(ColorNoise())),
RockImage()), ArcTangent(ColorNoise())), RockImage()), ColorNoise()), Blur(Add({0.214469, -0.05106278, -0.8334819},
Blur(Blur({-0.44914088, 0.86714524, -0.038012877}))))), Distort(Distort(Blur(Distort(Distort(-0.32682085, SunriseImage()),
ArcTangent(0.84263784))), LeafImage()), ArcTangent(ColorNoise()))), ArcTangent(ColorNoise())),
ExpandRange(RockImage())), 0.315652), ColorNoise()), Blur(ExpandRange(Add({0.44753784, 0.15750253, -0.9017423},
{0.214469, -0.05106278, -0.8334819})))), ExpandRange(RockImage())), ColorNoise()), Blur(Rotate({0.10588625,
0.2359776, -0.20337643}, {0.2809281, -0.97692156, -0.49766022}))))
Maples
Original
Compressed 56:1
Multi-resolution
JPEG 14:1
Evolver 56:1
Multi-resolution
JPEG 14:1
Evolver 56:1
Related work

Searching for programs





Palsberg, Lucier & Mamillapalli
Karl Sims
Programmatic image compression



Massalin’s Superoptimizer
Frigo & Johnson’s FFTW
Fractal compression
MPEG-7
Steerable search techniques in graphics


MLT
Radiosity
Conclusions



It is tractable to search the space of all
programs!
The “visually similar” criterion makes
computer graphics an interesting domain
Keep using JPEG for now…
Future directions




Improve image quality/compression
How can the design of searchable
languages be formalized?
How do expressive constructs affect the
search problem?
Other interesting domains: animation,
audio compression, image search, robot
controllers
Questions
http://www.cs.brown.edu/people/morgan/evolver