Perceptually Guided
Interactive Rendering
David Luebke
University of Virginia
Always start with a demo…
Motivation:
Preaching To The Choir

Interactive rendering of large-scale
geometric datasets is important
 Scientific
and medical visualization
 Architectural and industrial CAD
 Training (military and otherwise)
 Entertainment
Motivation:
Model Size

Incredibly, models are getting bigger as
fast as hardware is getting faster…
Big Models:
Submarine Torpedo Room

700,000 polygons
Courtesy General Dynamics, Electric Boat Div.
Big Models:
Coal-fired Power Plant

13 million polygons
(Anonymous)
Big Models:
Plant Ecosystem Simulation

16.7 million polygons (sort of)
Deussen et al: Realistic Modeling of Plant Ecosystems
Big Models:
Double Eagle Container Ship

82 million polygons
Courtesy Newport News Shipbuilding
Big Models:
David:
56,230,343 polygons
 St. Matthew:
372,422,615 polygons

Courtesy Digital Michelangelo Project
The Digital Michelangelo Project
Motivation: Level of Detail
Clearly, much of this geometry is
redundant for a given view
 The basic idea: simplify the model,
reducing the level of detail used for:

 Distant
portions
 Small portions
 Otherwise unimportant portions
Traditional Level of Detail
In A Nutshell…

Create levels of detail (LODs) of objects:
249,924 polys
62,480 polys
Courtesy Jon Cohen
7,809 polys
975 polys
Traditional Level of Detail
In A Nutshell…

Distant objects use coarser LODs:
The Big Question
How should we evaluate and regulate the
visual fidelity of our simplifications?
Regulating LOD

LOD is often controlled by distance…
d1
d2
Courtesy Martin Reddy
Regulating LOD

…or by size
Measuring Fidelity

Fidelity of a simplification to the original
model is often measured geometrically:
METRO by Visual Computing Group, CNR-Pisa
Measuring Visual Fidelity

However…
 The
most important measure of fidelity is
usually not geometric but perceptual: does
the simplification look like the original?

Therefore:
 We
are developing a principled framework for
LOD in interactive rendering, based on
perceptual measures of visual fidelity
Perceptually Guided LOD

Several interesting offshoots:
 Imperceptible

How to guarantee simplification is undetectable?
 Best-effort

simplification
How best to spend a limited time/polygon budget?
 Silhouette

preservation
Silhouettes are important. How important?
 Gaze-directed

simplification
rendering
When can we exploit reduced visual acuity?
Related Work

Lots of excellent research on perceptually
guided rendering
 Bolin
& Meyer (SIGGRAPH 98)
 Ramasubramanian et al (SIGGRAPH 99)

But all this work has focused on realistic
rendering algorithms (e.g., path tracing)
 Different
time frame!
 Seconds or minutes versus milliseconds
Related Work
As a result, prior work has incorporated
quite sophisticated perceptual metrics
 Our goal: a simple, conservative
perceptual metric fast enough to run
thousands of times per frame

The Approach
The contrast sensitivity function or CSF
measures perceptibility of visual stimuli
 We test local simplification operations
against a model of the CSF to determine
whether they would be perceptible

Perception 101:
The Contrast Sensitivity Function

Perceptual scientists have long used
contrast gratings to measure limits of vision:
 Bars
of sinusoidally
varying intensity
 Can vary:





Contrast
Spatial frequency
Eccentricity
Velocity
Etc…
Perception 101:
The Contrast Sensitivity Function

Contrast grating tests produce a contrast
sensitivity function
 Threshold
contrast
vs. spatial frequency
 CSF predicts the
minimum detectable
static stimuli
Your Personal CSF
Campbell-Robson Chart by Izumi Ohzawa
Contrast Sensitivity Function:
An Empirical Model

The CSF is affected by many factors
 Background
illumination, adaptation, age, etc
 Attentive focus

We chose to sidestep these issues by
building an empirical model (lookup table)
 User
foveates on target, grating fades in
 Measuers threshold contrast across different
spatial frequencies, eccentricities
Contrast Sensitivity Function:
Complex Waveforms

The perceptibility of a complex signal is
determined by its harmonic components
 If
no frequency component of an image
feature is visible, the feature is imperceptible
and may be removed without visible effect
 This is the key idea that will allow us to
simplify the model

Next: need a framework for simplification
Framework:
View-Dependent Simplification

We use view-dependent simplification for
LOD management
 Traditional
LOD: create several discrete
LODs in a preprocess, pick one at run time
 Continuous LOD: create data structure in
preprocess, extract desired LOD at run time
 View-dependent LOD: extract most
appropriate LOD for the given view
View-Dependent LOD:
Examples

Show nearby portions of object at higher
resolution than distant portions
View from eyepoint
Birds-eye view
View-Dependent LOD:
Examples

Show silhouette regions of object at
higher resolution than interior regions
View-Dependent LOD:
Examples

Show more detail where the user is
looking than in their peripheral vision:
34,321 triangles
View-Dependent LOD:
Examples

Show more detail where the user is
looking than in their peripheral vision:
11,726 triangles
View-Dependent LOD:
Implementation
We use VDSlib, our public-domain library
for view-dependent simplification
 Briefly, VDSlib uses a big data structure
called the vertex tree

 Hierarchical
clustering of model vertices
 Updated each frame for current simplification
The Vertex Tree

Each vertex tree node represents:
A
subset of model vertices
 A representative vertex or proxy
Folding a node collapses its vertices to
the proxy
 Unfolding a node splits the proxy back
into vertices

Vertex Tree Example
8
7
R
2
I
10
II
6
9
3
10
1
A
1
4
2
B
7
4
5
C
6
8
5
Triangles in active list
Vertex tree
3
9
Vertex Tree Example
8
7
R
2
A
I
10
II
6
9
3
10
1
A
1
4
2
B
7
4
5
C
6
8
5
Triangles in active list
Vertex tree
3
9
Vertex Tree Example
8
R
A
I
10
II
6
9
3
10
A
1
4
2
B
7
4
5
C
6
8
5
Triangles in active list
Vertex tree
3
9
Vertex Tree Example
8
R
A
I
10
II
6
9
3
10
B
4
A
1
2
B
7
4
5
C
6
8
5
Triangles in active list
Vertex tree
3
9
Vertex Tree Example
8
R
A
I
10
9
II
3
10
B
Triangles in active list
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
8
R
A
C
9
I
10
II
3
10
B
Triangles in active list
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
A
C
I
10
II
3
10
B
Triangles in active list
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
A
C
II
I
10
II
3
10
B
Triangles in active list
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
A
II
I
10
10
B
Triangles in active list
II
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
A
I
II
I
10
10
B
Triangles in active list
II
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
I
II
I
10
B
Triangles in active list
II
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
I
II
I
II
R
10
B
Triangles in active list
A
1
2
B
7
4
5
C
6
8
Vertex tree
3
9
Vertex Tree Example
R
I
II
R
10
A
1
Triangles in active list
2
B
7
4
5
C
6
8
Vertex tree
3
9
The Vertex Tree:
Tris and SubTris

Node folding is the fundamental
simplification operation:
8
7
8
Fold Node A
2
A
10
10
6
9
9
3
1
4
6
3
Unfold Node A
5
4
Some triangles change shape upon folding
 Some triangles disappear completely

5
Perceptually Guided LOD:
Key Contribution

Our key contribution: a way to evaluate
the perceptibility of a fold operation
 Equate
the effect of the fold to a worst-case
contrast grating


Find the worst-case contrast induced in the image
Find the worst-case spatial frequency
Perceptually Guided LOD:
Key Contribution

Our key contribution: a way to evaluate
the perceptibility of a fold operation
 Equate
the effect of the fold to a worst-case
contrast grating

Find the worst-case contrast induced in the image
Bounded by the maximum change in luminance!

Find the worst-case spatial frequency
Perceptually Guided LOD:
Key Contribution

Our key contribution: a way to evaluate
the perceptibility of a fold operation
 Equate
the effect of the fold to a worst-case
contrast grating

Find the worst-case contrast induced in the image
Bounded by the maximum change in luminance!

Find the worst-case spatial frequency
Bounded by the minimum spatial frequency (in our case)
Perceptually Guided LOD:
Key Contribution

Our key contribution: a way to evaluate
the perceptibility of a fold operation
 Equate
the effect of the fold to a worst-case
contrast grating

Find the worst-case contrast induced in the image
Bounded by the maximum change in luminance!

Find the worst-case spatial frequency
Bounded by the minimum spatial frequency (in our case)
= bounded by greatest possible spatial extent!
Worst-Case Contrast



Find maximum possible
change in color
Map to luminance, then
to contrast
This is the largest
contrast that the fold
could possibly induce in
the final image
Original
Color
Change
Simplified
Worst-Case Contrast

Find maximum possible
change in color



Original
Note: depends on
silhouette status!
Map to luminance, then
to contrast
This is the largest
contrast that the fold
could possibly induce in
the final image
Color
Change
Simplified
Worst-Case Spatial Frequency

Lower frequencies = more perceptible
At least, where we are concerned
 Can enforce this assumption

 = Size

Minimum spatial frequency
determined by projected
screenspace extent of node
Signal representing
maximum change
produced by node
simplification
Bringing It All Together

If simplifying a region is
imperceptible, go ahead
and simplify!
Original
Simplified
Imperceptible Simplification

Imperceptible simplification: only fold
nodes whose effect is predicted to be
imperceptible
 It
works! Verified with simple user study
 Problem 1: overly conservative
 Problem 2: nobody cares


Important result, important issues, but…
If you need imperceptible simplification that badly,
you probably won’t simplify at all
Imperceptible Simplification:
Results
69,451 polygons


wireframe
Here, the user’s gaze is 29 degrees from the bunny
Silhouettes and strong details preserved:



29,866 polygons
Line of the haunch
Shape of the ears
But subtle (low-contrast) details removed

E.g., top of the leg
Best-Effort Simplification

More pertinent: best-effort simplification to
a budget
 Idea:
order nodes to be folded based on the
distance at which you could perceive the fold
 Nice, physical error metric

After simplifying to (say) 50K tris, system can
report, “this would be imperceptible from 8 feet.”
Best-Effort Simplification:
Results
96,966  18,000 faces
Standard VDSlib error metric
(projected screenspace size)
96,966  18,000 faces
Perceptual error metric
(contrast + spatial frequency)
Silhouette Preservation

Researchers:
 Have
long known silhouettes are important
 Have long used heuristics to preserve them

Our model gives a principled basis for
silhouette preservation by accounting for
the increased contrast at silhouettes
 Detect
silhouette nodes using a quantized
normal cube
 Set contrast to maximum for silhouette nodes
Gaze-Directed Rendering:
Eccentricity

Visual acuity falls
off rapidly in periphery
 Fovea:
central few degrees
of vision
 35-fold reduction from
fovea  periphery
 Eccentricity: angular distance
from center of gaze

Gaze-Directed Rendering:
Eccentricity

Can model the falloff of
acuity with eccentricity in CSF:
 = Size
Eccentricity
Gaze-Directed Rendering:
Velocity (Future Work!)

Visual acuity also falls
off for fast-moving objects
 Eye
1 deg/s
tracking object: render
background at lower resolution
 Eye tracking background: render
object at lower resolution
 Very powerful in conjunction with
eccentricity!
20 deg/s
Gaze-Directed Rendering:
Velocity (Future Work!)

Can model the effect of retinal velocity on
the CSF:
Extending The Framework:
Other Rendering Paradigms
This framework applies to almost any
hierarchical rendering technique
 We have extended it to QSplat, the pointbased renderer of Rusinkiewicz and Levoy

 Hierarchy
of bounding spheres
 Used for simplification, culling, backface
rejection, and rendering
 Heavily optimized for extremely large models
Extending The Framework:
QSplat

Promising results from QSplat prototype:
QSplat’s highest quality:
2.9 million splats
Gaze-directed QSplat:
0.8 million splats (29o)
Extending the Framework:
QSplat
QSplat’s highest quality:
simplified points in blue
Gaze-directed QSplat:
user’s eye on torch
Summary

Novel framework for interactive rendering
 Based
directly on perceptual metric (CSF)
 Applied to polygonal simplification & QSplat

Addresses several interesting issues:
 Imperceptible
& best-effort simplification
 Silhouette preservation
 Gaze-directed rendering

Still in nascent form, but an important start
Future Work

Lots of opportunities for future research!
 Improve


Dynamic lighting using normal masks
Address overly conservative contrast & frequency
estimates using texture deviation metric (APS)
 Extend



the current system:
the perceptual model, incorporating:
Retinal velocity
Visual masking using texture content frequencies
Temporal contrast (“flicker”) sensitivity
Gaze-Directed Rendering:
Applicability

Gaze-directed rendering clearly has limits
 Eye
tracking not yet commodity technology
 But head tracking may turn out quite useful…




Gaze direction stays within 15o of head direction
Video head tracking increasingly mature
Wide-area FOV displays increasingly common
Even with multiple viewers, may still get lots of
simplification in right environments.
Acknowledgements

Students:
 Ben
Hallen
 Keith Shepherd, Dale Newfield, Tom Banton

Colleagues:
 Martin
Reddy
 Ben Watson

Funding:
 National
Science Foundation
The End
Questions?
Appendix:
References

Perceptually guided offline rendering:
Bolin, Mark. and G. Meyer. “A Perceptually Based Adaptive Sampling
Algorithm”, Computer Graphics, Vol. 32 (SIGGRAPH 98).
Ferdwada, James, S. Pattanaik, P. Shirley, and D. Greenberg. “A Model
of Visual Masking for Realistic Image Synthesis”, Computer Graphics,
Vol. 30 (SIGGRAPH 96).
Ramasubramanian, Mahesh, S. Pattanaik, and D. Greenberg. “A
Perceptually Based Physical Error Metric for Realistic Image
Synthesis”, Computer Graphics, Vol. 33 (SIGGRAPH 99).
Appendix:
References

Perceptually guided interactive rendering:
Reddy, Martin. “Perceptually-Modulated Level of Detail for Virtual
Environments”, Ph.D. thesis, University of Edinburgh, 1997.
Scoggins, Randy, R. Machiraju, and R. Moorhead. “Enabling Level-ofDetail Matching for Exterior Scene Synthesis”, Proceedings of IEEE
Visualization 2000 (2000).

Gaze-directed rendering:
Funkhouser, Tom, and C. Sequin. “Adaptive display algorithm for
interactive frame rates during visualization of complex virtual
environments”, Computer Graphics, Vol. 27 (SIGGRAPH 93).
Oshima, Toshikazu, H. Yamammoto, and H. Tamura. “Gaze-Directed
Adaptive Rendering for Interacting with Virtual Space”, Proceedings of
VRAIS 96 (1996).
Appendix:
References

View-dependent simplification:
Hoppe, Hughes. “View-Dependent Refinement of Progressive Meshes”,
Computer Graphics, Vol. 31 (SIGGRAPH 97).
Luebke, David, and C. Erikson. “View-Dependent Simplification of
Arbitrary Polygonal Environments”, Computer Graphics, Vol. 31
(SIGGRAPH 97).
Xia, Julie and Amitabh Varshney. “Dynamic View-Dependent
Simplification for Polygonal Models”, Visualization 96.


This research:
Hallen, Benjamin and David Luebke. “Perceptually Guided
Interactive Rendering”, UVA tech report CS-2001-01. See
http://www.cs.virginia.edu/~luebke/temp/tech.report.pdf
VDSlib (software library): http://vdslib.virginia.edu