Image Synthesis
Interactive Terrain Rendering
computer graphics & visualization
Motivation
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Mobile Terrain Visualization
• Mobile GPUs
–
–
–
–
250M pixels / second
5M triangles / second
6 Textures
OpenGL ES / Direct3Dm
• Challenges
–
–
–
–
1,25 MB Video-RAM…
Fixpoint formats
Bandwidths
Compression, streaming etc.
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Requirements
• Games
– High, constant frame rates
– Quality advantageous but not priorized
• Simulators
– Constant frame rates (~30fps)
– High degree of realism and detail, optionally stereo
• GIS
– Interactive frame rates advantageous (15+ fps)
– High resolution and precision
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Challenges
Oberbayern and Alps
Heightfield with texture
~ 350 MB
Crater Lake, Oregon, US
Heightfield with bathimetry and texture
~ 500 MB
Puget Sound, WA, USA
Heightfield with texture
~ 1.25 GB
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Challenges
Mars, MOLA mission
Heightfield with texture
~ 4.5 GB
USA complete, USGS data
Heightfield
~ 40 GB
Mars, MarsExpress mission
Heightfield with texture
(Video)
~ 175 GB
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Challenges
Data sets are:
• Large to gigantic
– Data volume is increasing rapidly
• Sometimes multi-modal
– Color, Albedo, Cloud coverage, Scattering properties, …
• Sometimes annoted and commented
– Additional layers with meta data
Efficient Rendering ?
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Terrain:
– A height field over a
rectangular domain
– Given at discrete
sample points
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
Rendering lit and textured triangles
– Vertices, texture coords, colors, normals
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Ray-Tracing
– Ray-tracing triangle meshes
• Trace rays until a triangle is hit
• Implicit occlusion culling
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Ray-Tracing - performance issues
– Mesh data structure has to be stored
– Hierarchical representation necessary for improved
max/min(H)
intersection test
Octree or kD-tree
max/min(H)
max/min(H)
•Store max/min heights for subregions
•Skip regions below minimal height of ray
•Employ ray coherences
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Rendering polygonal models – analysis
– Aliasing due to undersampling of small features
– No exploitation of the LOD-nature
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Rendering techniques for terrains
– Render each triangle separately
VX
V0
VX+1 VX+2 VX+3
V1
V2
V3
...
...
VX-1
– Draw: (v0,v1,vX), (v1,vX,vX+1), ...
– Each vertex is rendered (transmitted, transformed)
six times
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Rendering techniques for terrains
– Render triangle strips: v0,vX,v1,vX+1,v2,vX+1, ...
VX
V0
VX+1 VX+2 VX+3
V1
V2
V3
...
...
VX-1
– Triangle defined by new point and previous two
points
– Each vertex is rendered only once
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Height field pre-processing
– Reduce number of primitives by mesh decimation
– Mesh decimation based on any decimation ctriterion
• Curvature („flatness“)
• Use fewer but larger triangles where the height field is flat
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Even with mesh decimation there are still too
many triangles
– Some features might not be visible because they
are too far from the viewer
• Feature size is smaller than size of a pixel
– Level-of-detail should be adapted to the size of
details when displayed rather than to their size in
world space
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Approach: adaptively decimate the mesh with
regard to the current view
• General idea:
– Top-down approach
– Start with coarse resolution (2 triangles) and
adaptively refine until desired level-of-detail is
reached
– Consider world space and screen space deviation
– Re-build mesh for every frame
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Recursive split operation (as in ROAM)
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Terrain rendering
• Error criterion for split operation
– Deviation in world space
• Difference between height value at center vertex and
average of left and right vertex
– Deviation in screen space
• Determine upper bound for screen space length of world
space difference vector (0,d,0)
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Previous Work
- Vast body of literature… see www.vterrain.org
- Key Insight: Dynamic remeshing too expensive
– Process and render Batches of geometry: (P-)BDAM
• Huge pre-processing requirements
• Hardware requirements (SCSI Raid)
• Average Quality
– Hierarchical, regular Grids: (Geometry Clipmaps)
• Good compression, good framerates
• Error control very hard, quality suffers
• Losasso & Hoppe SIGGRAPH 2004
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Geometry clipmaps
- Terrain as mipmap pyramid
- LOD using nested grids
- LOD is shifted over domain, with
varying height values
Coarsest Level
Finest Level
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Geometry clipmaps
• Advantages:
– Regular structure of nested grids enables fast
rendering using triangle strips
– Only regular and continuous chunks of memory
have to be read
• Disadvantages:
– Does not allow to control local world/pixel error
• Renders more triangles than necessary
• Introduces a significant higher pixel error in general
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
GPU-friendly terrain rendering
• Tile-based, restricted Quadtree
• Lmax error metric allows for precise error control
• Low pre-processing times (15M vertices/min)
• Nested hierarchy
• Pre-filtered, tiled S3TC-Mipmaps
• High-quality Lanczos pre-filtering
• Reduces artifacts at tile borders
• S3TC virtually for free
• Precise Level of Detail Oracle
• Based on the Jacobian of the projection
• Does not make linear approximations
• Better anti-aliasing control
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Tile-based…
• Partition data into square tiles
– Can be pre-processed independently
– Frustum culling per tile
– Progressive data transfer per tile
• T-Vertices: Fix with zero-area triangles
– Only dynamic triangulation during run-time.
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• Tiled meshes are decimated in a pre-process
• Mesh decimation as in ROAM (Real-Time Optimally
Adapting Meshes - see www.vterrain.org)
– Exploits a triangle bintree structure: split along the base
edge
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• The recursive split operation
– A triangle should be split but has a base neighbor
from a coarser level
• Force split of base neighbor first
• Recursively force further splits if necessary until diamond
is found
• Diamond can be split without further splits
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• Recursive split operation
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• Vertices can be shared across levels
– „Coarse“ Vertices used in all finer levels
– Progressive encoding, less memory consumption
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• By generating approximations of the height field
with decreasingly lower approximation error, a
mesh hierarchy that represents the original terrain
at ever finer scales is constructed
• To generate a discrete set of hierarchy levels, an
error vector (e0; e1; … ;e(n-1)) of exponentially
decreasing entries ei := 2(n-1-i) is specified
• Hierarchy levels that satisfy these errors are
constructed via adaptive triangulation
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• Each hierarchy level can be render as „recursive triangle fans“
– Fans around „quadtree centers“
– Exploit primitive restarts on nVidia architectures
• A flag in the primitive stream that indicates a new fan
• Used to avoid frequent calls from application program
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Restricted Quadtree Hierarchy
• Nested Mesh Hierarchy:
„The parameter domain of each triangle is a subset of the
parameter domain of exactly one triangle at the coarser levels.“
• Quadtree refinement:
– Automatically results in nested mesh
– Each new triangle re-uses two old vertices
• Elegant geomorphing possible
– Requires only one height value per vertex & level
– Can be implemented in vertex shader
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Tiled, S3TC Mipmaps
• Tiled Mipmaps:
– Start with large 2D texture
– Generate large Clipmap using Lanczos filter
– Tile after filtering  artifacts at boundaries reduced
• S3 Texture Compression
–
–
–
–
–
–
Designed for Games  Supported by virtually all GPUs
Fast encoding in driver
Decoding virtually for free
Block Truncation Code, 4x4 pixels per block
Lossy compression: 6:1  Quality ?
We are researching other GPU-friendly methods…
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
S3TC Quality ?
Pros: „rough“ textures
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
S3TC Qualität
And Cons… - „smooth“ textures
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
• Vector quantization
Input mapping
Encoder
in=E(Xn)
Xn
Codebook C
with codewords
in
Decoder
Output mapping
X‘n=C(in)
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Vector Quantization
• Find centroids (representative vectors) to which all
elements are assigned
• Elements get assigned to „closest“ representative
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
Vector quantization: the LBG-Algorithm
1. Inital codebook {Yi(0)}i=1..M; IRn: set of input vectors;
2. Initialization k=0, D(0) =0, set 
3. Find quantization regions Vi (k) = {XI: d(X,Yi) < d(X,Yj) j≠i, j=1..M}
4. Compute distortion
M
D
(k )

 d ( X ,Y
i 1 X Vi( k )
(k )
i
)
5. Test: If (D(k)-D(k-1))/D(k) < ) stop else continue
6. Update: increment k, find new codebook {Yi(k)}i=1..M from regions
Vi (k-1); goto 3
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
True-Color (3*8 Bit) vs. 12 Bit Quantization + Codebook
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
8 Bit
6 Bit
4 Bit
2 Bit
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
Region quantization
2 bpp
2 bpp, 2x2 pixel encoding
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Confocal scan
2
1024 x64
32 bpp
RGBA
1 bpp
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
CT scan
3
256
Original
Hierarchical encoding 21:1
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Engine CT scan
Original
2
256 x128
Hierarchical encoding 21:1
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Visible Male RGB slice 2048x1216
Original 24 bpp
Hierarchical encoding 0.3 bpp
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Vortex simulation 100 x
Original, 200 MB
3
128
Hierarchical encoding 11 MB
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Shockwave simulation 96 x
Original, 1.4 GB
3
256
Hierarchical encoding 70 MB
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture compression
• Vector quantization for GPUs
– Easy to decode in fragment shader
• Only requires a dependent texture fetch (uses index to
lookup the entry in the codebook)
– Does not allow for texture filtering (so far)
• Interpolation between indices gives errorenous results
• Eventually larger regions can be decoded, rendered into
texture render target and then used to texture the object
• Functionality is available but mipmapping might be a
problem
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Scale of detail
• Question:
– Now that we have a geometry and a texture
MipMap, how is the resolution determined at
which a particular tile has to be rendered
• Guess:
– Depends on distance to viewer
– Depends on projection used for rendering
– Depends on orientation of tile
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Scale of detail
• Projection is non-linear (hom. division)
• Pixel coords s(v), where v vertex in object space
• Object space scale to pixel correspondence
– Similar to Mipmapping
– Requires partial derivatives
– Can be obtained by the inverse Jacobian of s(v):
• Jacobian s(v) = ∂si/∂vi
• Jacobian-1 s(v) = ∂vi/∂si
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
From scale to level of detail
• Pre-process: levels L with errors 2L meters
LOD =log2() for bounding box corners of tiles
• Tri-linear interpolation of  in box
– LOD per vertex for geomorphs
• Find minimum of  for each box
– Finest detail determines used topology
– Project vertices to appropriate level (geomorph)
– Requires pre-computed height for each vertex & level
• But only for coarser levels
• And can be done in reduced precision (i.e. 8bits)
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Level of detail: benefits
• Near-optimal geometry filtering
– Geometry properly anti-aliased
– Geometry properly reduced
– No over-estimation of error
• No linear approximations of the projection
• … but so far no anisotropic geometry filter
• Still works very well in practice
• LOD for Textures is easier:
– Mipmaps
– Anisotropic texture filtering
– All supported by GPU without major performance penalty
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture filtering - MipMap
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Texture filtering - anisotropic
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization
Demos
Puget Sound
Oberbayern und Alpen
Image Synthesis – WS 07/08
Dr. Jens Krüger – Computer Graphics and Visualization Group
computer graphics & visualization