Photorealistic Rendering of Rain Streaks
Kshitiz Garg
Shree K. Nayar
Department of Computer Science
Columbia University
SIGGRAPH Conference
July 2006, Boston, USA
Sponsors: NSF, ONR
A Rain Scene from “The Matrix Revolutions”
A Rain Scene from “The Matrix Revolutions”
Frame 1
Frame 2
Frame 3
Rendering Streaks using Simple Brightness Model
Commercial Software
Heuristic-based [Starik and Werman 03]
Hybrid-Spectral Method[Langer et al. 04]
ATI ToyShop Demo [N. Tataturschuk et al. 05]
Rendering Streaks using Hand Drawn Textures
Frame from “The Matrix Revolutions”
Original Image Courtesy Warner Bros.
Hand Drawn Textures
Original Image Courtesy Andy Lomas
Study of Visual Appearance of Rain Streaks
• Rain Streak Appearance Model
• Rain Streak Texture Database
• Image-Based Rain Rendering Algorithm
Oscillations in a Rain Drop
[Beard and Kubesh 1989
Andsager et al. 1992]
Turbulent
Flow of air
Turbulent
Flow of air
A Falling Rain Drop
Oscillations in a Rain Drop
[Beard and Kubesh 1989
Andsager et al. 1992]
Oblate-Prolate Mode
Transverse Mode
Oscillations in a Rain Drop
Appearance of a Rain Drop
Specular Reflection
Le (sˆ)
S
n̂
L(nˆ )
N
R
Le (rˆ)
Refraction
Le ( pˆ )
P
Internal Reflection
L(nˆ )  R Le (rˆ)  S Le (sˆ)  P Le ( pˆ )
Appearance of a Non-Oscillating Rain Drop
S
n̂
L(nˆ )
N
R
P
Non-Oscillating Rain Drop
Instant
Time Shots
Falling
Drop
Rain
Streak
Appearance of an Oscillating Rain Drop
S
n̂
L(nˆ )
S
N
R
P
R
P
Oscillating Rain Drop
Instant
Falling
Time Shots Drop
Rain
Streak
Rain Streak Appearance Model
[Beard and Kubesh 1989
Andsager et al. 1992]
Shape of Rain Drop:
?
?
 A2, 0
Fall Direction
fdrop= (00 – 3600)
Not Rotationally
Symmetric
 A3,1
Oblate-Prolate
Mode
Transverse
Mode
Measurement of Rain Streak Appearances
Experimental Setup
Drop
generator
Height: 15 m
y,ydrop
Lighting and Viewing
Configuration
qlight = (500, 900, 1300)
flight = (100 , 300,…, 1700)
qview = (700, 900,
1100)
Oscillation Parameters
qvie
w
x
fdrop
xdrop
qlig
ht
flig
ht
-z
(A2,0 A3,1 fdrop) =(10 images per
light & view)
Estimating the Oscillation Amplitudes
qvie
w
qlig
ht
flig
110
130
130
90
90
10
70
50
30
10
130
150
30
10
70
50
90
110
50
170
130
30
170
50
90
90
110
50
Measured
ht
Estimation Result
 A2,0
 A3,1
Oscillation Amplitudes: ( A2, 0 , A3,1 )  (0.2,0.1) ; (0.1,0.1)
130
30
Comparison of Actual vs. Rendered Rain Streaks
qvie
w
qlig
ht
flig
Rendered
Measured
ht
110
130
130
90
90
10
70
50
30
10
150
130
30
10
70
50
90
110
50
170
130
30
170
50
90
90
110
50
130
30
Precomputed Database of Streak Textures
Factors Affecting the Appearance
Render Offline (Database)
Lighting and Viewing Direction
Oscillations Parameters
Rain Drop Size
Render Online (Algorithm)
Source and Camera Distance
Camera Parameter
Database contains 6300 HDR Rain Streak Textures (400 MB)
( Publicly available: Please send email to rainstreaks@cs.columbia.edu )
Rain Rendering Pipeline
User Input
Rain Dynamics
Render Novel
Streak Textures
Streak texture
(Source 1)
Camera and
Rain
Parameters
Light Source
Properties
Coarse
Depth Map
Perspective
Effects
Scale
Rotate
Raindrop
Distribution
and Trajectories
Using
Particle System
+
Streak texture
(Source n)
+
Streak texture
(Ambient)
Scene
Image/Video
Render Streak
Appearance
Rain Streak Database
Camera
Effects
Exposure
Crop/
Blend
Defocus
Blur
Scene
Image/Video
With Rain
Rendered Rain with Varying Light Direction
-150o
150o
Light Source Direction
Constant Brightness Streak Model
Comparison with Constant Brightness Model
Streak Appearance Model
Constant Brightness Model
o
o
-150
150
Light Source Direction
Night Scene with Isotropic Source
Coarse Depth Map
Image of the Scene
Night Scene with Isotropic Source
Rendered Rain
Night Scene with Isotropic Source
Defocus Effects
Scene with Colored and Anisotropic Sources
Coarse Depth Map
Scene with Multiple Sources
Scene with Colored and Anisotropic Sources
Rendered Rain
Scene with Colored and Anisotropic Sources
Streak Appearance Model
Constant Brightness Model
Scene with Changing Sky Illumination
Input Scene Video
Scene with Changing Sky Illumination
Rendered Rain
A Scene with Moving Light Sources
Input Scene Video
A Scene with Moving Light Sources
Rendered Video
Future Extensions
• Real-time Implementation of Rendering Algorithm
• Incorporating the Effects of Wind and Turbulence
Summary
Lighting Angle
 A3,1
Rendered
Rain Streaks
 A2,0
Real Images of
Rain Streaks
Viewing Angle
Rain Streak Appearance Model
Rain Streak Texture Database
( Publicly available: rainstreaks@cs.columbia.edu )
Image-Based Rain Rendering Algorithm