Mean-Field Theory and Its Applications In
Computer Vision3
1
Gaussian Pairwise Potential
Expensive message passing can be
performed by cross-bilateral filtering
Spatial
Range
2
Cross bilateral filter
BF [ I ]p 
1
Wp
 G || p  q ||
qS
s
G r | I p  I q | I q
p
p
q
output
output
input
input
reproduced
3
from [Durand 02]
Efficient Cross-Bilateral Filtering
• Based on permutohedral lattice (PLBF)2
• Embed the points on the permutohedral lattice
• Apply Gaussian Blurring
4
Efficient Cross-Bilateral Filtering
• Based on permutohedral lattice (PLBF)2
• Embed the points on the permutohedral lattice
• Apply Gaussian Blurring
• Based on the domain-transform (DTBF)3
• Project the point to lower dimension
• Perform filtering in the transformed domain
5
Efficient Cross-Bilateral Filtering
• Based on permutohedral lattice (PLBF)2
• Embed the points on the permutohedral lattice
• Apply Gaussian Blurring
• Based on the domain-transform (DTBF)3
• Project the point to lower dimension
• Perform filtering in the transformed domain
• Filtering in frequency domain
• Apply fast fourier transform
• convolution in (s) domain=multiplication in (f) domain
6
Barycentric Interpolation
7
Efficient Cross-Bilateral Filtering
8
Permutohedral Lattice based filtering
• For each pixel (x, y)
• Downsample all the points
(dependent on standard deviations)
  X   Y   I ( X ,Y )  

( x , y , z )     ,   , 

  s   s    r  
9
Embed to the permutohedral lattice
• Embed each downsampled points to the lattice
10
Embed to the permutohedral lattice
• Embed each downsampled points to the lattice
11
Embed to the permutohedral lattice
• Embed each downsampled points to the lattice
12
Embed to the permutohedral lattice
• Embed each downsampled points to the lattice
13
Gaussian blurring
• Apply Gaussian blurring along axes
14
Gaussian blurring
• Apply Gaussian blurring along axes
15
Gaussian blurring
• Apply Gaussian blurring along axes
16
Splatting
• Upsample the points
17
Splatting
• Upsample the points
18
PLBF
• Final upsampled points
19
Domain Transform Filtering
• Project points in low-dimension preserving the
distance in the high dimension
• Filtering performed in low-dimension space
• Projecting to the original space
20
Distance in high-dimension space
21
Filtering in high-dimension space
Inefficient
Spatial
Range
22
Projection in low-dimension space
• Project to low-dimension
• Maintain geodesic distance high-dimension space
23
Projection in low-dimension space
• Project to low-dimension
• Maintain geodesic distance high-dimension space
24
Projection in low-dimension space
• Project to low-dimension
• Maintain geodesic distance high-dimension space
25
Gaussian blurring in low-dimension
• Apply Gaussian blurring in low-dimension space
26
Project
• Project the blurred values in the original space
27
Project
• Project the blurred values in the original space
28
PLBF Vs DTBF
• Processing Time:
• Both linear in the number of pixels
• Filter parameter:
• PLBF runtime is inversely proportional to the
kernel size defined over space and range
• Use PLBF with the relatively large (~10) range
• Use DTBF with relatively smaller (~1-2) range
29
Filtering in frequency domain
30
Convergence
• Iteration vs. KL-divergence value
• In theory: (since parallel update) convergence is not guaranteed
• In practice: converges observe a convergence
31
MSRC-21 dataset
• 591 colour images, 320x213 size, 21 object classes
32
MSRC-21 dataset
• 591 colour images, 320x213 size, 21 object classes
Runtime
Unary
Classifiers
Standard ground truth
Accurate ground truth
Global
Average
Global
Average
84.0
76.6
83.2±1.5
80.6±2.3
Grid CRF
1 sec
84.6
77.2
84.8±1.5
82.4±1.8
Robust Pn
30 sec
84.9
77.5
86.5±1.0
83.1±1.5
86.0
78.3
88.2±0.7
84.7±0.7
Dense CRF 0.2 sec
33
PascalVOC-10 dataset
• 591 colour images, 320x213 size, 21 object classes
34
PascalVOC-10 dataset
• 591 colour images, 320x213 size, 21 object classes
Dense CRF
Runtime
Overall
Av. Recall
Av. I/U
0.67 sec
71.63
34.53
28.4
35
Long-range connections
• Accuracy on increasing the spatial and range standard
deviations
• On MSRC-21 spatial – 61 pixels, range – 11
36
Long-range connections
• On increasing the spatial and range standard deviations
• On MSRC-21 spatial – 61 pixels, range – 11
37
Long-range connections
• Sometimes propagates misleading information
38
Mean-field Vs. Graph-cuts
• Measure I/U score on PascalVOC-10 segmentation
• Increase standard deviation for mean-field
• Increase window size for graph-cuts method
• Both achieve almost similar accuracy
39
Mean-field Vs. Graph-cuts
• Measure I/U score on PascalVOC-10 segmentation
• Increase standard deviation for mean-field
• Increase window size for graph-cuts method
•Time complexity very high, making infeasible to work with large neighbourhood system
40