Outdoor Image Processing 1 Photometric stereo for outdoor webcams "Photometric stereo for outdoor webcams" Ackermann, J.; Langguth, F.; Fuhrmann, S.; Goesele, M.; , CVPR 2012 Overview: Photometric stereo from time lapse video captured over a long time span. Retrieves Surface Normals Basic Materials Material Mixtures Indirect light 2 Assumptions GPS location of the camera, object and sky mask, per image time stamp are available 3 4 Selecting Subsets of images Image Filtering: 1) Discard images with 10% of the image or the object is overexposed 2) Select only daytime images, zenith < 85 degrees 3) Discard bad weather images – select only top 50% according to score: SI= Isky + IObj Isky = median of sky pixel intensities IObj = 75th percentile of object pixel intensities 5 Selecting Subsets of images Two Image subsets required; 1) Clear sky images for camera calibration, 2) Images with good weathers and well illuminated object for photometric stereo Iteratively select required number of images by updating penalty using a 2D Gaussian function and selecting the best image at that iteration 6 Obtaining light direction Image Alignment: Align gradient images to the average gradient Camera Calibration: 1) Radiometric response obtained using Kim et al. ( uses pixels under the same lighting conditions to solve for the response function). 2) Absolute zenith, azimuth of the Sun obtained using cam location, timestamp. 3) Use the sky as calibration target ( Lalonde et al.) to find camera zenith, azimuth Shadow Detection Imax,p / Imin,p < 1.4 => always shadowed Otherwise, Ii,p < 1.5*median10% darkest pixels => shadowed in Ii End of Stage 1 ( obtain subset of images with light direction) 7 Photometric Stereo stage Intensity of image i at pixel p and channel c, Ii,p,c = Isun,i,p,c + Isky,I,p,c Reflectance at a pixel is linear combination of basis materials, fm,c Sun light model Intensity of the sun Surface normal Material mixing coeff Portion of sky visible at p Sun direction 8 Light Model Sky light model Assume Finally, Optimize for li,c, fm,c, np, γp,m . Vp is replaced by using images Ip s.t. pixel p is not in shadow 9 Initialization Set Sp,c to zero , assume constant light intensities and Lambertian scene 1. Obtain initial estimates for surface normals and albedo 2. Use these to find initial estimates of the light intensities 3. Cluster the albedos to get an initialization of material properties at each pixel 1) Surface normal and albedo Solve for classical photometric stereo 10 Initialization 2) Relative light intensities: Need six surface points with similar albedo and differing normal 1) Cluster albedos in 4 groups 2) Cluster normals for pixels with the most frequent albedo 3) Pick normal from different clusters 3) Initial material estimation: Cluster albedos in sRGB Identify pure pixel sets for each of the fundamental materials Solve for the BRDF parameters 11 Iterative Refinement Intensity estimate: updated in each following step 1) Material Fitting Find optimal parameters for all materials simultaneously, not for only pure pixels Minimize 12 Iterative Refinement Light intensity optimization: Minimize , Material and normal map optimization: Minimize Material parameters, light intensities are fixed, only normals, sky light at each pixel, and material mixing coeff are optimized 13 Shadow Detection and Removal Single-Image Shadow Detection and Removal using Paired Regions Ruiqi Guo, Qieyun Dai, Derek Hoiem. CVPR 2012 Employs a region based approach. Perform pairwise classification (of illumination conditions) of regions based on appearance. Graph cut is used for the labeling. Soft matting for refinement Shadow free image is obtained by relighting pixels under shadow 14 Shadow Detection Maximize cishadow – single region classifier confidence * region area cijdiff , cijdiff – pairwise classifier confidence * f(region areas) y – shadow labels for regions 15 Shadow Detection Single region classifier (with χ2 kernel) features 1. Color histograms 2. Texton histograms Internal appearance of a given region is not enough Comparison between regions of same material needed Pairwise region classifier (RBF kernel) features 1. Χ2 distance between color and texton histograms 2. Ratios of RGB average intensity ( ρr = Ravg1 /Ravg2, …) 3. Chromatic alignment (ρr/ρg) 4. Normalized distances between the regions 16 Pairwise region graph Different illumination black-white Same-illumination Green Not related Orange 17 Pairwise region graph 18 Apply Graph cut Reformulate the cost function to apply Graph cut 19 Shadow Removal Hard shadow mask Soft shadow matt Soft shadow Solution: Shadow matting 20 Slide from Guo et al. Shadow Removal Simple light model: direct light + env. Light Relighting: Estimate how much direct light is occluded at each pixel and light up by that amount 1. Find the fractional shadow coefficients using matting technique 2. Find ratio of direct to environment light. Direct light Surface Reflectance Shadow coefficient Environmental light 21 content from from Guo et al. Light Model Non-shadow: ti=1 Umbra: ti=0 Penumbra = 0 < ti < 1 Ld + Le Ld kLd + Le Le Relighting : Iishadow-free = (Ldcosθi + Le)Ri 22 Figure from from Guo et al. Shadow model as a matting problem Ii = γiFi + (1- γi)Bi F: Foreground image, B: background Image Rewrite shadow model as: Ii = ki(LdRi + LeRi) + (1-ki)LeRi Similar to matting eqn. Solve for matting: minimize E(k) = kTL k + λ(k-k’)TD(k-k’)T optimal k obtained by solving sparse system: (L + λD)k = λdk’ 23 Finding the light ratio Final r obtained by voting 24 content from from Guo et al. Results UCF Dataset (Zhu et al.) 245 images outdoor scenes, manual annotations 25 content from from Guo et al. Experiments: Datasets UCF Dataset (Zhu et al.) 245 images outdoor scenes, manual annotations content from from Guo et al. Experiments: Datasets New UIUC Shadow Dataset 108 images, indoor/outdoor, automatic annotation Evaluate both shadow detection and removal Input image Groundtruth Nonshadow Shadow mask content from from Guo et al. Results on UCF Dataset Input image Groundtruth Shadow mask Detection Removal result content from from Guo et al. Results on UIUC Dataset Input image Detection Removal result Groundtruth content from from Guo et al. Results: Shadow Detection Pixel accuracy Accuracy UCF dataset UIUC dataset (ours) Full model Single region Zhu et al. 0.900 0.875 0.887 0.883 0.796 - content from from Guo et al. Results: Shadow Detection Confusion matrices on UCF dataset full model Single region classification Shadow Non-shadow Shadow (GT) 0.750 0.250 Nonshadow(GT) 0.070 0.930 Shadow Non-shadow Shadow (GT) 0.515 0.485 Nonshadow(GT) 0.057 0.947 content from from Guo et al. Results: Shadow Detection Confusion matrices on UCF dataset full model Shadow Zhu et al. 2010 Shadow Non-shadow Non-shadow Shadow (GT) 0.750 0.250 Shadow (GT) 0.639 0.361 Nonshadow(GT) 0.070 0.930 Nonshadow(GT) 0.067 0.934 content from from Guo et al. Failure Example Input image Detection Removal result content from from Guo et al. End 34