advertisement

Super-Resolution Authors: Wagner, Waagen and Cassabaum Presented By: Mukul Apte Introduction Definition: Create High Resolution visual output from Low Resolution visual input. Mathematical assistance to features viz. motion detection, face recognition, person detection. Action Packed Sports Images Astronomy Medical Imaging Surveillance Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO) Basic idea Given: A set of degraded (warped, blurred, decimated, noised) images: Required: Fusion of the measurements into a higher resolution image: Types Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out Simple Example Concepts I: Raster, CCD, Image in mathematical domain, Matlab Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain Simple Example For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible. D D Simple Example Due to our limited camera resolution, we sample using an insufficient 2D grid 2D 2D Simple Example However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain 2D 2D Simple Example Similarly, by shifting down we get a third image 2D 2D Simple Example And finally, by shifting down and to the right we get the fourth image 2D 2D Simple Example - Conclusion It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed. This is SuperResolution in its simplest form Uncontrolled Displacements In the previous example we counted on exact movement of the camera by D in each direction. What if the camera displacement is uncontrolled? Uncontrolled Displacements It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square). Uncontrolled Rotation/Scale/Disp. In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object. What if the camera rotates? Gets closer to the object (zoom)? Uncontrolled Rotation/Scale/Disp. There is no sampling theorem covering this case Further Complications Sampling is not a point operation – there is a blur Motion may include perspective warp, local motion, etc. Samples may be noisy – any reconstruction process must take that into account. Static Super-Resolution Y1 YN Y2 Y3 Low Resolution Measurements Static Super-Resolution Algorithm X̂ f Y1, Y 2 , Y 3 , , Y N X̂ High Resolution Reconstructed Image t Dynamic Super-Resolution Low Resolution Measurements Yt t t X̂t f Yt , X̂t 1 High Resolution Reconstructed Images Dynamic Super-Resolution Algorithm t X̂t t Approach Image Registration Motion Estimation Projection onto High-Resolution Grid Non-Uniform Interpolation Frequency domain – alias correction Projection Registration Low-res Images Registration (sub-pixel grid) High Res Grid 1.1 Registration (angle) •Rotation Calculation •Correlate 1st LR image with all LR images at all angles OR •Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle Anglei = max index(correlation(I1(θ), Ii (θ))) LR image 1 LR image 2 i = 2,3,..,N (number of LR images) Energy at angle Ii(θ) Energy at angle I2(θ) 1.1 Registration (shift) Shift Calculated using Frequency Domain Method Fi (uT) = ej2πuΔsF1(uT) Δs [Δx Δy]T Δs = angle( Fi (uT) / F1 (uT) u [fx fy] ) 2πu • Used only 6% lower u (high freq could be aliased) • Used least square to calculate Δs 2.1 Frequency Domain Input: Down-sampled aliased images Goal I: Correct the low-freq aliased data Goal II: Predict the lost high freq values -π π Original High-Res -π π Down-sampled Aliased (fix it) Lost (find it) -π/2 π/2 Up-sampled π -π π Desired High-Res 2.2 Projection onto High-res grid Papoulis-Gerchberg Algorithm Projection onto convex sets Known pixel values Known Cut-off freq in the HR image Algorithm: I (known pixel positions) = Known Values I_fft = fft2(I) I_fft(higher Freq) = 0 I= ifft2 (I_fft) SSR – The Model Geometric Warp Blur HighResolution Image F1=I H1 Decimation Y1 D1 V1 X LowResolution Images Additive Noise FN HN YN DN VN N 1 Y k Dk H k Fk X V k , V k ~ N0, Wk k 1 Warp – Linear Operation Z X Per every point in X find a matching point in Z x1 xj x N 1 0 1 F[j,i]=1 0 z1 z j 1 z N Model Assumptions We assume that the images Yk and the operators Hk, Dk, Fk,& Wk are known to us, and we use them for the recovery of X. Yk – The measured images (noisy, blurry, down-sampled ..) Hk – The blur can be extracted from the camera characteristics Dk – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation Wk – The noise covariance can be extracted from the camera characteristics Special Condition - Noiseless Y1 D1H1F1 Y D H F 2 2 2 2 X Y D H F N N N N Clearly, this linear system of equations should have more equations than unknowns in order to make it possible to have a unique solution. SSR – Handling Problems Single image de-noising Y X V Single image restoration Y HX V Single image scaling Y DX V Motion compensation average Yk Fk X V T Using AX X ST WS X 1 X̂ H H S WS HT Y T T 1 X̂ DT D ST WS DT Y X̂ FkT Fk ST WS k 1 N N k k 1 1 X̂ I S WS Y T 1 N T F k Yk k 1 SSR Standard Equation N X̂ j1 X̂ j FkT HTk DTk Wk Y k Dk H k Fk X̂ j ST WS X̂ j k 1 Back projection Simulated error Weighted edges All the above operations can be interpreted as iterative operations performed on images. (Typically 15-20 iterations are performed.) Thumb Rule on Desired Resolution Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then – L N M Papoulis – Gerchberg Algorithm Initial Setup FFT (Initial image) Taj Mahal – Low-res image I Papoulis – Gerchberg Algorithm Known Pixel Values Image after 1st iteration Image at iteration 0 FFT I(high freq) =0 Papoulis – Gerchberg Algorithm Known Pixel Values Image at iteration 1 Image after 10 iterations FFT I(high freq) =0 Papoulis – Gerchberg Algorithm After 50 iterations Bilinear Interpolation Bicubic Interpolation SR Reconstructed image Results (Images - I) • Input: 4 snaps using a high-res digital camera • Cropped the same part of each image • SR algorithm compared with bicubic interpolation Original Low-res images (Courtesy: Patrick Vandewalle) Results (Images - I) Bicubic Interpolation Results (Images - I) Super-resolution Results (Images - II) Low-Res Image I Low-Res Image II • SR Algorithm didn’t work as expected !!! • Reason: • Motion was not restricted to shifts & rotation • Images had affine mapping • Rule I - Need Correct Registration Results (Frames - I) Original Bicubic SR Why didn’t SR work??? • Low-res images were created by forcing shifts at critical velocities • Rule II - If low-res frames are at critical velocities, can’t create good high-res frame Results (Frames - II) Original Bicubic SR Why did SR work so well??? • Low-res images were created by forcing shifts at non-critical velocities • Rule III If low-res images have all the info about high-res then HR image can be perfectly constructed Future Work • Superresolution with multiple motions between frames - create high res video • Predict the high-res frequency components using wavelet methods Predict Predict Predict Acknowledgements Prof John Apostolopoulos Prof Susie Wee Patrick Vandewalle THANK YOU!!! QUESTIONS? COMMENTS?