CS-MUVI Aswin Sankaranarayanan, Lina Xu, Christoph Studer, Yun Li, Kevin Kelly, and Richard G. Baraniuk Single pixel camera Photo-detector Digital micro-mirror device Single pixel camera • Each configuration of micro-mirrors yield ONE compressive measurement • Non-visible wavelengths • Sensor material costly in IR/UV bands • Light throughput Photo-detector Digital micro-mirror device • Half the light in the scene is directed to the photo-detector • Much higher SNR as compared to traditional sensors Single pixel camera • Each configuration of micro-mirrors yield ONE compressive measurement Photo-detector • static scene assumption • Key question: Can we ignore motion in the scene ? Digital micro-mirror device SPC on a time-varying scene time varying scene measurements t=1 t=W compressive recovery • Naïve approach: Collect W measurements together to compute an estimate of an image what happens ? SPC on a time-varying scene t=1 t=W (small) Small W Less motion blur Lower spatial resolution t=W (large) Large W Higher spatial resolution More motion blur Tradeoff Temporal resolution vs. spatial resolution SPC on a time-varying scene sweet spot Lower spatial res. Higher temporal res. Higher spatial res. Lower temporal res. fast slow static scene motion speed SPC Video Uncertainty Principle N2 (N/2)2 (N/4)2 number of measurements per frame W Dealing with Motion • Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression Dealing with Motion • Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression naïve reconstruction motion estimates Key points • Motion blur and the failure of the sparsity assumption – Use least squares recovery ? • Recover scene at lower spatial resolution – Lower dimensional problem requires lesser number of measurements – Tradeoff spatial resolution for temporal resolution • Least squares and random matrices – Random matrices are ill-conditioned – Noise amplification • Hadamard matrices – Orthogonal (no noise amplification) – Maximum light throughput – Optimal for least squares recovery [Harwit and Sloane, 1979] Hadamard + least sq. recovery Hadamard Random slow static relative speed fast SPC Video Uncertainty Principle (N/2)2 (N/4)2 (N/2)2 (N/4)2 window length (b) random+LS (c) DSS+LS Hadamard + least sq. recovery Designing measurement matrices • Hadamard matrices – Higher temporal resolution – Poor spatial resolution • Random matrices – Guarantees successful l1 recovery – Full spatial resolution • Can we simultaneously have both properties in the same measurement matrix ? Dual-scale sensing (DSS) matrices Key Idea: Constructing high-resolution measurement matrices that have good properties when downsampled 1. Start with a row of the Hadamard matrix 2. Upsample 3. Add high-freq. component CS-MUVI: Algorithm outline 1. obtain measurements with DSS matrices 2. lowresolution initial estimate 3. motion estimation 4. compressive recovery with motion constraints t=1 t=W t=t0 t=t0+W t=T Simulation result CS-MUVI on SPC Single pixel camera setup InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M Object CS-MUVI: IR spectrum initial estimate Upsampled Recovered Video Joint work with Xu and Kelly CS-MUVI on SPC Naïve frame-to-frame recovery CS-MUVI Joint work with Xu and Kelly Real data • Number of compressive measurements: 65536 • Total duration of data acquisition: 6 seconds • Reconstructed video resolution: 128x128x256 Preview (6 different videos) *animated gifs* CS-MUVI (6 different videos) *animated gifs* Comparisons Frame-to-frame wavelets; no optical flow 3D total variation; no optical flow CS-MUVI (6 different videos) *animated gifs* Real data • Number of compressive measurements: 65536 • Total duration of data acquisition: 6 seconds • Reconstructed video resolution: 128x128x256 Preview (6 different videos) *animated gifs* CS-MUVI (6 different videos) *animated gifs* CS-MUVI summary • Key ingredients – Novel Measurement matrix design – Exploiting state-of-the-art motion model • One of first practical video recovery algorithm for the SMC dsp.rice.edu CS-MUVI summary • Limitations – Need a priori knowledge of object speed – Motion at low-resolution – Robustness to errors in motion estimates • Future work – Dual-scale to multi-scale matrix constructions – Multi-frame optical flow – Online recovery algorithms dsp.rice.edu