Eulerian Video Magnification
Baby Monitor
Nik Cimino
Eulerian Video Magnification
Wu, Hao-Yu, et al. "Eulerian video magnification for revealing subtle changes in the world." ACM Trans. Graph. 31.4 (2012): 65.
•
•
•
•
Spatially Decompose (e.g. Laplacian Pyramid)
Temporally Filter
Amplify
Reconstruct
Considerations
• Used 30FPS camera, 15Hz is the fastest detectable
frequency (Nyquist theorem), yields 33ms/frame:
1
1𝑠
0.03333𝑠 33.33𝑚𝑠
𝑇𝑓𝑟𝑎𝑚𝑒 =
=
=
=
30𝑓𝑝𝑠 30𝑓𝑟𝑎𝑚𝑒𝑠
1𝑓𝑟𝑎𝑚𝑒
1𝑓𝑟𝑎𝑚𝑒
• Higher resolutions yield better signal detection
• Use good sensor/good lighting, high ISO = high noise
• Minimize camera and subject motion
• Compression can cause problems, inter-frame
redundancy and loss of low-level details
Goal
• Detect babies heart rate from spatial
magnification
• Detect babies respiration rate from movement
magnification
Videos Tested and Referenced
Side Video
Front Video
Swing Video
Area of Interest Selection
• Manual for now
• Future:
– Face recognition
– Skin tone detection
– Area of motion
could possibly be
revealed by
average image
minus current
Side Video
Algorithms Tested
• Gaussian blur and down sample
– Ideal bandpass temporal filter
• Laplacian Pyramid
– Ideal bandpass temporal filter
– Butterworth (subtraction of two lowpass filters)
temporal filter
– IIR (subtraction of two lowpass filters) temporal
filter
y1[n] = r1*x[n] + (1-r1)*y1[n-1]
y2[n] = r2*x[n] + (1-r2)*y2[n-1]
(r1 > r2)
y[n] = y1[n] - y2[n]
Algorithm Parameters
• Alpha - Amplification Factor (applies to pyramid
levels 1, 2, 3) [verify]
• Levels - Number of levels to use in the pyramids
• Frequency Low - Low end of our band pass
• Frequency High - High end of our band pass
• Frames per Second - FPS of the film, needed to
accurately perform temporal filtering
• Chrome Attenuation - Similar to alpha but only
applied to pyramid levels 2 and 3.
Example Post-Processed
Front Video
Single Pixel Slice of Time
Front Video
Spatiotemporal Profile
• Using that single pixel slice a spatiotemporal
profile is created
• Visualize spatial amplification
Front Video : Face
BPM Extraction Attempt 1
• Using Otsu’s threshold method
• Did not work as desired
Front Video : Face
BPM Extraction Attempt 1 (con’t)
• Again with Otsu’s threshold method
Side Video : Face
BPM Extraction Attempt 1 (con’t)
• Manual Threshold
• This is better, but still not reliable
Side Video : Face
BPM Extraction Attempt 2
• Fourier Analysis
• Proved to be robust for dominant frequency
detection
• Clearly shows dominant bands
Side Video : Face
Single Pixel Intensity versus Frames
195
190
185
Intensity
0-255
180
175
170
165
0
500
1000
Frames 0-~1500 (@ 33.33ms/frame)
Swing Video : Face
1500
All Pixel Frequency Response
7
x 10
-3
Frequency Response
6
5
4
3
2
1
0
-15
-10
-5
0
Frequency (Hz)
Swing Video : Face
5
10
15
Average Frequency Response
6
x 10
-3
Frequency Response
5
4
3
2
1
0
-1
-15
-10
-5
0
Frequency (Hz)
Swing Video : Face
5
10
15
Average Frequency Response of Interest
• 0.666 Hz (40 BPM) to 3 Hz (180 BPM)
Frequency Response
14
12
10
8
6
4
2
0
0.5
1
1.5
2
Frequency (Hz)
Swing Video : Face
2.5
3
Movement Recovery
• Notice that very soon after movement the
spatiotemporal profile shows recovery
Swing Video : Face
Average Frequency Response
7
x 10
-3
Frequency Response
12
6
x 10
-3
Frequency Response
10
5
8
4
6
3
4
2
2
1
0
0
-1
-15
-10
-5
0
Frequency (Hz)
Side Video : Face
5
10
15
-2
-15
-10
-5
0
Frequency (Hz)
5
Front Video : Face
10
15
Average Frequency Response of Interest
Frequency Response of Interest
Frequency Response of Interest
14
7
12
6
10
5
8
4
6
3
4
2
2
1
0
0.5
1
1.5
2
Frequency (Hz)
Side Video : Face
2.5
3
0
0.5
1
1.5
2
Frequency (Hz)
Front Video : Face
2.5
3
Robust Spatial Detection
• Side video has a very small amount of pixels
showing face, but still detected accurately
• Front video is dark and has a fair amount of
noise, but the frequency response is still good
• Swing video had movement and yet recovered
very well
Spatiotemporal Profiles
Front Video : Chest
Swing Video : Chest
Side Video : Chest
Frequency Responses
5
x 10
-3
Frequency Response
Frequency Response
0.012
4
0.01
3
0.008
2
0.006
1
7
x 10
0.004
Frequency Response
-3
0
0.002
6
-1
-15
-10
-5
0
5
Frequency (Hz)
5
10
15
0
-15
-10
-5
0
Frequency (Hz)
5
10
4
Swing Video : Chest
Front Video : Chest
3
2
1
0
-15
Side Video : Chest
-10
-5
0
Frequency (Hz)
5
10
15
15
Frequency Responses
Frequency Response of Interest
Frequency Response of Interest
90
30
80
25
70
60
20
50
15
40
30
10
Frequency Response of Interest
150
20
5
10
0
0.1
0.2
0.3
0.4
0.5
0.6
100
Frequency
(Hz)
0.7
0.8
0.9
0
0.1
0.2
0.3
Swing Video : Chest
0.4
0.5
0.6
Frequency (Hz)
0.7
0.8
Front Video : Chest
50
0.2 Hz (12 BPM) to
0.8 Hz (50 BPM)
Side Video : Chest
0
0.1
0.2
0.3
0.4
0.5
0.6
Frequency (Hz)
0.7
0.8
0.9
0.9
Poor Motion Magnification
• There isn’t a clean band where the respiratory
rate should’ve been
• The spatiotemporal profiles have very flat
areas
• Calculated dominant frequencies aren’t too
far off, but the numbers have no confidence
associated with them
Face Output
• Front : Face - Detected Dominant Frequency at (Hz):
2.5922
Dominant BPM:
156 < tested 151 beats per minute
• Side : Face - Detected Dominant Frequency at (Hz):
2.0297
Dominant BPM:
122 < tested 130 beats per minute
• Swing : Face - Detected Dominant Frequency at (Hz):
2.0062
Dominant BPM (possible interference resonating at 2Hz):
120 < tested 144 beats per minute
Chest Output
• Swing : Chest - Detected Dominant Frequency at (Hz):
0.6532
Dominant BPM:
39 < tested 38 respirations per minute
• Side : Chest - Detected Dominant Frequency at (Hz):
0.8086
Dominant BPM:
49 < tested 29 respirations per minute
• Detected Dominant Frequency at (Hz):
0.7864
Dominant BPM:
47 < tested 33 respirations per minute
Conclusions
• Eulerian video magnification is surprisingly
robust for spatial magnification
• Movement magnification works, but is not as
reliable
• Laplacian pyramid magnification worked best
for motion magnification
• Gaussian blur and down sample worked best
for spatial amplification
Future Work
• Add area of interest selection for video
• Change BPM detection to be on a small time
interval
• Change dominant BPM detection to be a
dominant band detection
• Stitch both chest and face into same video with
numerical BPM for heart and respiratory rates
• Try phase based approach for movement
magnification
Questions?
Face – Heart Rate
Front : http://youtu.be/F9YzPnMd7J0
Side : http://youtu.be/bpivOKZcONI
Swing : http://youtu.be/FM-ga4zBv_0 @ 53 seconds for noise
Chest – Respiration Rate
Front : http://youtu.be/Otd42ZqqSRU
Side : http://youtu.be/vXdUPsx6ANg
Swing : http://youtu.be/28T5VssqeE0 @ 53 seconds for noise
Phase-Based Magnification
• Uses phase in Fourier domain (translation)
Wadhwa, Neal, et al. "Phase-based video motion processing." ACM Transactions on Graphics (TOG) 32.4 (2013): 80.