Digital Media
Dr. Jim Rowan
ITEC 2110
Video Part 2
Coping with Video Size
Consider human vision limitations
1) Chrominance sub-sampling
Compression - two versions
2) Spatial
3) Temporal
differencing
vectoring
Coping with Video Size
Consider human vision limitations
1) Chrominance sub-sampling
Compression - two versions
3) spatial
4) Temporal
differencing
vectoring
Chrominance sub-sampling
• Humans can’t distinguish changes in color as
well as they can distinguish luminance
changes
– http://en.wikipedia.org/wiki/Chroma_subsampling
• In our cameras…
– Of every 4 frames
– store the luminance for each frame
– only store a proportion of the color info
– 4:2:0
Chrominance sub-sampling
http://dougkerr.net/pumpkin/articles/Subsampling.pdf
Chrominance sub-sampling
Luminance, Cr, Cb
http://en.wikipedia.org/wiki/Chroma_subsampling#Sampling_systems_and_ratios
Coping with Video Size
• Aside from screen size and frame rate...
• Consider human vision limitations
1) Chrominance sub-sampling
• Compression - two versions
2) spatial
3) Temporal
differencing
vectoring
Coping with Video Size
• Spatial compression
• Individual images can be compressed using the
techniques discussed in the bitmapped section
• Doesn’t result in very much compression for
video
• Doesn’t take into consideration the other
frames that come before or after it
• Our video cameras do this, compressing each
frame to jpeg
Coping with Video Size
• Aside from screen size and frame rate...
• Consider human vision limitations
1) Chrominance sub-sampling
• Compression - two versions
2) spatial
3) Temporal
differencing
vectoring
Temporal Compression
differencing
• Use the Difference in two frames
– A naive approach can result in good compression
– Works well for a small amount of movement
• Security cameras spend most of their time “seeing” the
same thing all night long
– A Tarantino film? not so much…
• Most pixels change with nearly every frame
Captured
Frame 1
Captured
Frame 2
Saved
Frame 1
Saved
Frame 2
Image Differencing
• To subtract one image from the next
• Do it one pixel at a time
– red minus red
– green minus green
– blue minus blue
• Store the difference
• To play it back
– Play frame one 1
– Add frame 2 to frame 1
• Next, an example in black & white
Example 1, the difference of two identical images
Example 1, the
difference of two
identical images
The result ===>
Example 2, the difference of two similar images
Example 2, the
difference of two
similar images
The result ===>
Temporal Compression
vectoring
• When an OBJECT moves
– compute its trajectory
– fill in the resulting exposed background
Captured
Frame 1
Captured
Frame 2
Stored
Background
– BUT there’s a problem...
– why isn’t this an easy thing to do?
Stored
Object
Movement
Vector
More on differencing
• The differencing can happen in a
forward manner and a backward
manner
• It might be more economical (in data
size) to create a frame from a frame that
follows it...
MPEG-2
iFrame
pFrame
bFrame
GOP
http://en.wikipedia.org/wiki/Inter_frame
http://en.wikipedia.org/wiki/Iframes#Intra_coded_frames_.28or_sli
ces_or_I-frames_or_Key_frames.29
iFrame:
-a keyframe
-spatially compressed
(a fully specified image)
pFrame
-predicted frame
-contains only the difference
between the current frame and
the previous iFrame
(smaller in size than iFrame)
bFrame
-bi-predicted frame
-contains difference between
current and both the preceding
and following iFrames
-even smaller in size than iFrame
3 mpeg 2 video streams
http://www.spiritalchemy.com/blog/wp-content/uploads/2009/10/IPB_frame_example.jpg
Group(s) Of Pictures (GOPs)
All spatially (intra-frame)
compressed
The largest
Spatially compressed
and predictive
(difference)
Smaller…
but more
computation
Spatially compressed
forward and
backward predictive
(difference)
Play sequence: 1 2 3 4 5 6 7 8
9 10
Transmit sequence: 1 4 2 3 7 5 6 10 8 9
Smallest…
but more
computation
and it is
transported
out of order!
Mpeg compression technique 1
Mpeg compression technique 2
Mpeg compression technique 3
Frame capture sequence: 1 2 3 4 5 6 7 8
9 10
So…
How does this ===>
happen?
Mpeg encoding
http://www.100fps.com/codec_quality_comparison.htm
• It’s COMPLICATED…
• Images are broken up and transmitted as
macroblocks
• They can be a variety of sizes
– Typically 8X8 or 16X16
• When missing (or there are video errors)
– May use a lower resolution block (that was
previously received)
– May just use a solid color block instead
Video Compression
What does this?
http://en.wikipedia.org/wiki/Video_codec
• Coder/Decoder - Codec
– encodes and decodes video
• Can be symmetric
• it takes as long to compress as decompress
• Can be asymmetric
• it takes longer to compress or decompress than it does to
decompress to compress
Video Compression
Can be very complex
Each image is just jpeg compressed
Each image is jpeg compressed; the p-frames
are the result of frames being subtracted from
the i-frames
Each image is jpeg compressed; the p-frames
are the result of frames being subtracted from
the i-frames; b-frames are the result of
frames being subtracted from i-frames and
future p-frames
A final worry...
• Codecs compress video to make it
smaller
• There are a variety of codecs to do this
• Which to choose?
– It is a tradeoff between compression
technique, its computational complexity
and its artifacts
So... How do codecs vary?
• compression and decompression complexity
– affects the artifacts that are created
– affects the time required to carry them out
– affects the volume of the data stream created
– affects the type and expense of the equipment used
– affects whether or not it can be implemented in
hardware of software
• WMV, DivX, Cinepak, Intel Indeo & Sorenson
Which is the original?
Notice the artifacts?
http://www.100fps.com/codec_quality_comparison.htm
Center one is the original
Left is “sharpened” Right is “blurred”
http://www.100fps.com/codec_quality_comparison.htm
That’s It!