Video As A Datatype
Ketan Mayer-Patel
Comp 294-9 :: Fall 2003
The mothers of all video...
•
•
•
•
NTSC - American standard
PAL - European standard
SECAM - French standard (ugly stepchild)
For better or worse, this is where it all starts.
Comp 294-9 :: Fall 2003
Color Spaces
• One luminance component
• Two chrominance components
– Original TV was black and white.
– Adding color had to be done in a compatible way.
• NTSC: YIQ
• PAL: YUV
• In general: YUV and YCrCb used as terms.
Comp 294-9 :: Fall 2003
YUV vs. RGB
• Simple linear transform from one cube to
the other.
• Specific transform coefficients vary for
specific systems, but a common one:
– Y = 0.299 R + 0.587 G + 0.114 B
– Cr = -0.147 R - 0.289 G + 0.437 B + 0.5
– Cb = 0.615 R - 0.515 G - 0.100 B + 0.5
Comp 294-9 :: Fall 2003
YUV Challenges
• YUV is like taking RGB cube, standing it
on a corner with (0,0,0) on bottom and
(1,1,1) at top, rotating slightly, and taking
the bounding box.
• What problems might this incur?
– Some RGB colors are illegal.
– A lot of YUV colors are illegal.
Comp 294-9 :: Fall 2003
A Note On Color
• Previous view of color is EXTREMELY
simplified.
– Color is complicated
• Frequency dependent response for contrast,
lightness, etc.
– Gamma correction.`
Comp 294-9 :: Fall 2003
Scanning
• Analog video signal is continuous voltage
signal that gets scanned along the screen.
• The electron stream controlled by two
orthogonal sets of magnets.
– Horizontal: Beam is moved from left to right
and then quickly back.
– Vertically: Beam is moved from top to bottom
and then quickly back.
Comp 294-9 :: Fall 2003
Voltage
Scanning illustrated
Time
Comp 294-9 :: Fall 2003
Structure of Video
• Blinn’s article.
– Macrostructure
• Frequency peaks at frame rate.
– Microstructure
• Frequency peaks at line rate
• Adding color is a good first example of sourceaware channel coding.
–
–
–
–
What was the problem?
What was the solution?
Why does it work?
What are its drawbacks (i.e., when does it fail)?
Comp 294-9 :: Fall 2003
Interlacing
• Progressive = every scan line done in turn.
• Interlaced = every other scan line done.
– Creates two fields: odd field and even field.
– NTSC: 262.5 lines per field at 60 fields per sec.
– PAL: 312.5 lines per field at 50 fields per sec.
• Fields are separated in time.
Comp 294-9 :: Fall 2003
Analog Bandwidths
• True meaning of the bandwidth.
• Y, U, and V signals are all continuous along
a scan line.
– A bit of a hybrid signal: discrete vertically,
continuous horizontally.
• NTSC:
– Y is 4.2 MHz wide, I is 2 MHz, Q is 1 MHz
• PAL:
– Y is 6 MHz wide, U is 3 MHz, V is 3 MHz
Comp 294-9 :: Fall 2003
Corresponding Data Rates
• How much data can you put in 6 MHz band?
– Depends on S/N ratio.
– Depends on modulation scheme.
• Typical: 27 - 36 Mbs
• How many cable channels do you get?
– If 50, then 1.3 - 1.8 Gbs coming into your home.
• The real question for multimedia is:
– Why haven’t we found a better use for 1.3 Gbs
than continuous broadcast of Real World.
Comp 294-9 :: Fall 2003
Digital Video Frames
•
•
•
•
Almost always progressive
3 planes of pixel values (Y, U, and V)
Pixel depth
Geometry of each plane: width x height
– Chrominance is generally subsampled.
• How the planes relate to each other.
• Frame rate.
Comp 294-9 :: Fall 2003
CCIR-601
• Standard established for digitizing NTSC
and PAL signals.
Digital
Component
Analog
Counterpart
CCIR-601
Recommendation
Height
Lines
Width
Continuous
voltage changes
along scan line.
NTSC: 480
PAL: 576
Sample at 13.5 MHz
Results in 702 pixels
Recommended 720
Pixel depth
Voltage range.
Gamma-corrected
8-bit sample.
Chrominance
subsampling
Bandwidth
differences
4:2:2, 4:2:0, 4:1:1
Comp 294-9 :: Fall 2003
Why 8-bits for chrominance?
• What’s another way to cut chrominance
bandwidth in half?
– Use 4-bits per pixel.
• Why won’t that work?
– Need the dynamic range for color.
• But what about when Y is either small or large?
– Don’t need the range, but lots of YUV combinations
that won’t ever be used.
Comp 294-9 :: Fall 2003
4:2:2
• For every 4 luminance samples, take 2
chrominance samples from odd lines and 2
from even lines.
• Chrom. planes just as tall, half as wide.
• JPEG does this.
Comp 294-9 :: Fall 2003
4:2:0
• 2 chrominance samples for every 4
luminance samples, odd lines only.
• Chrominance halved in both directions.
• MPEG generally does this.
Comp 294-9 :: Fall 2003
4:1:1
• What should this be?
– 1 chrominance for every 4 luminance for both
odd and even lines.
– And that is what it is.
• But sometimes used to refer to this:
Comp 294-9 :: Fall 2003
De-interlacing
• Since analog video fields are separated in
time by 1/2 the frame rate, at least half of a
digital frame is missing no matter where
you sample from.
• Normal solution: linear interpolation of
even fields to produce matching samples for
odd fields.
• Even better: linear interpolation of both
fields to produce matching samples and thus
digital frame rate will equal field rate.
Comp 294-9 :: Fall 2003
Film Frame Rates
• Film is a different beast altogether.
– Continuous both vertically and horizontally
– Projection is simultaneous for all parts of the
picture.
– Expensive medium.
– Combination of all of this motivates 24 fps.
• Film to digital is easier than video to digital.
– No interlacing, sample where you want to.
• Film to video is harder.
Comp 294-9 :: Fall 2003
3:2 Pulldown
• Converts 24 frames to 60 fields.
1
2
3
4
5
1e 1o 2e 2o 3e 2o 3e 3o 4e 4o 5e 5o
Comp 294-9 :: Fall 2003
Overall sampling lesson
• Can’t recover what you don’t have.
• Conversion between representations
requires estimation of missing samples.
• Interpolation causes errors:
– Spatially: at the edges.
– Temporally: when moving.
Comp 294-9 :: Fall 2003
Common Digital Video Sizes
•
•
•
•
•
•
CCIR-601
720x480
SIF
360x240
CIF
360x288
4:3 HDTV
1440x1152
9:16 HDTV 1920x1152
4CIF, 16CIF, QCIF
Comp 294-9 :: Fall 2003
4:2:2, 4:2:0
4:2:0
4:2:0
4:2:2, 4:2:0
4:2:2, 4:2:0
Digital Bitrates
• Current television:
– 30 fps * 720 * 480 * 1.5 * 8 = 124 Mb/s
• 9:16 HDTV
– 30 fps * 1920 * 1152 * 1.5 * 8 = 796Mb/s
• This motivates compression.
Comp 294-9 :: Fall 2003