Media Scaling /
Content Adaptation
Ketan Mayer-Patel
CS 294-9 :: Fall 2003
Media Scaling Overview
• Change encoding of the media to
accommodate changes in available bandwidth.
• Components of the problem:
–
–
–
–
–
Estimating bandwidth.
What can be adjusted.
How to adjust.
Cross media issues.
Transcoding pre-encoded media.
CS 294-9 :: Fall 2003
Estimating Bandwidth
• Requires feedback.
– How often?
– In what form?
• TCP-friendliness
– Current thought: act like TCP.
– Padhye’s equation:
M  packet size
BTCP 
t RTT
M

2 Dl
3Dl 

l 1  32l 2
 tout min 1,3

3
8



l  loss fraction

CS 294-9 :: Fall 2003
D  number of packets ACK' ed
by one ACK
Feedback
• How often?
– Design of TFRC says you want to limit this to
once per RTT.
– Otherwise, rate of feedback is a design decision.
• Tradeoff: responsiveness vs. feedback bandwidth
• Should be related to when the source can use the info.
CS 294-9 :: Fall 2003
Feedback
• In what form?
– Media independent information:
• Loss rate and delay.
– Media specific information:
• Quality of reconstructed media.
• Not necessarily well related to loss and delay.
• User preferences
– For example: Regions of interest.
• Media specific feedback very application dependent.
CS 294-9 :: Fall 2003
Scaling Dimensions
• Different media can be scaled along
different dimensions.
• Two major considerations:
– Encoding support.
• The media representation often restricts the range
and type of scalability.
– Perceptual quality.
• Non-linear subjective response to changes.
• Think of frame size changes.
CS 294-9 :: Fall 2003
Video
• Temporal scaling
– Reduce the resolution of the stream by reducing the
frame rate
• Spatial scaling
– Reduce the number of pixels in an image
• Frequency scaling
– Reduce the number of DCT coefficients used in
compression
• Amplitude scaling
– Reduce the color depth of each pixel in the image
• Color space scaling
– Reduce the number of colors available for displaying
the image
CS 294-9 :: Fall 2003
Perceptual Impact and MPEG
Perceptual
Impact
Effectiveness Representable
Temporal Scaling
Medium
Highly
Possible
Spatial Scaling
High
Highly
Unsupported
Frequency Scaling
Medium
Very
Supported
Amplitude Scaling
Low
Not very
Unsupported
Color Scaling
Low
Not very
Unsupported
CS 294-9 :: Fall 2003
Scaling Dynamics
• Given some adjustment parameter p:
– How can we adjust p given target rate r?
• Depends on how p is related to r.
• 3 basis situations:
– Well known in advance.
– Well known for future.
– Indeterminate
CS 294-9 :: Fall 2003
Scaling Dynamics
• Two types of “well known” relationships.
– Immediate relationship.
• Given target bitrate r and media lifespan, parameter
can be perfectly adjusted to give desired number of
bits.
• This is rare.
– Future relationship.
• Parameter is adjusted after the fact to conform to
target rate next time media is produced.
• Example: frame rate.
CS 294-9 :: Fall 2003
Scaling Dynamics
• If relationship is indeterminate, what can we do?
– Incremental adjustments in appropriate direction.
– Basic algorithm:
• At time ti Encode using pi
• Calculate rate mismatch di
• pi+1 = adjustment(pi, di)
– Issues:
• Accounting for errors between target and achieved rates.
• Designing a good adjustment function.
CS 294-9 :: Fall 2003
Bitrate Accounting
• Cumulative accounting.
– Spectrum of accounting choices:
• No carry.
– Deficit/surplus simply informs the adjustment function.
• Indefinite carry.
– In the limit, overall average rate matches target exactly.
• In between, a variety of decay and window-based
possibilities.
CS 294-9 :: Fall 2003
Adjustment Functions
• Basic issues:
– Shape
– Stability
• Lots of control theory here.
• Informal rules of thumb:
– Damp small changes.
– Keep history (feedback)
• Take advantage of domain-specific
knowledge.
CS 294-9 :: Fall 2003
Quality Example
• Quality/rate tradeoff for video generally
looks like this:
Rate
The problem is
that if we don’t
know what the
curve is.
Useful quality adjustment range
Quality
CS 294-9 :: Fall 2003
Quality Example
• Possible strategies?
– Multipass solutions.
• Generally take too long.
– Content analysis
• Complex and only work in restricted domains.
– Model the tradeoff curve.
• Simple
• Only works well is model is close to reality.
• Also known as a “rate-distortion framework”
CS 294-9 :: Fall 2003
Rate-Distortion
• Ideal:
– bits vs. quality as a closed form, invertible formula
• Strategies
– Choose a proxy for quality
• SNR measures
• Playback characteristics
– Example: recovered frames per second
• User feedback
– Choose a proxy for rate
• Q Factor
• Number of non-zero coefficients
• Framerate
CS 294-9 :: Fall 2003
Rate-Distortion cont’d
– Fit curve
• Choice of function family based on representative
sampling of content domain.
• Derive parameters
– Static for content domain.
– Adaptively based on recent ADU’s.
– Analysis of sub-sampled ADU.
CS 294-9 :: Fall 2003
Cross Media Issues
• Aggregate behavior of multiple related
streams may require different per stream
behavior.
– Example: 64 kbs audio with 300 kbs video.
– Suppose available bandwidth drops to 182 kbs
• 32 kbs audio, 150 kbs video
• 64 kbs audio, 118 kbs video.
• Integrate the adaptation process
• Cross inform the adaptation process
CS 294-9 :: Fall 2003
Transcoding
• Have assumed that media encoded on the fly
so far. What about pre-recorded material?
• Transcoding issues:
– Computational complexity
• Compressed domain processing.
– Scalability and availability of the service.
• More common solution:
– Preset, multiple representations.
CS 294-9 :: Fall 2003
InfoPyramid
• Holistic approach to multimedia documents.
• Simultaneous adaptation of all related
media.
• Utility curves defined for each media type
along its adaptive dimension.
• Utility curve for document is summation of
components.
• Document adaptation done as a resource
allocation problem.
CS 294-9 :: Fall 2003