A Quest for an Internet Video
Quality-of-Experience Metric
Athula Balachandran, Vyas Sekar,
Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
1
Internet Video is taking off
Improve Users’ Quality of Experience
2
Video Quality Metrics: The State of the Art
Subjective Scores
(e.g., Mean Opinion
Score)
Objective Score
(e.g., Peak Signal to Noise Ratio)
3
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
4
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
5
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
6
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
7
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
8
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Buffer
empty
Buffer
filled up
Switch
bitrate
9
Problem 1: New Effects, New Metrics
PLAYER Joining
STATES
Playing
Buffering
Playing
EVENTS
Buffer
filled up
Join Time
Buffer
empty
Buffering Ratio
Rate of buffering
Buffer
filled up
Switch
bitrate
Rate of switching
Average bitrate
10
Problem 2: Opinion Scores  Engagement
Opinion Scores
- Not representative of “in the wild”
experience
- Combinatorial explosion of parameters
Engagement as replacement for opinion score.
(e.g., Play time, customer return rate)
11
Internet Video QoE
Subjective Scores
MOS
Objective Scores
PSNR
12
Internet Video QoE
Subjective Scores
MOS
Engagement
(e.g., Fraction of video viewed)
Objective Scores
PSNR
13
Internet Video QoE
Subjective Scores
MOS
Engagement
(e.g., Fraction of video viewed)
Objective Scores
PSNR
Join Time, Avg. bitrate, …?
14
Internet Video QoE
Subjective Scores
MOS
Engagement
(e.g., Fraction of video viewed)
Objective Scores
PSNR
Join Time, Avg. bitrate, …?
f(Join Time, Avg. bitrate, …)
15
Internet Video QoE
Subjective Scores
MOS
Engagement
(e.g., Fraction of video viewed)
Objective Scores
PSNR
Join Time, Avg. bitrate, …?
f(Join Time, Avg. bitrate, …)
16
Outline
• Need for a unified QoE
• What makes this hard?
• Our proposed approach
17
Engagement
Challenge: Complex Engagement-to-metric
Relationships
Quality Metric
18
Engagement
Engagement
Challenge: Complex Engagement-to-metric
Relationships
Non-monotonic
Average bitrate
Quality Metric
[Dobrian et al. Sigcomm 2011]
19
Engagement
Engagement
Challenge: Complex Engagement-to-metric
Relationships
Non-monotonic
Quality Metric
[Dobrian et al. Sigcomm 2011]
Engagement
Average bitrate
Threshold
Rate of switching
20
Challenge: Complex Metric Interdependencies
Join Time
Rate of
buffering
Bitrate
Rate of
switching
Buffering
Ratio
21
Challenge: Complex Metric Interdependencies
Join Time
Rate of
buffering
Bitrate
Rate of
switching
Buffering
Ratio
22
Challenge: Complex Metric Interdependencies
Join Time
Rate of
buffering
Bitrate
Rate of
switching
Buffering
Ratio
23
Challenge: Complex Metric Interdependencies
Join Time
Rate of
buffering
Avg. bitrate
Rate of
switching
Buffering
Ratio
24
Need to learn these complex engagement-to-metric
relationships and metric-to-metric dependencies
25
Casting as a Learning Problem
Need to learn these complex engagement-to-metric
relationships and metric-to-metric dependencies
Engagement
Quality Metrics
MACHINE LEARNING
QoE Model
26
Impact of the ML algorithm
• Classify engagement into uniform classes
• Accuracy = # of accurate predictions/ # of cases
ML algorithm must be expressive enough to handle the
complex relationships and interdependencies
27
Challenge: Confounding Factors
Live and VOD sessions experience similar quality
28
Challenge: Confounding Factors
However, user viewing behavior is very different
29
Challenge: Confounding Factors
Devices
Connectivity
User Interest
Need systematic approach to identify and handle confounding factors
30
Domain-specific Refinement
Engagement
Quality Metrics
MACHINE LEARNING
QoE Model
31
Domain-specific Refinement
Engagement
Confounding
Factors
Quality Metrics
MACHINE LEARNING
QoE Model
32
Improved prediction accuracy
Refined ML models can handle confounding factors
33
Concluding Remarks
• Internet Video needs unified quantitative QoE
• What makes this hard?
–
–
–
Complex engagement-to-metric relationships
Complex metric-to-metric interdependencies
Confounding factors (e.g., genre, device)
• Promising start
–
Machine learning + domain-specific refinements
• Open Challenges
–
–
Coverage over confounding factors
System Design
34