How Many Bits Per Rating
Daniel Kluver1 , Tien T. Nguyen1 , Michael Ekstrand1 ,
Shilad Sen2 , John Riedl1
1 GroupLens Research
Dept. of Computer Science and Engineering
University of Minnesota
2
Math, Stats, and Computer Science Dept.
Macalester College
September 11, 2012
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
1 / 49
?
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
2 / 49
Math
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
3 / 49
Math
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
4 / 49
Math
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
5 / 49
Math
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
6 / 49
Math
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
7 / 49
Better
Math
Item-Item, User-User, Matrix Factorization,
Feature Weighted Linear Stacking
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
8 / 49
Math
Predict, Recommend, Explain Predictions,
Diversify Recomendations
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
9 / 49
Math
Normalization, Re-rating based Denosing
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
10 / 49
Math
User training, Surgery, Fraud detection,
Intercrainial Preference Elicitation
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
11 / 49
Math
Problem - ?
Better - ?
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
12 / 49
Better
Math
Better - Prediction Accuracy
Issue - Magic Barrier
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
13 / 49
Math
Better - ?
Problem - Users rate inconsistantly
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
14 / 49
Goal: Choose the best user interface.
Define Best
Looks good
Makes users happy
Ratings are fast
Most information about user preferences
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
15 / 49
Goal: Choose the best user interface.
Define Best
Looks good
Makes users happy
Ratings are fast
Most information about user preferences
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
15 / 49
Words to know
Preference bits A measure of information about user
preferences.
Bits per rating Measures how much preference
information is contained in ratings.
Bits per second Measures the efficiency of an interface
at capturing preference bits.
Bits per prediction Measures how much preference
information is contained in predictions.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
16 / 49
Goal: Measure information about user preferences.
Define Preference
Define Information
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
17 / 49
Goal: Measure information about user preferences.
Define Preference
The tendency to consistently behave as if you placed
value (positive or negative) on something.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
18 / 49
Preferences n. - The tendency to consistently behave as if
you placed value on something.
πScience
Figure: My value for science is rather large
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
19 / 49
Articulated Values
Deterministic
πBatman
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
RBatman
How Many Bits Per Rating
September 11, 2012
20 / 49
Basic Values
πBat
Memories
...
πMan
Brain
Math
RBatman
πBombs
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
21 / 49
Partially Articulated Values
Not Deterministic
πBatman
RBatman
All noise in rating comes from completely hidden
context.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
22 / 49
Goal: Measure information about user preferences.
Define Information
Thanks to Claude Shannon, Information is a solved
problem.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
23 / 49
Mutual Information
I (X ; Y ) =
P(x,y )
P(x,
y
)log
(
y
P(x)P(y ) )
P P
x
Measurement of how much knowing X increases our
certainty about Y on average.
Normally given in bits
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
24 / 49
Bringing it all together
We can use mutual information to measure how
much information anything tells us about user
preferences.
We call this measurement Preference Bits.
If something has a lot of preference bits then it is
good at explaining user preference.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
25 / 49
Words to know
Preference bits A measure of information about user
preferences.
Bits per rating Measures how much preference
information is contained in ratings.
Bits per second Measures the efficiency of an interface
at capturing preference bits.
Bits per prediction Measures how much preference
information is contained in predictions.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
26 / 49
Goal: Measure information entering the recommender.
Ratings(Ru,i ) enter the recommender.
Ratings measure user preferences (πu,i ).
Therefore we want to measure this:
Measure This
πu,i
Kluver, Nguyen, Ekstrand, Sen, Riedl
I(π,R)
()
How Many Bits Per Rating
Ru,i
September 11, 2012
27 / 49
Goal: Measure information entering the recommender.
Observable
πu,i
Ru,i
Not Observable
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
28 / 49
Measuring Input Preference Bits
π
R1
R2
Prior work solves this problem with two ratings.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
29 / 49
Information Processing Inequality
X
Y
Z
X and Z are conditionally independent given Y
When this is true, I (X ; Z ) ≤ I (X ; Y )
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
30 / 49
Measuring Input Preference Bits
π
R1
R2
For two conditionally independent re-ratings
I (R1 ; R2 ) ≤ I (R; π)
We will use this to measure input preference bits.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
31 / 49
Measuring Input Preference Bits
2 Big assumptions:
R1 conditionally independent with R2 given π
R1 and R2 are generated by the same π
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
32 / 49
Picking User Interfaces
Split users between rating interfaces
Have users rate a bunch of movies
Some time later, have the users rate the same items
Compare preference bits between conditions
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
33 / 49
Picking User Interfaces
Split users between rating interfaces
Have users rate a bunch of movies
Some time later, have the users rate the same items
Compare preference bits between conditions
We haven’t run this (yet)
No one else has either
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
33 / 49
Dataset Analysis
From Cosley et. al. Seeing is believing.
2-point 6-point 10-point
0.423
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
0.825
How Many Bits Per Rating
0.813
September 11, 2012
34 / 49
Preference bits per rating
Effect of Rating Scale on Input Preference Bits
1.8
1.6
1.4
1.2
1.0
0.8
0.6
10
20
30
40
mR
50
60
70
More rating choices, more bits
Information hits a limit
I
I
More noise less bits
More preference options more bits
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
35 / 49
Words to know
Preference bits A measure of information about user
preferences.
Bits per rating Measures how much preference
information is contained in ratings.
Bits per second Measures the efficiency of an interface
at capturing preference bits.
Bits per prediction Measures how much preference
information is contained in predictions.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
36 / 49
A Problem
More rating options ⇒ more information
More rating options ⇒ more cognitive load
More rating options ⇒ slower ratings 1
slower ratings ⇒ less ratings
less ratings ⇒ less information.
More rating options ⇒ less and more information?
1
Sparling et. al. Rating: How difficult is it?
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
37 / 49
Slow Rating
High Information
Fast Rating
Low Information
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
38 / 49
Solution: Bits Per Second
Measuring bits per rating is easy.
Measuring ratings per second is also easy.
It turns out measure bits per second is also easy.
# Bits
# Ratings
# Bits
×
=
1 Rating
1 second
1 second
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
39 / 49
Bits per second: Does it matter?
Using Sparling et. al. Rating: how difficult is it?
and Cosley et. al. Seeing is Believing we can
estimate.
2-point = 0.1082
10-point = 0.1878
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
40 / 49
Words to know
Preference bits A measure of information about user
preferences.
Bits per rating Measures how much preference
information is contained in ratings.
Bits per second Measures the efficiency of an interface
at capturing preference bits.
Bits per prediction Measures how much preference
information is contained in predictions.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
41 / 49
Output Preference Bits
Predictions (Pu,i ) leave the recommender.
Predictions predict user preferences (πu,i )
The amount of preference information leaving the
recommender with I (π; P)
We measure this as I (R; P) ≤ I (π; P)
Yes, its just a fancy accuracy measure,
But it handles varying scales well
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
42 / 49
Suggested use: Choosing how many stars to use.
From Jester dataset (Goldberg et. al.)
Out Preference Bit
0.25
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.20
0.15
●
●
●
●
●
●
●
0.10
●
0.05
0.00
●
2−point
5−point
10−point
20−point
100−point
Output Scale
Input Scale
●
2−point
●
5−point
●
10−point
●
20−point
●
100−point
More prediction options, more bits
Information hits a limit (again)
I
I
input scale controls limit
most bits at 10 point scale
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
43 / 49
Preference Bits
Measure with: Mutual information
Math
A measure of information about user
preferences.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
44 / 49
Bits Per Rating
Mutual information between ratings and preferences I(π;R)
Measure with: I(R1;R2)
Math
Measures how much preference
information is contained in ratings.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
45 / 49
Bits Per Second
Measure with: Bits per rating times Ratings per second
Math
Measures the efficiency of an interface
at capturing preference bits.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
46 / 49
Bits Per Prediction
Mutual information between prediction and preferences I(π;P)
Measure with: I(R;P)
Math
Measures how much preference
information is contained in Predictions.
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
47 / 49
The Next Steps
How many stars should we use?
What information helps users the most?
What are the difference the preference bits of
different domains?
Does preference bits hold any relationship with user
satisfaction?
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
48 / 49
Thank you
Kluver, Nguyen, Ekstrand, Sen, Riedl
()
How Many Bits Per Rating
September 11, 2012
49 / 49