Kevin Savage
Toilet Stats
Measuring usage
•
•
•
•
•
This talk is about measuing usage
What we measure
How we use R to predict future usage
Specific code examples
Things that didn't work for us
Background
•
•
•
•
•
•
Mendeley is an academic reference
management system
Desktop client
Automatic metadata extraction
Crowdsourced library of documents
Website, iOS, Android, APIs...
Make it easy to manage and reference
academic literature
Background
•
•
•
•
Mendeley bought in 2013 by the publisher
Elsevier
Up until then we used burn rate
Our targets were changed to growth
Measured in quite a complicated way
Burn rate
•
•
•
•
As a start-up you have funding from
investers and income from customers
You have to make a profit before you
"burn" all your investment money
If you don't, the business fails
You measure your "burn rate"
Core Users
Core Users = Measured Core Users +
Estimated Core Users
Core Users
Core Users = Measured Core Users +
Estimated Core Users
MCU if > X sessions in the past 24 weeks
Core Users
Core Users = Measured Core Users +
Estimated Core Users
MCU if > X sessions in the past 24 weeks
ECU if > Y sessions in the second week
and they are <24 weeks old
Targets
Graph showing model
Targets
This seems like a reasonable idea
This seems like a reasonable way of
measuring it
Ideally we would like to know we are
increasing core users when we make
software changes
Can we do some analysis?
Properties of Core Users
Long term, hard to see cause and effect
We have an issue with delayed event
capture
Estimated Core Users is not a very good
estimate
Our usage is very seasonal
Long term
Messages arriving after the event
Classifiers
Estimated core users is a prediction of
becoming a measured core user
How good a prediction is it?
Measuring classifiers
Precision: if we predict someone will be a
Measured Core User, how likely is it they
will be?
Recall: if someone becomes a measured
core user, how likely is it that we predicted
this?
Performance of our classifier
Precision ~ 0.5
Recall ~ 0.5
Improving the classifier
•
•
•
•
•
•
Logistic regression
K-means
PCA/segmentation
GLA
Random forests
Decision trees
Seasonality of the data
Graph showing model
STL
Seasonal Decomposition of Time Series by
Loess (Cleveland et al 1990)
Decomposes into seasonal, trend and
remainer
Cleveland shows that it is performant even
for long data series
Loess smoothing
LOcal regrESSion
Generate local d-polynomial fit to nearest q
points weighted by distance
As q -> ∞ becomes ordinary least squares
With d = 0 this is moving average
We use d = 1 (locally linear) and q = season
length (a year)
STL
•
•
•
•
•
We want to calculate Y=S+T+R
STL works iteratively
We use an initial value for T = 0
We only use the ‘inner loop’ and only
iterate twice but you can also use an
‘outer loop’
STL then post smooths the seasonal
component
For the parameters we use this is...
•
•
•
•
•
•
•
•
Calculate Y - T
Break into cycle-subseries
Smooth the above with loess to give C
Low pass filter of C to give L
Detrend C by calculating C - L to give new
S
Smooth Y - S to give new T
Repeat
Smooth S and calculate R as Y – S - T
Code
series <- read.table(file="data.csv",
header=F, sep=";")$V2
myts <- ts(data=series, start=c(2011, 43),
frequency=52)
mystl <- stl(myts, s.window="periodic")
plot(mystl)
Result of decomposition
[Graph of decomp]
Prediction
Take the trend
Use a linear model to predict future trend
Add back in the seasonal
Add in some error estimates based on fit
There is an R package for this on CRAN
Code
library(forecast)
mystlf <- stlf(ts, s.window="periodic")
plot(mystlf)
Prediction
Looking good...
Looking ok...
Looking...
Oh…
What happened?
?
Prediction vs Reality
[Graph of prediction]
Assuming we had the data
Why are we underestimating?
Possibilities:
• The underlying trend is not linear
• We’re doing better than expected:
–
–
–
iOS app
integration with other products
improvements to existing products
Where are we now?
We might already have hit the target
Once new messages come in we should
exceed the target
Very high chance of success (according to
our model)
It could still go wrong...
Next steps
Can we improve our results by assuming a
non linear trend?
Can we use seasonality to make better
predictions?
Can we feed core user estimates back into
our design process?
Does the metric need changing?
Communication
Toilet stats
[picture of toilet]
Questions
?