On Appropriate Assumptions to Mine
Data Streams: Analyses and Solutions
Jing Gao† Wei Fan‡ Jiawei Han†
†University of Illinois at Urbana-Champaign
‡IBM T. J. Watson Research Center
Introduction (1)
1
1
• Data Stream
– Continuously arriving
data flow
– Applications: network
traffic, credit card
transaction flow, phone
calling records, etc.
0
1
1
0
1
1
1
0
0
Introduction (2)
• Stream Classification
– Construct a classification model based on past
records
– Use the model to predict labels for new data
– Help decision making
Fraud?
Labeling
Fraud
Classification
model
Framework
………
Classification
Model
………
?
Predict
Existing Stream Mining Methods
• How to use old examples?
– Throw away or fade out old examples
– Select old examples or models which
match the current concepts
Matchthe
themodel?
training
• How to update
distribution!
– Real Time Update
– Batch Update
Existing Stream Mining Methods
• Shared distribution assumption
– Training and test data are from the same
distribution P(x,y) x-feature vector, y-class label
– Validity of existing work relies on the shared
distribution assumption
• Difference from traditional learning
– Both distributions evolve
…
…
………
training
………
………
test
………
Appropriateness of Shared Distribution
• An example of stream
data
– KDDCUP’99 Intrusion
Detection Data
– P(y) evolves
• Shift or delay inevitable
– The future data could be different from current data
– Matching the current distribution to fit the future one
is a wrong way
– The shared distribution assumption is inappropriate
Appropriateness of Shared Distribution
• Changes in P(y)
– P(y)  P(x,y)=P(y|x)P(x)
– The change in P(y) is attributed to changes in
P(y|x) and P(x)
Time
Stamp 1
Time
Stamp 11
Time
Stamp 21
Realistic and relaxed assumption
The training and test distributions are similar to the
degree that the model trained from the training
set D has higher accuracy on the test set T than
both random guessing and predicting the same
class label.
Model
Random
Guessing
Training set
Test set
Fixed
Guessing
Realistic and relaxed assumption
• Strengths of this assumption
– Does not assume any exact relationship
between training and test distribution
– Simply assume that learning is useful
• Develop algorithms based on this
assumption
– Maximize the chance for models to succeed on
future data instead of match current data
A Robust and Extensible
Stream Mining Framework
f 1 ( x, y)
f ( x, y )  P(Y  y | x)
C1
f 2 ( x, y)
C2
1 k i
f ( x, y )   f ( x, y )
k i 1
E
Training set
Test set
……
y | x  arg max y f E ( x, y)
f k ( x, y )
Ck
Simple Voting(SV)
1 model i predicts y
f i ( x, y )  
otherwise
0
Averaging Probability(AP)
f i ( x, y)  probabilit y of predicting y for model i
Why ensemble?
• Ensemble
– Reduce variance caused by single models
– Is more robust than single models when the
distribution is evolving
• Expected error analysis
– Single model:
Err S  EP( x, y ) [ P( y | x)2  2P( y | x) EP( M ) P( y | x, M )  EP( M ) ( P( y | x, M )2 )]
– Ensemble:
Err A  EP( x, y ) [ P( y | x)2  2P( y | x) EP( M ) P( y | x, M )  EP( M ) ( P( y | x, M )) 2 ]
Why simple averaging?
k
• Combining outputs
f ( x, y )   wi f i ( x, y )
E
i 1
– Simple averaging: uniform weights wi=1/k
– Weighted ensemble: non-uniform weights
• wi is inversely proportional to the training errors
– wi should reflect P(M), the probability of model M
after observing the data
• Uniform weights are the best
– P(M) is changing and we could never estimate the
true P(M) and when and how it changes
– Uniform weights could minimize the expected
distance between P(M) and weight vector
An illustration
• Single models (M1, M2, M3) have huge variance.
• Simple averaging ensemble (AP) is more stable and
accurate.
• Weighted ensemble (WE) is not as Single
good as AP since
Models
training errors and test errors may have different
distributions.
AP
M1
M2
M3
WE
Weighted
Ensemble
Average
Time
Probability
Stamp
A
Time
Stamp
B
Training Error
Test Error
Experiments
• Set up
– Data streams with chunks T1, T2, …, TN
– Use Ti as the training set to classify Ti+1
• Measures
– Mean Squared Error, Accuracy
– Number of Wins, Number of Loses
– Normalized Accuracy, MSE
h( A, T )  h( A, T ) / max A (h( A, T ))
Experiments
• Methods
– Single models: Decision tree (DT), SVM, Logistic
Regression (LR)
– Weighted ensemble: weights reflect the accuracy on
training set (WE)
– Simple ensemble: voting (SV) or probability
averaging (AP)
Experimental Results (1)
35
30
Time 40
25
DT
SVM
LR
WE
SV
AP
20
15
10
5
0
#Wins
#Loses
60
Time 100
50
DT
SVM
LR
WE
SV
AP
40
30
20
10
0
#Wins
#Loses
Comparison on Synthetic Data
Experimental Results (2)
50
45
40
35
30
25
20
15
10
5
0
DT
SVM
LR
WE
SV
AP
#Wins
#Loses
Comparison on Intrusion Data Set
Experimental Results (3)
Classification Accuracy Comparison
Experimental Results (4)
Mean Squared Error Comparison
Conclusions
• Realistic assumption
– Take into account the difference between
training and test distributions
– Overly matching the training distribution is
thus unsatisfactory
• Model averaging
– Robust and accurate
– Theoretically proved the effectiveness
– Could give the best predictions on average
Thanks!
• Any questions?