Learning with Positive and
Unlabeled Examples using
Weighted Logistic
Regression
Wee Sun Lee
National University of Singapore
Bing Liu
University of Illinois, Chicago
Personalized Web Browser
• Learn web pages that are
of interest to you!
• Information that is
available to browser when
it is installed:
– Your bookmark (or cached
documents) – Positive
examples
– All documents in the web –
Unlabeled examples!!
Direct Marketing
• Company has database with details of its
customer – positive examples
• Want to find people who are similar to their
own customer
• Buy a database consisting of details of
people, some of whom may be potential
customers – unlabeled examples.
Assumptions
• All examples are drawn independently
from a fixed underlying distribution
• Negative examples are never labeled
• With fixed probability , positive example
is independently left unlabeled.
Are Unlabeled Examples
Helpful?
• Function known to be
either x1 < 0 or x2 > 0
• Which one is it?
x1 < 0
++u +
u +u +
+ ++ +
x2 > 0
uu u
u uu
uu
Not learnable with only positive
examples. However, addition of
unlabeled examples makes it
learnable.
Related Works
• Denis (1998) showed that function classes
learnable in the statistical query model is
learnable from positive and unlabeled examples.
• Muggleton (2001) showed that learning from
positive examples is possible if the distribution of
inputs is known.
• Liu et.al. (2002) give sample complexity bounds
and an algorithm based on EM
• Yu et.al. (2002) gives algorithm based on SVM
• …
Approach
• Label all unlabeled examples as negative (Denis
1998)
– Negative examples are always labeled negative
– Positive examples are labeled negative with
probability 
• Training with one-sided noise
• Problem:  is not known
• Also, what if there is some noise on the negative
examples? Negative examples occasionally
labeled positive with small probability.
Selecting Threshold and
Robustness to Noise
• Approach: Reweigh examples and
learn conditional probability P(Y=1|X)
• If you weight the examples by
– Multiplying the negative examples with
weight equal to the number of positive
examples and
– Multiplying the positive examples with
weight equal to the number of negative
examples
Selecting Threshold and
Robustness to Noise
• Then P(Y=1|X) > 0.5 when X is a positive
example and P(Y=1|X) < 0.5 when X is a
negative example, as long as
– + < 1 where
•  is probability that positive example is labeled negative
•  is probability that negative example is labeled positive
• Okay, even if some of the positive examples are
not actually positive (noise).
Weighted Logistic Regression
• Practical algorithm: Reweigh the examples and
then do logistic regression with linear function to
learn P(Y=1|X).
– Compose linear function with sigmoid then do
maximum likelihood estimation
• Convex optimization problem
• Will learn the correct conditional probability if it
can be represented
• Minimize upper bound to weighted classification
error if cannot be represented – still makes
sense.
Selecting Regularization Parameter
• Regularization important when learning with
noise
• Add c times sum of squared values of weights to
cost function as regularization
• How to choose the value of c?
– When both positive and negative examples available,
can use validation set to choose c.
– Can use weighted examples in a validation set to
choose c, but not sure if this makes sense?
Selecting Regularization Parameter
• Performance criteria pr/P(Y=1) can be estimated
directly from validation set as r2/P(f(X) = 1)
– Recall r = P(f(X) = 1| Y = 1)
– Precision p = P(Y = 1| f(X) = 1)
• Can use for
– tuning regularization parameter c
– also to compare different algorithms when only positive
and unlabeled examples (no negative) available
• Behavior similar to commonly used F-score
F = 2pr/(p+r)
– Reasonable when use of F-score reasonable
Experimental Setup
• 20 Newsgroup dataset
• 1 group positive, 19 others negative
• Term frequency as features, normalized to
length 1
• Randomly split
– 50% train
– 20% validation
– 30% test
• Validation set used to select regularization
parameter from small discrete set then retrain
on training+validation set
Results
F-score averaged over 20 groups

Opt
0.3
0.757
0.754
0.646
0.661
1-Cls
SVM
0.15
0.7
0.675
0.659
0.619
0.59
0.153
pr/P(Y=1) Weighted S-EM
Error
Conclusions
• Learning from positive and unlabeled
examples by learning P(Y=1|X) after setting
all unlabeled examples negative.
– Reweighing examples allows threshold at 0.5 and
makes it tolerant to negative examples that are
misclassified as positive
• Performance measure pr/P(Y=1) can be
estimated from data
– Useful when F-score is reasonable
– Can be used to select regularization parameter
• Logistic regression using linear regression
and these methods works well on text
classification