PAKDD 2006 Data Mining Competition
Institution:
School of Computer Engineering
Nanyang Technological University
Group members:
1. Le Minh Tam
2. Vu Thuy Linh
3. Duong Minh Chau
Problem statement
A telco operator would like to make full use of existing customer usage and demographic
data to identify potential customers to their newly-launched 3G mobile communication
network. This is a typical classifying problem.
Data sets
1. Overview
Each customer data is represented by a record of 251 fields, ranging from demographic
characteristics to mobile usage (in the last 6 months). The target field is
CUSTOMER_TYPE, which identifies the type of customer (i.e. is he/she likely to switch
to the new 3G service). 24000 sample data records are provided. Three quarters (18000)
of the sample data are given with target field available, for training/testing purpose. The
rest of the sample is to be used for prediction.
2. Statistics
The ratio 3G:2G records in the sample data is 1:5 (i.e. 20k 2G and 4k 3G records). The
training portion and testing portion also have the same ratio. We discarded the
SERIAL_NUMBER attribute as it is irrelevant to the classification. After summarizing
the attributes, we got the following statistics:
-
-
10% of the fields have value of zero in all training records. Those are of numeric
type only.
Some attributes are somehow related to each other, yet cannot be aggregated as
each of them has its own meaning. Examples of those related groups are:
(SUBPLAN, SUBPLAN_PREVIOUS), (TOP1_INT_CD, TOP2_INT_CD,
TOP3_INT_CD). These are all categorical fields.
Most of the numeric attributes possess a skewed distribution (right-skewed, in
fact). Around 50% of those have only one peak at 0.
3. Division of training records
We used 3 methods to pick the subsets of attributes: pick all attributes (excluding the
SERIAL_NUMBER), pick the top-20 attributes whose entropies are smallest, and
randomly pick a subset of attributes. The method to compute entropy will be represented
in this document shortly after.
In random attribute selection, we made use of the domain fact that 3G customers are
likely to use more of non-voice service such as games, chat, GPRS, WAP, etc. Another
fact to be considered was that demographic data such as AGE or GENDER may have an
impact on the CUSTOMER_TYPE (e.g. younger people tend to follow suit technology
trend, and hence may likely switch to 3G service)
During training, the training data set was divided into 2 proportional sets with
corresponding ratio of 3:1, 4:1, 9:1, 19:1. The method used to distribute the records is
based on random number generator in Java (with seed number taken from machine’s
timestamp) to guarantee the randomness of the activity. During the distribution, the
proportion 3G:2G in the data sets were maintained. Here is the pseudo code for random
distribution into 2 sub-datasets with ratio a:b
rand = new Random( timestamp );
v = new Vector()
fill v with record-index (SERIAL_NUMBER) of records
d[] is an array to mark which record belongs to which sub data
sets
turnPositive = 0;
turnNegative = 0;
while (v has more records){
temp = abs(rand.nextInt());
if (temp is in v){
remove temp from v
if (record numbered temp is positive){
if (turnPositive < a)
d[temp] = 0;
else
d[temp] = 1;
turnPositive = (turnPositive + 1) mod (a + b);
}
else{
if (turnNegative < a)
d[temp] = 0;
else
d[temp] = 1;
turnNegative = (turnNegative + 1) mod (a + b);
}
}
move to next record of v;
}
use d[] to copy the records to the corresponding sub-dataset 0
and 1;
The next section presents how we calculate the entropies of the attributes.
4. Calculating Entropy
A. Entropy Formula
The entropy e i of the i th interval:
k
ei   pi j log 2 pij
i
where pij  mi j / mi is the probability of class j int the i th interval.
The total entropy:
n
e   wi ei
i 1
Where m is the number of value, wi  mi / m is the fraction of values in the i th
interval, n is the number of inverval.
B. Implementing Entropy
According to the data dictionary, there are two types of data: categorical and numeric
as known as nominal and contiguous data type. Therefore, it’s up to the data types that
we apply the entropy formula differently.
For the categorical data:
There will be a set of distinct values for that data. At each distinct value
(including the missing/null value), we can count the number of customers who are
classified as 2G, 3G. After getting the count results, we can calculate the entropy for each
distinct value in the value set of that data. To finally get the entropy of the data, we have
to do one more step according to the total entropy formula. We find out the fraction of
each value in the set (which is the proportion of the number of customers having that
value above the total number of customers in the data training file). Then we apply the
formula to get the final entropy value of the categorical data from its distinct entropy
values in the set.
For the numeric data:
We need to find the position to split data into 2 subsets, the entropies of which
will be computed and combined into the final total entropy.
Firstly, we sort all the value to get an order array. For example, after sorting, we
have the array:
60
70
75
85
Split positions are identified by taking midpoints between two adjacent sorted
values:
55
65
72
80
87
We use test condition  to get data, count the number of 2G with value greater
than 55, and the number of 3G with value greater than 55:
5
5
>
2
0
3
3
0
7
G
G
For the next node of split position 65, we check the CUSTOMER_TYPE of value
60, if it is 2G, we increase the number of 2G with value smaller than 65 and decrease the
number of 2G with value greater than 65. If it is 3G, we increase the number of 3G with
value smaller than 65 and decrease the number of 3G with value greater than 65. One
after one, we create the table:
3G
2G
Entropy
2G
2G
2G
3G
60
70
75
85
55
65
72
80
87
 >
0 3
0 7
0.8813
 >
0 3
1 6
0.8265
 >
0 3
2 5
0.7635
 >
0 3
3 4
0.6897
 >
1 2
3 4
0.8755
The minimum entropy gives the position split. In this case, the split position is 80
and entropy equals to 0.6897
5. Measurements
After picking the attributes, dividing the records into 2 sub-datasets, we used the larger
sets for training and the smaller for testing. We measured the performance based on the
precision and recall yielded on the testing subset. We combine the 2 measurements in the
F1-index, which was computed by:
F1 
2* precision * recall
precision  recall
The higher the F1-index, the better the performance. From the equation above, we see
that for each test, a balance between precision and recall would results in the largest F1index.
Classifiers
Among all available classifying algorithms, we used Decision Tree (C4.5) and SVM
(Support
Vector
Machine).
C4.5
is
provided
in
WEKA
(http://www.cs.waikato.ac.nz/ml/weka/) and SVM is provided in SVMlight
(http://svmlight.joachims.org/) in open-source form. Only some typical sample results
will be presented below, although the same process might have been carried out for
several times randomly.
1. Decision tree C4.5
Subset of attributes
All 250 attributes
20 entropy-picked attributes
20 randomly-picked atributes
81 attributes selected based on shape of distribution
Precision
48.83%
51.72%
60.66%
52.78%
Recall
2.56%
3.5%
45.8%
48.63%
F1-index
4.86%
6.55%
54.76%
50.62%
According to the result above, we conclude that decision tree is not a good solution
for this particular problem. Our explanation for this is that decision tree is mainly based
on computation and comparison of information gain, and the rest of the decision mostly
based on majority. Therefore, given a heavily-skewed distribution, with a high peak on
the zero position, and only up to about 2% of the data set lies on the tail, the decision tree
algorithm would not yield a good result. With all-attributes-picked, or entropy-picked, we
can observe a precision of approximately 50%, while recall measure is miserably as low
as less than 5%. The result indicates that the algorithm was only able to classify a few of
the positive records which lie on the tail, while failing to classify the larger portion lies in
the zero peak (because there is obviously much more 2G than 3G in the peak, majority
rule would favor 2G). We suspected the reason why 20 randomly-picked attributes gave
unexpectedly higher performance was the randomness of the selection which helped
eliminating some of the attributes whose distributions are heavily skewed.
We also made an attempt to prove the effect of distribution shape on the classifying
performance. Out of 251 attributes, we selected 81, whose distribution shapes are
distinguishable (i.e. not possessing a zero peak, and a small tail like the rest). The result
was 52.78% precision and 48.63% of recall, which implies an F1-index of approximately
50%, which is better than all-picked, or entropy-picked, which consist of skewed fields.
We conclude that the shape of distribution does have a great impact on the
performance of the classification algorithm, as well as the choice of which algorithms to
be used.
2. Support Vector Machine (SVM)
Again, only typical sample results are given in this section, although the same test
case might have been carried out for several times (including randomly distributing the
subsets again and again)
The first time we tested with SVM, we didn’t include the “multiple codes” categorical
attributes (such as NATIONALITY, OCCUP_CD, etc.). We also made a mistake when
splitting the binary flag attributes so that missing values were grouped together with zerovalue. Follow are the results:
Subset of attributes
All 250 attributes
20 entropy-picked
attributes
20 randomly-picked
attributes
Ratio
training
testing set
3:1
4:1
9:1
19:1
3:1
9:1
3:1
9:1
19:1
of Cost
vs. factor
2.8
2.5
2.59
2.55
2.5
2.5
2.47
2.515
2.51
Precision Recall
F1-index
53.93%
54.62%
55.67%
51.37%
49.68%
48.59%
51.33%
55.03%
53.33%
53.96%
54.86%
55.67%
50.68%
44.88%
44.08%
51.33%
54.85%
53.33%
54%
55.1%
55.67%
50%
40.93%
40.33%
51.33%
54.67%
53.33%
We made a second time testing with SVM, correcting the mistakes, and the results are
as follow:
Subset of attributes
All 250 attributes
20 entropy-picked
attributes
20 randomly-picked
attributes
Ratio
of Cost
training vs. factor
testing set
3:1
3.25
9:1
3.3
19:1
3.3
9:1
2.4
19:1
2.3
3.1
2.6
9:1
2.6
19:1
2.6
Precision Recall
F1-index
66.4%
65.22%
66.67%
47.67%
51.3%
53.97%
55.52%
58%
66.26%
65.11%
66.67%
47.67%
51.98%
53.24%
55.42%
58%
66.13%
65%
66.67%
47.67%
52.67%
52.53%
55.33%
58%
Based on the tables above, it is apparent that the SVM yields a better result than
decision tree C4.5, in this particular problem. The F1-index, tested with all-attributespicked, varies from 65% to above 70%, which is far better than the 4.86 F1-index of C4.5
When we ignored “multiple codes” fields, the result was lowered by approximately
10%. Mistakenly grouping missing values and zero-value resulted in inaccurate modeling
of the data.
We could also observe that all-attributes-picked set always yielded better result than
the rest. Hence, we may conclude that the greater the number of attributes involved, the
better SVM performs.
Another observation worth of noticing is that SVM’s runtime tends to be faster as the
number of attributes increases. The fewer the number of attributes picked, the more time
it takes to optimize the model
We also made several attempts to make use of the AVG and STD pairs of attributes in
the data set. For example, we tried using the 99% confidence interval (AVG ± 3.27 STD),
75% confidence interval (AVG ± 1.5 STD) rather than using AVG and STD directly. Our
idea was that when a pair of AVG and STD was fed into the classifying algorithm, it is
treated as 2 separate numeric fields, without any relation in between, while in fact they
together describe a distribution. However, the performance yielded was approximately
the same (from 65% to 73%).
Therefore, we conclude that SVM performs better, and it is able to select attributes by
itself (rather than requiring manual selection of attributes).
The submitted prediction data was generated by SVMlight, with cost factor of
3.3