Angel algorithm: A novel global estimation level
algorithm for discovering extended association rules
between any variable types.
Chatzigiannakis-Kokkidis Angelos - angel@economist.net
Panteion University, 136 Syngrou Ave, Athens 17671, Greece
Summary. The association rules analysis is a key part of the data mining methodology. Association
rules represent interesting patterns between itemsets in a database. The research on this topic so far
has produced many different algorithms that are more or only suitable to analyze either categorical or
quantitative data. In this paper we present a new algorithm that can analyze itemsets that contain either quantitative or categorical data or a mix of them on either side of an association. Moreover both
the independent and the dependent variables of an association can be any number and of any type.
Therefore our algorithm can be used for analysis of both market-basket data and financial databases
taking into account dimensional attributes, such as time, distance etc.
1. Introduction
Association rules have been discussed thoroughly in the past. They were formally introduced in [AIS93] and were refined by the Apriori algorithm ([AS94]). Both algorithms addressed the market basked analysis problem trying to find all possible associations between
sets of items, where each item represents a Boolean variable for the presence or absence of
a product in the basket of a consumer. In [SA96] the authors of Apriori published the quantitative Apriori algorithm which extended the capabilities of the former to analyze categorized quantitative data.
A very good algorithm that was proposed in the context of analyzing quantitative data
for associations was the Window algorithm of Aumann and Lindell, published in [AUL03].
Aumann and Lindell provided a very good framework on categorizing the independent
quantitative variables according to their impact on the quantitative dependent variables.
Moreover the quantitative dependent variables are not categorized, because the authors use
a statistical interestingness measure, such as the mean or the variance of the dependent
variable.
The algorithm described in this paper uses the Window algorithm to categorize the quantitative independent variables and extends the algorithm of Aumann and Lindell [AUL03]
by the points described below:
950
1. Analysis of rules with multiple variables on both sides of the rule.
2. Introduction of the homogeneity measures.
3. Ability to analyze categorical and quantitative data on both sides, i.e. Categorical or
Quantitative Categorical or Quantitative rules.
4. Ability to analyze intertransactional associations.
2. Definitions
2.1 Association Rules
Let E = {e1,…,em} be the set of variables (or fields) for a database table D of transactions
T. Let I = {i1,…,in} be a set of literals, called items of the following form:
ik={ej , vL, vU}
(1)
with ej ∈ E, ik ∈ I and vL, vU two values of this variable, where vL ≤ vU if ej is a quantitative
variable and vL = vU if ej is a categorical variable. For example ej can be the amount of a
bank transaction and ik = {amount, 10.000, 20.000} a categorized expression of this variable, noted as item. Furthermore, if ej was a categorical variable such as the city of the account owner then ik = {city, ‘Athens’, ’Athens’} could be a valid item of this variable. For
simplicity reasons we will also denote the above item as ik = {city, ‘Athens’}
Let XQ be a set of quantitative-variable items in I. Also, let XC be a set of categoricalvariable items in I. Let YQ be a set of quantitative variables in E. Also let YC be a set of
categorical variables in E.
An association rule is an implication of the following form:
X ⇒ Y, measure(Y)
(2)
X = {x1,…,xn1}, xr ∈ XQ U XC , and
Y = {y1,…,yn2}, ys ∈ YQ U YC
The left-hand side of a rule defines a subset of the database table D and is called profile.
The profile can be a set of r number of items (noted as r-itemset) which are either categorical found in XC or quantitative found in XQ. For a transaction t = {⟨e1,v1⟩,…,⟨em,vm⟩}, we
say that t has profile Xr if t coincides with Xr whenever Xr is defined. We denote this set of
transactions as TXr.
The right-hand side of a rule consists of a vector of measure values for a set of quantitative or categorical variables in YQ or YC respectively, with the measure taken over TX, the
transactions which match the profile of the left-hand side of the rule. In this paper we use
the mean value as the measure to define the significance of a rule; however other measures
may be used as well. We denote as yTx the mean value of y ∈ Y for the profile TX.
An r-itemset profile may contain other (r-1)-itemset profiles. We say that the (r-1)itemset profile Xr-1 is a superset of Xr if r > 1 and TXr ⊆ TXr-1. For 1-itemset profiles X1 their
superset is the database table D denoted as TX1 ⊆ TD. Likewise we say that the profile Xr is
a subset of the profile Xr-1.
where
951
An example could be the following two itemsets:
X2 = {age, 20, 29}, {smoker, yes} defines transaction set TX2
X1 = {age, 20, 29} defines transaction set TX1 with TX2 ⊆ TX1
A rule points to an extraordinary behavior of the variables in Y, if the measures (in our
case: means) of those variables in the transactions TXr, defined by the profile, is significantly different from the same measures of the same variables in the transactions of a superset of the profile, TXr-1. Note that although the mean values may be different in the specific database, this may not be the case for the entire population, thus a statistical test has to
be measure the significance level of the difference. In our case we assume that all variables
follow the normal distribution, thus we use the Z-test. However the statistical test used may
vary. Thus we define the significance of a rule as:
MeanY(TXr) ≠ MeanY(TXr-1)
(3)
2.2 Definitions for Intertransaction Association Rules
Let E = C∪D∪E’ ={c1,…,cnc} ∪{d} ∪ {e1,…,ene) be the set of variables for a database
table D of transactions t = {⟨c1,vc1⟩,…,⟨cnc,vcnc⟩,⟨d,vd⟩, ⟨e1,ve1⟩,…,⟨ene,vene⟩}. We will define
the following types of variables (fields):
Dimensional Variable d: This is a field in the database table D that defines the location
in the dimensional space for each transaction t.
Context Identifiers C: These are the fields in the database table D that define a unique
entity value for the current analysis such that each value set {⟨c1,vc1⟩,…,⟨cnc,vcnc⟩} appears
once and only once in the transaction set T for each value of the dimensional variable.
General Variables E’: This is the set of all other variables in D that are neither dimensional nor context identifiers.
3. Algorithm
The Angel algorithm described in this paper performs the following steps:
(1) Create the extended table, (2) Create Boolean variables for all variables in YC, (3) Generate all items for itemsets XQ and XC by categorizing quantitative independent variables
and finding all distinct values for categorical independent variables, (4) Find the candidate
itemsets, (5) Prune itemsets containing non-large itemsets, (6) Find the large itemsets, (7)
Find the rules between the itemsets in X and the variables in Y, (8) Perform a homogeneity
check to decide which itemsets should be included in the next candidate itemset generation.
The first step is based on the work done in [LFH00], although with many changes, the
steps 4, 5, and 6 are based on [SA96] and the step 3 is based on [AUL03] with several
changes to support multiple variables on the right side of the rule (Y).
952
3.1 Extend Dimensions
The first step performed in the algorithm is to extend the table D in case the analyst has
chosen to perform an intertransactional analysis. The algorithm uses a user defined value
maxspan ∈ N which defines the depth of the dimensional analysis. If maxspan = 0 then this
step of extending the table D is not performed.
In this step the algorithm creates an extended table De (see example in table 2), which is
based on the initial table D (see example in table 1) with a set Te of transactions each of
which has the following form:
te = {⟨c1,vc1⟩,…,⟨ccn,vcn⟩,⟨d,vd⟩, ⟨Δ0(e1),Δ0(ve1)⟩,…,⟨Δ0(een),Δ0(ven)⟩,
⟨Δ1(e1),Δ1(ve1)⟩,…,⟨Δ1(een),Δ1(ven)⟩, ⟨Δ2(e1),Δ2(ve1)⟩,…,⟨Δ2(een),Δ2(ven)⟩,
…
⟨Δmaxspan(e1), Δmaxspan(ve1)⟩,…,⟨ Δmaxspan(een), Δmaxspan(ven)⟩}
where
(4)
⟨Δ0(ei), Δ0(vei)⟩ ∈ Te(k) = ⟨ei,vei⟩ ∈ T(k) and
⟨Δj(ei), Δj(vei)⟩ ∈ Te(k) = ⟨ei,vei⟩ ∈ T(k+j)
In other words we denote with ⟨Δ0(ei), Δ0(vei)⟩ the value of the field ei in the current
transaction and ⟨Δj(ei), Δj(vei)⟩ in the jth transaction where j is the distance (span) of this
transaction from the current one. The extended transaction contains all the fields and values
of the non-extended transaction plus the extended fields which come from the next transactions of the table D. Each extended transaction contains maxspan extended fields. Also, if
the transaction in row k+j, T(k+j), has other value for one of the context identifier variables
ci than transaction in row k, then the transaction k is not included in the extended table.
Table 1. Example of initial database table D
Client ID c1 Date d
Account balance e1 Transaction amount e2
1051
10/1/2007
500 EUR
100 EUR
1051
11/1/2007
600 EUR
110 EUR
1052
5/1/2007
1830 EUR
40 EUR
1052
6/1/2007
1870 EUR
-100 EUR
1052
7/1/2007
1770 EUR
-200 EUR
1053
20/1/2007
1000 EUR
200 EUR
After the extension, each transaction te of the extended database table De has all the variables needed for an intratransactional algorithm to perform the association rule analysis.
Table 2. Extended Table De
Client ID c1
Date d
Account bal. Transaction Account bal. Transaction
Δ0(e1) amount Δ0(e2) Δ1(e1) amount Δ1(e2)
953
1051
10/1/2007
500 EUR
100 EUR
600 EUR
110 EUR
1052
5/1/2007
1830 EUR
40 EUR
1870 EUR -100 EUR
1052
6/1/2007
1870 EUR
-100 EUR
1770 EUR -200 EUR
3.2 Categorization
The categorization of the quantitative independent variables in E is performed using the
Window procedure described in [AUL03]. The Angel algorithm has three differences in the
way it performs the Window procedure:
a) It performs the procedure for each member of Y in order to support multiple dependent
variables.
b) It creates categories for window-below, window-above, and window-equal results.
c) It performs the procedure to extract categorical items of the form ik (See Eq. 1).
Therefore each quantitative independent variable has up to three different categorizations
for each independent variable. All categories extracted from this process are added to XQ,
the set of quantitative independent items. The pseudo-code of this process is shown in Figure 1.
Fig. 1. The usage of the Window procedure to define variables in X
foreach quantitative e in E do
foreach y in Y do
If not y is extended variable
X = X U {x : x in Window-above(D,e,y,mindif)}
X = X U {x : x in Window-below(D,e,y,mindif)}
X = X U {x : x in Window-equal(D,e,y,mindif)}
endfor
endfor
Additionally in order to support Quantitative
Categorical rules the algorithm splits
each categorical variable with c possible categories into c Boolean variables with value vc=
1 if the transaction contains the category c or value vc = 0 if the transaction does not contain
the category c. The new Boolean variables are then treated like quantitative variables.
3.3 Rule Generation
The rule generation process of Angel consists of the candidate itemsets generation, the
large itemsets generation, the pruning phase, the rule generation, and the homogeneity
check.
The set of candidate itemsets Cn is the set of the Xn itemsets that will participate in the
rule generation process of the iteration n. On the first iteration the algorithm creates the set
C1 which is the set of all items in XC and XQ, noted as C1 = Xc ∪ XQ. Each item in C1 is a 1item itemset. On the next iterations the algorithm forms the candidate itemset as Cn = Nn-1 ×
L1. Nn-1 is a set of (n-1)-itemsets that is the result of the homogeneity check, to be described
later, and L1 is the set of large 1-itemsets, also to be described below.
954
The set of large itemsets Ln is the set of the Xn itemsets that are found in Cn and pass the
minimum support threshold. On each iteration a table scan is performed to determine which
items of Cn contain the minimum number of transactions in the database. The process is
similar to the one described in [AS94] and [SA96].Formally we denote this as:
Xn ∈ Cn ∧ Count(TXn) ≥ minsup ⇒ Xn ∈ Ln
(5)
After the candidate itemset generation, a pruning of the candidate itemsets is performed.
During this phase the algorithm checks if any n-itemset of Ci contains (n-1)-itemsets that
are not contained in the Ln-1 set. If it finds such itemset it prunes it 1 . Formally we can describe this as:
If ∃ Xn-1 ⊆ Xn ∧ Xn-1 ∉ Ln-1 ⇒ Xn ∉ Ci
(6)
The rule generation process finds all rules of the form of Eq. 2 which satisfy the criterion
defined by the analyst as “measure”. As noted in section (2) we will use the mean criterion
assuming that all variables in D follow the normal distribution. Thus RuleGen statistically
compares the mean of each dependent variable in Y for the profile Xn against the mean of
the same dependent variable for the profile of each superset of X. If the two means are statistically different, the rule X ⇒ y, mean(y) is added to the rules set S.
Finally, in contrast with other algorithms, the current algorithm does not combine all
large n-1 itemsets to find the candidate n-itemsets. It performs first a homogeneity check
and combines only those itemsets that pass this check, and also adds them to the Nn set. To
do so the algorithm performs a check on each itemset in Ln to determine the maximum statistical difference that any subset of this profile can have without taking into account any
subordinate Xn+1 profile. In our case we use the max-Z criterion which is the maximum
value of the Z-test perceived to conclude the equality between the mean of the Xn profile
and a subordinate profile. If the maximum Z value that can be produced by the subordinate
profiles is such that it is less than the maxZ defined by the analyst then the itemset Xn is not
inserted into Nn although it might be a member of Ln. Formally we can describe this as follows:
If not ∃ Xn+1 : Z-Test(yTxn+1, yTxn) > maxZ ∀ y ∈ Y ⇒ Xn ∉ Nn
Fig. 2. The Angel Algorithm main loop
1
The pruning phase was first introduced in [SA94].
(7)
955
Angel (D, X, Y, mindif, minsup, maxspan, maxZ,toleratePercent)
1. D = ExtendDimensions(D)
2. AdjustCategories(D)
3. For each yi in Y do
4.
μ(yi) ← Average (yi) in D ; add μ(yi)
M1(Y)
5.
σ(yi)  Variance (yi) in D; add σ(yi)
S1(Y)
6. For each quantitative variable e ∈ EX do
7.
Categorize (e, Y, M1(Y), S1(Y), mindif)
8. C1  All items in X = XQ U XC
9. L1  LargeGen(C1)
//find the first large itemsets array
10. S  S U RuleGen(D,L1,X, Y, M1(Y), M0(Y),S0(Y), maxZ, mindif, minsup)
11. N1  CheckHomogen(L1,Y, M0(Y),S0(Y),maxZ,mindif,minsup)
12. n=2
13. While Nn-1 ≠ ∅
14. {
15.
Cn = Nn-1 x L1
//N1: Large itemsets checked for homogeneity (maxZ)
16.
Cn = Prune (Cn, Ln-1)
17.
Ln  LargeGen(Cn)
//find the first large itemsets array
18.
S  S U RuleGen(D,Ln,X, Y, Mn(Y), M0(Y),S0(Y), maxZ, mindif, minsup)
19.
Nn  CheckHomogen(Ln,Y, M0(Y),S0(Y),maxZ,mindif,minsup)
20.
n = n +1
21. }
22. OutputRules (S, toleratePercent)
4. Experiments
The database that was used for the test is a financial database with bank transactions and
was previously used in the PKDD 1999 Challenge. Our dataset D1 is a profile of the 4.500
accounts in the database and contains information on an account level about properties of
the account, age of the account owner and population of the city where the owner lives. It
also contains summarized information on the transactions made through this account, such
as the summary of insurance payments, interest payments and the average balance, without
containing information on a transaction level, such as transaction date or amount.
The rules produced by the algorithm strengthen the theoretical framework presented in
this study. The following aspects are of a higher scientific interest:
1. The homogeneity check as a combination measure of the itemsets is efficient.
2. The different categorization of the quantitative independent variables is necessary for
multivariate analysis.
Fig. 3. Homogeneity check as a combination measure
956
No rule for: sex = 'M' => amount_loan Mean: -11993,3078 equal to 0 = 0 with
mean -12278,2896 Z-Test: 0,2422
Itemset sex = 'M' - Variable amount_loan - Set Size 225 with maxZ: -36,3612
will be researched further
No rule for: d_region = 'west Bohemia' => amount_loan Mean: -10537,1846
equal to 0 = 0 with mean -12278,2896 Z-Test: 0,8691
Itemset d_region = 'west Bohemia' - Variable amount_loan - Set Size 225 with
maxZ: -2,9408 will be researched further
Rule: sex = 'M' AND d_region = 'west Bohemia' => amount_loan Mean: 6041,0966 different from sex = 'M' with mean -11993,3078 Z-Test: 2,1902
Itemset sex = 'M' AND d_region = 'west Bohemia' - Variable amount_loan Set Size 225 with maxZ: -0,0729 will not be researched further
5. Conclusions
The current algorithm provides a framework for a unified association rules multivariate
analysis of any type of variable, either categorical or quantitative. However there has not
been done any work in optimizing the algorithm’s execution time. In previous studies many
methods have been proposed for optimization of the Apriori algorithm2 . However due to
the change in the nature of the algorithm it is not possible to use the optimizations proposed
for the Apriori algorithm without changes. Also, much of the execution time is spent on the
Window procedure of the categorization of the independent quantitative variables, especially if the number of quantitative variables on the right-hand side is large or if the categorical variables of the right-hand side have many distinct values and therefore produce a
large number of Boolean dependent variables. We combated this inefficiency by splitting
one execution with more variables on each side into smaller executions with less variables
on each side. The algorithm has been tested with datasets of up to 1 million transactions
which is generally considered a small dataset. Performance in larger datasets may be poor
and unacceptable especially if we include the categorization step. Future work has to be
done on improving Angel’s performance.
References
[AIS93] Rakesh Agrawal, Tomasz Imielinski, Arun Swami. Mining association rules between sets of
items in large databases. Proceedings of the 1993 ACM SIGMOD international conference on
Management of data, Pages: 207 - 216, Washington, D.C., United States 1993
2
See [PYC97] for adjusting accuracy to increase performance, [FMMT96] for using parallel bucketing and sampling.
957
[AS94] Rakesh Agrawal, Ramakrishnan Srikant. Fast Algorithms for Mining Association Rules in
Large Databases. International Conference Very Large Data Bases (VLDB), pages 487-499,
Santiago, Chile Sept. 1994
[AUL03] Yonatan Aumann, Yehuda Lindell. A Statistical Theory for Quantitative Association Rules.
Journal of Intelligent Information Systems, Volume 20 , Issue 3, Pages: 255 - 283, May 2003
[LFH00] Hongjun Lu, Ling Feng, Jiawei Han. Beyond intratransaction association analysis: mining
multidimensional intertransaction association rules. ACM Transactions on Information Systems
(TOIS), Volume 18 , Issue 4 , Pages: 423 - 454, ACM Press New York, NY, USA October 2000
[SA96] Ramakrishnan Srikant, Rakesh Agrawal. Mining quantitative association rules in large
relational tables. Proceedings of the 1996 ACM SIGMOD international conference on
Management of data, Pages: 1 - 12, Montreal, Quebec, Canada 1996