advertisement

On Characterization and Discovery of Minimal Unexpected Patterns in Data Mining Applications Balaji Padmanabhan Operations and Information Management Department The Wharton School, University of Pennsylvania http://www.wharton.upenn.edu/faculty/padmanabhan.html balaji@wharton.upenn.edu Alexander Tuzhilin Information Systems Department Stern School of Business, New York University http://www.stern.nyu.edu/~atuzhili/ atuzhili@stern.nyu.edu On Characterization and Discovery of Minimal Unexpected Patterns in Data Mining Applications Abstract A drawback of traditional data mining methods is that they do not leverage prior knowledge of users. In many business settings, managers and analysts have significant intuition based on several years of experience. In prior work we proposed a method that could discover unexpected patterns in data by using this domain knowledge in a systematic manner. In this paper we continue our focus on discovering unexpected patterns and propose new methods for discovering a minimal set of unexpected patterns that discover orders of magnitude fewer patterns and yet retain most of the truly unexpected ones. We demonstrate the strengths of this approach experimentally using a case study application in a marketing domain. Keywords: data mining, databases, rule discovery, association rules, unexpectedness, interestingness, minimality. 1. Introduction A well-known criticism of many rule discovery algorithms in data mining is that they generate too many patterns, many of which are obvious or irrelevant [PSM94, ST95, BMU+97, S97, BA99]. It stands to reason that more effective methods are needed to discover fewer and more relevant patterns from data. One way to approach this problem is by focusing on discovering unexpected patterns [ST95, ST96, LH96, LHC97, Suz97, CSD98, Sub98, BT98, PT98, PT99, P99], where unexpectedness of discovered patterns is usually defined relative to a system of prior expectations. In particular, we proposed in our prior research [PT98, PT99, P99] a characterization of unexpectedness based on logical contradiction of a discovered pattern and prior beliefs and presented new algorithms for discovering such unexpected patterns. As demonstrated in [PT98, PT99, P99], this approach generated far fewer and more interesting patterns than traditional approaches. In this paper we extend our prior work on discovering unexpected patterns [PT98, PT99] and propose a new approach that further reduces the number of such patterns in a significant way, yet retaining most of the truly interesting patterns. This is achieved through defining a minimal set of unexpected patterns as those unexpected patterns that are not refinements of other unexpected patterns and hence cannot be monotonically inferred from other unexpected patterns. Moreover, we present efficient algorithms that discover the minimal set of unexpected patterns and test these algorithms on “real-world” data to see how well they perform in practice. In the context of discovering a minimal set of patterns in data mining, [BA99, LHM99, SLR+99, 1 TKR+95, SA96, BAG99] also provide alternate approaches to characterizing this concept. In Section 3 these approaches are described and contrasted to our method of discovering minimal sets of unexpected patterns. The power of the approach presented in this paper lies in combining two independent concepts of unexpectedness and minimality of a set of patterns into one integrated concept that provides for the discovery of small but important sets of interesting patterns. Moreover, our proposed methods are efficient in the sense that they focus directly on discovering minimal unexpected patterns rather than using any of the postprocessing approaches, such as filtering, to determine the minimal unexpected patterns from the set of all the discovered patterns. Further, the approach presented in this paper is effective in supporting decision making in real business applications where unexpected patterns with respect to managerial intuition can be of great value. The rest of this paper is organized as follows. In Section 2 we present an overview of the concept of unexpectedness and provide a summary of the relevant related work. We then present in Section 3 related work to the concept of minimality of a set of rules and definitions and formal characterizations of the minimal set of patterns and the minimal set of unexpected patterns. In Section 4 we first present naïve and semi-naïve approaches to discovering the minimal set of unexpected patterns; we then present MinZoomUR, the algorithm for discovering the minimal set of unexpected patterns. Experimental results are presented in Section 5 followed by conclusions in Section 6. 2. Overview of Unexpectedness Unexpectedness of patterns has been studied in [ST95, ST96, LH96, LHC97, Suz97, CSD98, Sub98, BT98, PT98, PT99, P99] and a comparison of these approaches is provided in [P99]. In this paper we follow our previous approach to unexpectedness presented in [PT98, PT99, P99] because it is simple and intuitive (as it is based on a logical contradiction between a discovered pattern and a belief) and lends itself to efficient algorithms that, as demonstrated in [PT98, P99], discover interesting patterns in various applications. Moreover, this paper builds on our previous work on unexpectedness [PT98, PT99] by proposing new methods that significantly improve the effectiveness of the methods described in [PT98, PT99]. To make the paper self-contained, we present an overview of our previous approach to unexpectedness from [PT98, PT99]. To define unexpectedness, we start with a set of beliefs that represent knowledge about the domain and use these beliefs to seed the search for all unexpected patterns defined as rules. In particular, let I = {i1, i2, …, im} be a set of discrete attributes (also called “items” [AIS93]), some of them being ordered and others unordered. Let D = {T1, T2, ..., TN} be a relation consisting on N transactions [AMS+95] T1, T2, ..., TN 2 over the relation schema {i1, i2, …, im}. Also, let an atomic condition be a proposition of the form value1 ≤ attribute ≤ value2 for ordered attributes and attribute = value for unordered attributes where value, value1, value2 belong to the finite set of discrete values taken by attribute in D. Finally, an itemset is a conjunction of atomic conditions. Then we assume that rules and beliefs are defined as extended association rules of the form X → A, where X is the conjunction of atomic conditions (an itemset) and A is an atomic condition. As defined in [AIS93], the rule has confidence c if c% of the transactions in D that contain X also contain A and the rule has support s in D if s% of the transactions in D contain both X and A. Finally, a rule is said to hold on a dataset D if the confidence of the rule is greater than a user-specified threshold value chosen to be any value greater than 0.5. Given these preliminaries, we define unexpectedness as follows. 1 Definition [PT98, P99] . The rule A → B is unexpected with respect to the belief X → Y on the dataset D if the following conditions hold: (a) B AND Y |= FALSE. This condition imposes the constraint that B and Y logically contradict each other. 2 (b) A AND X holds on a statistically large subset of tuples in D . We use the term “intersection of a rule with respect to a belief” to refer to this subset. This intersection defines the subset of tuples in D in which the belief and the rule are both “applicable” in the sense that the antecedents of the belief and the rule are both true on all the tuples in this subset. (c) The rule A, X → B holds (same level of threshold support and confidence). Since condition (a) constrains B and Y to logically contradict each other, the rule A, X → Y does not hold. A key assumption in this definition is that of the monotonicity of beliefs that is motivated in [PT98, P99]. In particular, if we have a belief Y → B that we expect to hold on a dataset D, then monotonicity assumes the belief should also be expected to hold on any statistically large subset of D. The above definition of unexpectedness states that the rule and the belief logically contradict each other, that the rule should hold on its intersection with the belief, and this intersection is supposed to be 1 We would like to point out that this definition is applicable not only to the specific structure of the rules defined above, but also to a broader set of rules. Moreover, this observation also holds for the definitions of the minimal set of rules and monotonicity inference introduced in Section 3. However, the discovery algorithms presented in this paper are designed for the rules having this specific structure introduced above. 2 One of the ways to define “large subset of tuples” is through the user-specified support threshold value. 3 statistically large. This condition is “unexpected” because, according to the monotonicity assumption, it is expected that the belief should hold on the subset of data defined by the belief’s intersection with the rule. Given the definition of unexpectedness, [PT98, P99] propose algorithm ZoomUR that discovers all the unexpected rules with respect to a set of beliefs that satisfy user-specified minimum support and confidence requirements. ZoomUR consists of algorithms for two phases of the discovery strategy - ZoominUR and ZoomoutUR. In the first phase of ZoomUR, ZoominUR discovers all unexpected patterns that are refinements to any belief. More specifically, given any belief X → Y, ZoominUR discovers all unexpected rules of the form X, A → B such that B AND Y |= FALSE. As done in the Apriori algorithm [AMS+95], ZoominUR first generates 3 “large” itemsets incrementally and then generates rules from the discovered itemsets. As originally proposed in Apriori [AMS+95], ZoominUR also uses the observation that subsets of large itemsets should also be large to limit the search for large itemsets. However, unlike Apriori, which starts from an empty set, ZoominUR starts with an initial set of candidate itemsets that are derived from beliefs such that every candidate itemset considered by ZoominUR must contain the body of the belief and an atomic condition that contradicts the head of the belief. In the second phase of ZoomUR, starting from all the unexpected rules that are refinements to a belief, ZoomoutUR discovers more general rules (generalizations) that are also unexpected. Specifically from each unexpected refinement of the form X, A → B ZoomoutUR discovers all the unexpected rules of the form X’, A → B where X’ ⊂ X. The rules that ZoomoutUR discovers are not refinements of beliefs, but more general rules that satisfy the conditions of unexpectedness as defined above. For example, if a belief is that “professional → weekend” (professionals tend to shop more on weekends than on weekdays), ZoominUR may discover a refinement such as “professional, december → weekday” (in December, professionals tend to shop more on weekdays than on weekends). ZoomoutUR may then discover a more general rule “december → weekday”, which is totally different from the initial belief “professional → weekend”. 4 Though ZoomUR discovers only the unexpected rules and also far fewer rules than Apriori , it still discovers large numbers of rules many of which are redundant in the sense that they can be obtained from other discovered rules. For example, given the belief diaper → beer and two unexpected patterns diaper, weekday → not_beer and diaper, weekday, male → not_beer the second unexpected pattern can be inferred from the 3 4 An itemset is said to be large if the percentage of transactions that contain it exceed a user-specified minimum support level. This is not surprising - the objective of Apriori is to discover all strong rules, while ZoomUR discovers only unexpected rules. 4 first one under the monotonicity assumption. Therefore, to improve the discovery process, we introduce in this paper the concept of a minimal set of unexpected patterns and present efficient algorithms that discover this set of rules. 3. Minimal Set of Patterns The notion of minimality is very broad and its studies go way back to the ancient times, including the work of Occam and his Occam’s razor. Since in this paper we focus on discovery of unexpected patterns, we will focus only on minimality as related to this problem. One of the early influential works related to minimality of a set of patterns was presented by Mitchell in [M82]. In particular, [M82] presents a unifying approach to the problem of generalizing knowledge (that, for example, can be represented as rules) by viewing generalization as a search problem. Moreover, based on the specific search methods used, Mitchell [M82] also categorizes several rule learning systems [BM78, P70, W75, HRM7, V78, M77, MUB82] that deal with generalization. In particular, [M82] deals with a broader set of objects (that can also include rules) and formulates the generalization problem as follows. Given a set of instances specified in an instance language, the generalization problem is formulated in [M82] as a search for descriptions in a generalization language such that these generalizations are consistent with a set of training examples that are labeled with the appropriate generalization. [M82] adopts a strong notion of consistency by defining a generalization to be consistent with training examples if it matches all the positive examples and does not match any negative examples. Moreover, [M82] introduces a partial order for generalizations (one generalization being more general than another one) and defines minimally specific generalizations based on this partial order. At the extremes of this partial order are the G-set (the most general set) and the S-set (the most specific set). Although [M82] deals with abstract objects, Mitchell’s approach is also applicable to rules and their generalizations. In the context of discovering a minimal set of rules in data mining, the approach presented in [M82] has the following two limitations. First, in most cases it may not be possible to have training examples (a set of discovered rules) that are classified into known generalizations. Therefore, rather than learning these generalization relationship among different objects, it is necessary to define them. Recent characterizations of various notions of minimality in the knowledge discovery literature take this approach [BA99, LHM99, SLR+99], and we will describe them shortly. Second, the concept of consistency in [M82] is too strong to eliminate rules, typically flagged as uninteresting by users. For example, requiring that all refinements of a discovered rule diaper → beer also be discovered in order to characterize the rule as minimal is too strong since there may be several rules of the form diaper, X → beer that do not have adequate support or 5 confidence. In practice it is rarely the case that all possible refinements to a discovered rule are also discovered. Hence we need a weaker notion of consistency than the one proposed in [M82]. To address these limitations, [BA99, LHM99, SLR+99, TKR+95, SA96, BAG99] provide alternate approaches to characterizing a minimal set of discovered rules. In particular, [BA99] presents an approach that finds the “most interesting rules”, defined as rules that lie on a support and confidence frontier. Further [BA99] proves that these rules necessarily contain the strongest rules discovered using several objective criteria other than just confidence and support. In [SLR+99] several heuristics for pruning large numbers of association rules have been proposed. One of these heuristics prunes out certain refinements of rules, thus, alluding to the concept of minimality of a set of rules. However, [SLR+99] focuses on the heuristics that prune redundant rules from a discovered set of rules and does not explore the concept of minimality fully, nor proposes any algorithms for discovering a minimal set of patterns. In [LHM99] a technique is presented to prune and then summarize an already discovered set of association rules. In particular, [LHM99] defines the concept of direction-setting rules and demonstrates how nondirection-setting rules can be inferred from them. Therefore, the set of direction-setting rules constitutes a set of rules that are “minimal” in some sense. This work is related to [SLR+99] in the sense that certain rule refinements are pruned out in the [LHM99] approach and therefore, they are not direction-setting. However, the approach presented in [LHM99] is different from [SLR+99] and from our approach in the sense that not all refined rules are non-direction setting according to [LHM99]. Moreover, [LHM99] focuses on pruning already discovered rules and does not address the issue of direct discovery of minimal sets. An approach to eliminating redundant association rules is presented in [TKR+95]. In particular, [TKR+95] introduces a concept of the “structural cover” for association rules and presents post-processing algorithms to find the structural cover. In this paper we, present an alternative formal characterization of the minimal set of patterns that corresponds to structural covers of [TKR+95] for the association rules but is also broader and applicable to more general classes of rules. Moreover, [TKR+95] focuses on pruning already discovered rules and does not address the issue of direct discovery of minimal sets. Finally, the work of [SA96] and [BAG99] is also related to the problem of discovering minimal sets of rules. In particular, [SA96] and [BAG99] provide methods for eliminating rules such that the support and/or confidence values of these rules are not unexpected with respect to the support and confidence values of previously discovered rules. However, this work is only marginally related to our approach because we focus 6 on a more general definition of minimality that does not directly depend on confidence and support of discovered rules. In this section we present our approach to minimality that is targeted to the discovery of unexpected patterns. Since the concept of unexpectedness of discovered patterns is based on the monotonicity assumption about the set of underlying beliefs, and since the monotonicity assumption is strongly linked to the concept of refinement of patterns, our concept of minimality is based on refinement of patterns. Therefore, we do not consider other kinds of minimality, such as direction-setting rules [LHM99] or SC-optimality [BA99]. In the rest of this section we formally define minimality of a set of patterns and the minimal set of unexpected patterns. However, before defining these concepts, we introduce some preliminary definitions in the next section. 3.1 Inference Under Monotonicity Assumption Before introducing minimal rules, we need to define formally which rules can be inferred to hold on a dataset due to the monotonicity assumption. Definition. Let X and Y be two itemsets. Then itemset Y is a generalized refinement of itemset X (denoted as Y = genref (X) ) if for all atomic conditions a from X (a) if a is of the form attribute = value then a ∈ Y. (b) if a is of the form value1 ≤ attribute ≤ value2 then Y contains an atomic condition value1 + δ ≤ attribute ≤ value2 - ε for some non-negative values of δ and ε. Lemma 3.1. Y is a generalized refinement of X if and only if there exists itemset Z such that Y = X AND Z. Sketch of the Proof. For unordered attributes this observation is trivial. For the ordered attributes from X having the form value1 ≤ attribute ≤ value2, Z contains an atomic condition from Y of the form value1 + δ ≤ attribute ≤ value2 - ε. Proposition 3.2. If X → Y holds on D and Z is a generalized refinement of X, then under the monotonicity assumption, Z → Y holds on D. Proof. According to Lemma 3.1, there exists itemset W such that Z = X AND W. Let D′ = { t ∈ D | Z = X AND W holds on t }. Then the monotonicity assumption states that X → Y should hold on D′ and hence X, W → Y holds on D. 7 The above proposition states that rule X, W → Y can be inferred to hold on D under monotonicity assuming that rule X → Y holds. However, this inference is applicable only to the rules having the specific structure that was defined in Section 2 and considered throughout this paper (because the definition of generalized refinement explicitly assumes this rule structure). We will provide an alternative characterization of generalized refinement below that can define inference under the monotonicity assumption in more general terms. The next proposition provides an alternative characterization of generalized refinement that will be used subsequently. Proposition 3.3. Itemset C is a generalized refinement of itemset A if and only if C |= A. Proof. The necessary condition immediately follows from Lemma 3.1. To prove the sufficient condition, assume that C |= A. If a is a condition from A involving an unordered attribute of the form attribute = value and a does not belong to C, then we can find an interpretation I such that attribute = value1 ≠ value and C is true in I. In this case, A is false in I, thus producing the contradiction. If a is a condition from A involving an ordered attribute of the form value1 ≤ attribute ≤ value2, and C does not contain any condition of the form value1 + δ ≤ attribute ≤ value2 - ε then we can find an interpretation I such that the value of attribute does not belong to the interval [value1, value2] and C is true for I. In this case, A is false in I, thus also producing the contradiction. We next present a definition and later show that it captures the concept of inference under the monotonicity assumption in more general terms than was done using Proposition 3.2. Definition. Rule (A → B) |=M (C → D) if 1. C |= A, and 2. D = B. Note that we defined relationship |=M in terms of logical implication. However, Proposition 3.3 provides an alternative characterization of |=M in terms of generalized refinements. Theorem 3.4. If X → Y holds on dataset D and X → Y |=M Z → V, then under the monotonicity assumption Z → V holds on D. 8 Proof. By definition of |=M, V = Y and Z |= X. Then from Proposition 3.3, we conclude that Z is a generalized refinement of X. By Proposition 3.2, Z → Y holds on D. This theorem demonstrates that the relational operator |=M defines inference of one rule from another under the monotonicity assumption. Moreover, this theorem gives a broader characterization of this inference than the one implied by Proposition 3.2 because |=M is defined in terms of logical implication as opposed to generalized refinement and hence can be applied to a broader class of rules than the ones introduced in Section 2. Example. Assume the rule diaper, weekday → not_beer holds on a dataset D. Consider the rule diaper, weekday, male → not_beer. Since diaper, weekday, male |= diaper, weekday it follows that diaper, weekday → not_beer |=M diaper, weekday, male → not_beer. Therefore according to Theorem 3.4, diaper, weekday, male → not_beer should also hold. Also notice that the itemset diaper, weekday, male is a generalized refinement of the itemset diaper, weekday. Proposition 3.5. The relation |=M is reflexive, transitive and not symmetric. Proof. The proof follows from the observation that the logical implication relation |= is reflexive and transitive but not symmetric. The following proposition establishes the relationship between |=M and classical logical inference relation |=. Proposition 3.6. If x |=M y then it has to be the case that x |= y, but if x |= y it does not follow that x |=M y (i.e. classical logical inference relation is a necessary but not sufficient condition for two rules to be related by |=M). Proof. The “necessary” part (if x |=M y then it has to be the case that x |= y) trivially holds. For the second part of the proof we provide a counter-example of a case where a rule logically follows from another rule, but where the relationship is not because of monotonicity assumption. A → B > 10 |= A → B > 9. However it follows from the definition that A → B > 10 |≠M A → B > 9. Notice that the inference relation |=M deals with rules that hold on data probabilistically, whereas logical implication |= deals with interpretations of logical formulas and does not directly deal with probabilistic relationships usually encountered in data mining. Therefore, we introduced the relationship |=M that deals with the type of probabilistic inference captured by the monotonicity assumption. 9 Using these definitions and results, we next present the definitions for the minimal set of rules and the minimal set of unexpected patterns. 3.2 Minimal Set of Rules In order to define the minimal set of (unexpected) rules, we first introduce a partial ordering relationship on a set of rules. Definition. The set of rules Y covers the set of rules X (denoted as Y ≺ X) if Y ⊆ X and ∀ xi ∈ X, ∃ yi ∈ Y such that yi |=M xi Definition. Let X and Y be sets of rules. Then Y is the minimal set of X if Y covers X and there is no set of rules Z that covers Y. The following proposition establishes an alternative characterization of a minimal set of rules. Proposition 3.7. Y is the minimal set of X if and only if the following conditions hold: (1) Y ⊆ X. (2) ∀ xi ∈ X, ∃ yi ∈ Y such that yi |=M xi. (3) ∀ y1 , y2 ∈ Y, y1 |≠M y2. Proof. Immediately follows from the definition of the minimal set of rules. The next proposition establishes uniqueness of the minimal set of rules. Proposition 3.8. For any set of rules X, the minimal set of X is unique. Proof. To prove this proposition, we define a directed graph G=(V, E) as follows. The set of nodes V consists of all the rules from X. Given two nodes n1 = A → B and n2 = C → D from V, there is an edge from node n1 to node n2 in E if A → B |=M C → D. Then it is easy to see that the minimal set of rules for X consists of all the nodes of G having no incoming edges (in-degrees of these nodes are 0). Since such set of nodes in G is unique, it follows that the minimal set of rules for X is also unique. Given the definition of minimality introduced above, we next define the minimal set of unexpected patterns. Definition. If B is a belief and X is the set of all unexpected patterns with respect to B, the minimal set of unexpected patterns with respect to B is the minimal set of X. 10 Example: For the belief diaper → beer let the set of all unexpected patterns be {diaper and weekday → not_beer, diaper and unemployed → not_beer, diaper and weekday and unemployed → not_beer, weekday → not_beer, unemployed → not_beer, weekday and unemployed → not_beer}. The minimal set of unexpected patterns in this case is {weekday → not_beer, unemployed → not_beer}. In this section we discussed minimality of a set of patterns and presented a definition for the minimal set of unexpected patterns. In the next section we present algorithms for discovering the minimal set of unexpected patterns. 4. Discovering the Minimal Set of Unexpected Patterns In this section we present three algorithms for discovering the minimal set of unexpected patterns. We first present, in section 4.1, a naive algorithm that discovers such a set of patterns. In section 4.3 we present an efficient algorithm for discovering the minimal set of unexpected patterns. However before presenting this, we describe in Section 4.2 a semi-naive algorithm that discovers only the minimal set of unexpected refinements. We describe this semi-naive algorithm because in many applications we are interested only in refinements (hence the algorithm is important in its own right) and also because the semi-naive algorithm illustrates some important points used in the algorithm presented in Section 4.3. The inputs to all the algorithms presented in this section are the same and constitute: 1. A set of beliefs, B. 2. The dataset D. 3. Minimum support and confidence values minsup and minconf. 4. Minimum and maximum width for all ordered attributes. Regarding input (4), in the case of ordered attributes the width of any condition of the form value1 ≤ attribute ≤ value2 is defined to be value2 - value1. We take as user inputs the minimum and maximum width for all ordered attributes. This is necessary and useful for the following reason. Assume that age is defined to be an ordered attribute and takes values ranging from 1 to 100 in the dataset. Clearly at the extreme a rule involving a condition of the form 1 ≤ age ≤ 100 is not useful since the condition 1 ≤ age ≤ 100 will hold for every record in the dataset. Extending this argument, larger ranges of age may hold for most records in the dataset, hence we allow the user to specify the maximum width for age that the user may be interested in considering. Similarly the user may not be interested in too small a range for ordered attributes and we allow the user to specify a minimum width for the attribute. Note that this is not a restrictive assumption in any way 11 since the default can be the smallest width and largest width respectively for these two parameters. However, the specification of this condition improves the efficiency of the algorithm. We now present the first algorithm, FilterMinZoomUR. 4.1 Algorithm FilterMinZoomUR FilterMinZoomUR is a post-processing method that discovers the minimal set of unexpected patterns by filtering patterns from the set of all unexpected patterns. This algorithm operates in two phases. In the first phase, Algorithm ZoomUR (outlined in Section 2) is applied to a set of beliefs to discover all unexpected patterns. For this set, in the second phase, each unexpected pattern is compared to the rest of the unexpected patterns and dropped from consideration if there exists another pattern from that set, from which it can be inferred under the monotonicity assumption. This strategy is equivalent to applying a minimal filter to the set of patterns discovered by ZoomUR to select the minimal set for the set of unexpected patterns. Algorithm FilterMinZoomUR is presented in Figure 4.1. For each belief, B, step 2 applies ZoomUR to generate the set of all unexpected patterns, Unexp(B). For each unexpected pattern, X, in Unexp(B) steps 4 through 9 adds X to the set of minimal patterns if it cannot be inferred under monotonicity from any other unexpected pattern. FilterMinZoomUR is a naïve algorithm in two senses: (1) it is a post-processing method that relies on ZoomUR to provide the set of all unexpected patterns and (2) each unexpected pattern is compared to the rest of the patterns in the process of selecting the minimal set of unexpected patterns. Inputs: Beliefs Bel_Set, Dataset D, minwidth and maxwidth for all ordered attributes and minimum support min_sup and minimum confidence min_conf Outputs: For each belief, B, MinUnexp(B) 1 forall beliefs B ∈ Bel_Set { 2 Unexp(B)= ZoomUR(Inputs) 3 MinUnexp(B) = {} 4 forall x ∈ Unexp(B) { 5 Other_unexp = Unexp(B) - x if not(∃ y ∈ Other_unexp such that y |=M x) { 6 7 MinUnexp(B) = MinUnexp(B) ∪ {x} 8 } 9 } 12 10 } Figure 4.1 Algorithm FilterMinZoomUR 5 Below we state and prove the completeness of FilterMinZoomUR. Theorem 4.1. For any belief, B, FilterMinZoomUR discovers the minimal set of unexpected rules. Proof. To prove that MinUnexp(B) is a minimal set of Unexp(B) we will show that all three conditions listed in the definition of the minimal set presented in Proposition 3.7 hold. The first and third conditions are trivially satisfied. To satisfy the second condition, it needs to be shown that for any X ∈ Unexp(B) there exists Y ∈ MinUnexp(B) such that Y |=M X. For any X ∈ Unexp(B) consider the two cases that arise from Steps 4 through 9 of FilterMinZoomUR: 1. X is added to the set of minimal patterns. Due to the reflexivity of |=M it is easily seen that there exists Y=X ∈ MinUnexp(B) such that Y |=M X. 2. X is not added to the set of minimal patterns since there exists a pattern P in Unexp(B) - X such that P |=M X. In this case, observing that |=M is transitive and Unexp(B) is finite, Steps 4 through 9 is applied iteratively and adds some Q to MinUnexp(B) where Q |=M P. Since P |=M X, it follows that Q |=M X. Hence there exists Y ∈ MinUnexp(B) such that Y |=M X. The third condition of the definition holds trivially since steps 7 and 8 of FilterMinZoomUR add a pattern X to MinUnexp(B) only if there exists no other pattern, Y, in Unexp(B) such that Y |=M X. One of the main sources of inefficiencies of this algorithm is that we first have to generate Unexp(B) and then filter the non-minimal patterns out of it. In the next two sections we present more efficient algorithms that avoid generating many unexpected patterns that may be known to be non-minimal. 4.2 Algorithm MinZoominUR In this section we present MinZoominUR, an algorithm for discovering the minimal set of unexpected refinements to a set of beliefs. Consider the belief body → head, having the structure specified in Section 2. We use the term CONTR(head) 5 The inputs to algorithms presented in this section are the same as the inputs to ZoomUR [PT98] described in Section 2. 13 to refer to the set of atomic conditions that contradict the atomic condition specified by head. Assume that v1, v2,...,vk are the set of unique values (sorted in ascending order if the attribute a is ordered) that a takes on in D. CONTR(head) is generated as follows: (1) If the head of the belief is of the form value1 ≤ attribute ≤ value2 (attribute is ordered), then condition value3 ≤ attribute ≤ value4 belongs to CONTR(head) if the ranges [value1, value2] and [value3, value4] do not overlap. (2) If the head of the belief is of the form attribute = val (attribute is unordered), then condition attribute = vp belongs to CONTR(head) if vp ∈ { v1, v2,...,vk } and vp ≠ val; Algorithm MinZoominUR is based on the Apriori algorithm [AMS+95] with several major differences. First, generation of large itemsets starts with a set of beliefs that seed the search. Second, MinZoominUR does not generate those itemsets that are guaranteed to produce non-minimal rules. Third, rule generation process is integrated into the itemset generation part of the algorithm – this process is immaterial for Apriori but results in significant efficiency improvements for MinZoominUR. Before presenting MinZoominUR, we first present a broad overview of the algorithm. Each iteration of MinZoominUR generates itemsets in the following manner. In the k-th iteration we generate itemsets of the form {C,body,P}, where C ∈ CONTR(head) and P is a conjunction of k atomic conditions. Observe that to determine the confidence of the rule body, P → C, the supports of both the itemsets {C,body,P} and {body,P} will have to be determined. Hence in the k-th iteration of generating large itemsets, two sets of candidate itemsets are considered for support determination: (1) The set Ck of candidate itemsets. Each itemset in Ck (e.g. {C,body,P}) contains (i) a condition that contradicts the head of belief, (i.e. any C ∈ CONTR(head)), (ii) the body {body} of the belief, and (iii) k other atomic conditions (P is a conjunction of k atomic conditions). (2) A set Ck' of additional candidates. Each itemset in Ck' (e.g. {body,P}) is generated from an itemset in Ck by dropping the condition, C, that contradicts the head of the belief. In each iteration, minimal unexpected rules are generated from the set of large itemsets. If an itemset generates an unexpected rule, it is deleted from consideration and therefore no superset of this itemset is even considered in subsequent iterations. As we prove in Theorem 4.2, this step avoids generation of itemsets producing non-minimal rules and significantly improves the efficiency of the algorithm. We explain the steps of MinZoominUR in Fig. 4.2 now. The following is a list of notations that are used in 14 describing the algorithm: • UNORD is the set of unordered attributes. • ORD is the set of ordered attributes. • minwidth(a) and maxwidth(a) are minimum and maximum widths for ordered attribute a. • Attributes(x) is the set of all attributes present in any of the conditions in itemset x. • Values(a) is the set of distinct values the attribute a takes in the dataset D. First, given a belief, B, the set of atomic conditions that contradict the head of the belief, CONTR(head(B)), is computed (as described previously). Then, the first candidate itemsets generated in C0 (step 3) will each contain the body of the belief and a condition from CONTR(head(B)). Hence the cardinality of the set C0 is the same as the cardinality of the set CONTR(head(B)). Inputs: Beliefs Bel_Set, Dataset D, minwidth and maxwidth for all ordered attributes ORD and thresholds min_support and min_conf Outputs: For each belief, B, MinUnexp(B) 1 forall beliefs B ∈ Bel_Set { 2 MinUnexp(B) = {} 3 C0 = { {x,body(B)} | x ∈ CONTR(head(B)) }; 4 C0’ = {{body(B)}}; 5 k=0 6 while (Ck != ∅ ) do { 7 8 9 10 forall c ∈ Ck ∪ Ck’, compute support(c) Lk = {x| x ∈ Ck, support(x) ≥ min_support } Lk’ = {x| x ∈ Ck’, support(x) ≥ min_support} forall (x ∈ Lk) { Let a = x ∩ CONTR(head(B)) /* this intersection is a single 11 element */ 12 rule_conf = support(x)/support(x-a) 13 if (rule_conf > min_conf) { MinUnexp(B) = MinUnexp(B) ∪ {x – a →a} 14 15 Lk = Lk - x 16 17 } } 18 k++ 19 Ck = generate_new_candidates(Lk-1, B) 15 Ck’ = generate_bodies(Ck , B) 20 21 } 22 forall x ∈ MinUnexp(B) { 23 24 Other_unexp = MinUnexp(B)-x if (∃ y ∈ Other_unexp such that y |=M x) { 25 MinUnexp(B) = MinUnexp(B) - {x} 26 27 } } 28 } Figure 4.2 Algorithm MinZoominUR To illustrate this, consider an example involving only binary attributes. For the belief x=0 → y=0, the set CONTR({y=0}) consists of a single condition {y=1}. The initial candidate sets, therefore, are C0 = {{y=1, x=0}}, C0' = {{x=0}}. Steps (6) through (20) in Fig. 4.2 are iterative: Steps 7 through 9 determine the supports in dataset D for all the candidate itemsets currently being considered and selects the large itemsets Lk and Lk’. Each itemset in Lk contains the body and the head of a potentially unexpected rule, while each itemset in Lk’ contains only the body of the potentially unexpected rule. Steps 10 through 17 generate unexpected rules such that large itemsets that contribute to unexpected rules are subsequently deleted in Step 15. Specifically, for each large itemset in Lk, if the unexpected refinement rule that is generated from the itemset has sufficient confidence, then two actions are performed: 1. Step 14 adds this rule to the set of potentially minimal unexpected refinements. 2. Step 15 deletes the corresponding itemset from Lk since any itemset that is a superset of this itemset can only generate unexpected refinements that can be monotonically inferred from the new rule generated in step 14. Theorem 4.2 below has a detailed proof. In step (19), function generate_new_candidates(Lk-1, B) generates the set Ck of new candidate itemsets to be considered in the next pass from the previously determined set of large itemsets, Lk-1, with respect to the belief B (“x → y”) in the following manner: 16 (A) Initial condition (k=1): In the example (involving binary attributes) considered above, assume that L0 = {{x=0, y=1},{x=0}}, i.e. both the initial candidates had adequate support. Further assume that p is the only other attribute (also binary) in the domain. The next set of candidate itemsets to be considered would be C1 = {{x=0,y=1,p=0}, {x=0,y=1,p=1}}, and C1’ = {{x=0, p=0}, {x=0, p=1}}. In general we generate C1 from L0 by adding additional conditions of the form attribute = value for unordered attributes or of the form value1 ≤ attribute ≤ value2 for ordered attributes to each of the itemsets in L0. More specifically, for a belief B, the set C1 is computed using the following rules. If itemset x ∈ L0 and x contains a condition that contradicts the head of the belief: 1. The itemset x ∪ {{a = val}} ∈ C1 if a ∈ UNORD (set of unordered attributes), val ∈ Values(a) and a ∉ Attributes(x). 2. The itemset x ∪ {{value1 ≤ a ≤ value2}} ∈ C1 if a ∉ Attributes(head(B)), a ∈ ORD (set of ordered attributes), value1 ∈ Values(a), value2 ∈ Values(a), value1 ≤ value2, and the resulting width for the attribute a should satisfy minimum and maximum width restrictions for that attribute. This process is efficient and complete because of the following reasons. 1. The attributes are assumed to have a finite number of unique discrete values in the dataset D. Only conditions involving these discrete values are considered. 2. For unordered attributes no condition involving an attribute already present in the itemset is added. This ensures that itemsets that are guaranteed to have zero support are never considered. For example, this condition ensures that for the belief month=9 → sales=low, the itemset {{month = 3}} is not added to the itemset {{sales = high}, {month = 9}}. 3. For ordered attributes, however, an itemset such as {{3 ≤ a ≤ 6 }} can be added to {{b=1}, {5 ≤ a ≤ 8}} to result in {{b=1}, {5 ≤ a ≤ 6 }} where the initial belief may be 5 ≤ a ≤ 8→ b=0 for example. Without loss of generality in this case we represent the new itemset as {{b=1}, {5 ≤ a ≤ 8}, {3 ≤ a ≤ 6 }} rather than as {{b=1}, {5 ≤ a ≤ 6 }}. We use this “long form” notation since (1) we assume that all itemsets in a given iteration have the same cardinality and (2) the body of the belief is explicitly present in each itemset. (B) Incremental generation of Ck from Lk-1 when k > 1: This function is very similar to the apriori-gen function described in [AMS+95]. For example, assume that for a belief, B, "x → y", c is a condition that contradicts y and that L1 = {{c, x, p}, {c, x, q}, {x, p}, {x, q}}. Similar to the apriori-gen function, the next set of candidate itemsets that contain x and c is C2 ={{x, c, p, q}} since this is the only itemset such that all its 17 subsets of one less cardinality that contain both x and c are in L1. In general, an itemset X is in Ck if and only if for the belief B, X contains body(B) and a condition A such that A ∈ CONTR(head(B)) and all subsets of X with one less cardinality, containing A and body(B), are in Lk-1. More specifically, Ck is generated from Lk-1 using the following rule: If a ∈ CONTR(head(B)), a ∈{x1, x2,..., xp} and {x1, x2,..., xp, v}, {x1, x2,..., xp, w}∈ Lk-1 then {x1, x2,..., xp, v, w} ∈ Ck if w ∉ Attributes({x1, x2,..., xp, v}). The above rule for generating Ck essentially limits itemsets to a single condition for each “new” attribute not present in the belief B. This however does not eliminate any relevant large itemset from being generated as the following example shows. Consider the case where starting from the belief x=1 → y=0 the set of large itemsets (generated by MinZoominUR at the end of the first iteration) L1 contains {y=1, x=1, 3 ≤ a ≤ 6} and {y=1, x=1, 5 ≤ a ≤ 7}. Combining these itemsets yields the equivalent itemset {y=1, x=1, 5 ≤ a ≤ 6} of the same cardinality as any itemset in L1 and if this itemset is large, it would already be present in L1. In step (20), as described previously, we would also need the support of additional candidate itemsets in Ck' to determine the confidence of unexpected rules that will be generated. The function generate_bodies(Ck,B) generates Ck' by considering each itemset in Ck and dropping the condition that contradicts the head of the belief and adding the resulting itemset in Ck'. Steps (22 – 27) are needed to detect any remaining non-minimal rules that arise due to the following special case of certain itemsets containing unordered attributes. To illustrate this special case, consider the following two itemsets: {{a=1}, {5 ≤ b ≤ 10}} and {{a=1}, {7 ≤ b ≤ 8}}. The special case is that neither of these sets is a superset of the other, yet (5 ≤ b ≤ 10 → a=1) |=M (7 ≤ b ≤ 8 → a=1) since (7 ≤ b ≤ 8) |= (5 ≤ b ≤ 10). Therefore, the rule 7 ≤ b ≤ 8 → a=1 should be eliminated in order to produce the minimal set of unexpected rules. Since Steps (6 – 21) of the algorithm do not eliminate such rules, the additional Steps (22 – 27) do this. Observe that in the case of only unordered attributes in the itemsets, Steps (22 – 27) of the algorithm are not needed since MinUnexp(B) after Step 21 is guaranteed to be minimal (see the proof of Theorem 4.2). Moreover, notice that elimination of remaining non-minimal rules can also be done in two other parts of the algorithm instead of Steps (22 – 27). First, it can be done in generate_new_candidates procedure (Step 19) by avoiding the generation of the itemsets that are generalized refinements of itemsets that have previously 18 produced unexpected rules. Second, the elimination procedure can also be done as new rules are added to the set MinUnexp(B) in Step 14 by comparing the new rule with all the other rules in MinUnexp(B). We analyzed both of these possibilites and realized that the approach taken in Steps (22 – 27) of Figure 4.2 is at least as efficient as these two other possibilities. Moreover, another advantage of doing elimination in Steps (22 – 27) is that it can be dropped altogether in case of only unordered attributes (as explained above). The computational complexity of Steps (1 – 21) is determined by the total number of candidate itemsets K generated in Steps (19 - 20) taken over all the iterations of the While-loop. The computational complexity of the elimination procedure in Steps (22 – 27) is O(n2), where n is the size of the set MinUnexp(B). In practice K >> n2. Therefore, the bottleneck of MinZoominUR algorithm lies in Steps (6 – 21). Moreover, the complexity of MinZoominUR in the worst case is comparable to the worst-case complexity of Apriori that is bounded by O(||C|| * ||D||), where ||C|| denotes the sum of the sizes of candidates considered, and ||D|| denotes the size of the database [AMS+95]. However, in the average case, the computational complexity of MinZoominUR is significantly lower than that of Apriori. This is the case because the average number of candidates considered in MinZoominUR is significantly lower than that for Apriori due to (a) minimality-based elimination procedure, and (b) presence of the initial set of beliefs that seed the search process. In Section 5 we experimentally compare the main algorithm, ZoomUR (presented in the next section), with Apriori not in terms of itemsets but directly in terms of the number of rules generated. A key strength of MinZoominUR, compared to ZoomUR [PT99] and Apriori [AMS+95], is that rule discovery is integrated into the itemset generation procedure and hence it can greatly reduce the number of itemsets generated in subsequent iterations - as [AMS+95] show, itemsets can grow exponentially in association rule discovery algorithms. The reason this is possible is the objective of MinZoominUR - to generate only the minimal set of unexpected patterns (as opposed to generating all patterns or all unexpected patterns). Below we prove the completeness of MinZoominUR. Theorem 4.2. For any belief, B, MinZoominUR discovers the minimal set of unexpected rules that are refinements to the belief. Sketch of the Proof. We will first show that for the case where there are unordered attributes only, MinZoominUR generates the minimal set of unexpected patterns without needing to apply the minimal filter (Steps 22 through 27 of Figure 4.2). 19 For unordered attributes only, it is easy to see that a rule X1=x1, X2=x2,…, Xn=xn → Y = y1 is minimal if and 6 only if there is no rule of the form Z → Y = y1, where Z ⊂ {X1=x1, X2=x2,…, Xn=xn} . For unordered attributes only, consider the belief A=a1 → B=b1 and any minimal unexpected rule A=a1, X → B=b2 which is a refinement that holds, i.e its support and confidence values are greater than the specified threshold values, where the itemset X is a conjunction of atomic conditions involving unordered attributes. Below we will show that A=a1, X → B=b2 will be discovered by MinZoominUR. Since A=a1, X → B=b2 holds, the itemset {A=a1, X, B=b2}, and all its subsets, have adequate support. Further since the rule is assumed to hold, it is clear that Steps 13 and 14 generate the rule A=a1, X → B=b2 if the itemset {A=a1, X, B=b2} is generated. Hence it needs to be shown that the itemset {A=a1 , X, B=b2} will be generated. Before the iterations of MinZoominUR, Step 3 generates {A=a1, B=b2} in the initial set of candidates. Based on Apriori and the completeness proof of ZoomUR [P99] it is clear that the itemset {A=a1, X, 7 B=b2} will be generated unless Step 15 deleted a “parent” of this itemset. We will next show that this is impossible, i.e. Step 15 could not have deleted a parent of this itemset in any previous iteration. Consider any parent, {A=a1, Y, B=b2} of the itemset {A=a1, X, B=b2} defined such that Y ⊂ X. Assume that the itemset {A=a1, Y, B=b2} was deleted in Step 15 in some iteration. Hence it has to be the case (see Steps 13-16 of Figure 4.2) that A=a1, Y → B=b2 holds. However if Y ⊂ X and A=a1, Y → B=b2 then the rule A=a1, X → B=b2 cannot be minimal. This is a contradiction and hence no itemset of the form {A=a1, Y, B=b2} will be deleted in previous iterations. Hence the itemset {A=a1, X, B=b2} will be generated. Hence MinZoominUR will generate any minimal unexpected rule A=a1, X → B=b2 which is a refinement to the belief. Given the observation that for unordered attributes only, a rule X1=x1, X2=x2,…, Xn=xn → Y = y1 is minimal if and only if there is no rule of the form Z → Y = y1, where Z ⊂ {X1=x1, X2=x2,…, Xn=xn}, it is easy to see that MinZoominUR does not generate any non-minimal rule. Hence, for the case where there are unordered attributes only, MinZoominUR generates the minimal set of unexpected patterns without needing to apply the minimal filter. For the case of ordered attributes, it 6 Note that this “syntactic” subset property is not true when dealing with ordered attributes, which is why the minimal filter in Steps 22-27 is necessary. 7 Parent is defined formally in Section 4.3.1 20 can easily be seen that only non-minimal rules are automatically excluded from MinZoominUR. However there is a special case involving ordered attributes that cannot guarantee only minimal rules before Steps 22-27. This special case arises since a syntactic subset check cannot capture containment when dealing with ranges of values for ordered attributes. An example of this special case was given above in Section 4.2. Hence the filter in Steps 22-27 removes any non-minimal rules remaining and its clear that MinZoominUR generates only the minimal set of unexpected refinements to a belief. As mentioned earlier, MinZoominUR only discovers the minimal set of unexpected refinements to a belief. We next present MinZoomUR, an algorithm that discovers for each belief the minimal set of unexpected rules. 4.3 Algorithm MinZoomUR In this section we present MinZoomUR, an algorithm that discovers for each belief, the minimal set of unexpected rules. Before describing the algorithm we present some preliminaries. 4.3.1 Preliminaries For a belief B, let x be any large itemset containing body(B) and one condition from CONTR(head(B)). We use the term parents(x) to denote the set of all subsets of x that contain the body of the belief and one 8 condition that contradicts the head of the belief considered in previous iterations during the candidate generation phase of the algorithm. Specifically, parents(x) = { a | a ⊂ x, body(B) ⊂ a, ∃c such that c ∈ CONTR(head(B)) and c ∈ a} An itemset y is said to be a parent of x if y ∈ parents(x). We use the term zoomin rules to denote unexpected rules that are refinements of beliefs and zoomout rules for unexpected rules that are more general unexpected rules. The large itemset x is said to generate a zoomin rule if confidence (x - c → c) > min_conf, where c ∈ CONTR(head(B)). The large itemset x is said to generate a zoomout rule if x generates a zoomin rule x - c → c and confidence( x - c - d → c) > min_conf, where c ∈ CONTR(head(B)), d ⊆ body(B) and d is not empty. 8 Recall that the candidate generation phase of these algorithms (Apriori, MinZoominUR and MinZoomUR) is iterative such that itemsets in subsequent iterations have greater cardinality (number of items). 21 Associated with each itemset, x, are two attributes: x.rule, that keeps track of whether a zoomin rule is generated from x, and x.dropped_subsets, which keeps track of the subsets of body(B) that are dropped during the discovery of zoomout rules. These two attributes are further explained below. We define x.rule as follows: x.rule = 1 if x generates a zoomin rule, = 0 otherwise. Further we define parentrules(x) to be TRUE if and only if x has a parent y such that y.rule = 1. If x generates a zoomin rule, zoomout rules are generated in ZoomoutUR [PT98, P99] by dropping one or more attributes belonging to body(B) from the zoomin rule generated. This process essentially drops nonempty subsets of body(B) from the zoomin rule generated to determine the zoomout rules that are unexpected. The set x.dropped_subsets is the set of subsets of body(B) that are dropped from itemset x to generate a given set of zoomout rules. Specifically, if P is a set of zoomout rules generated from x, then x.dropped_subsets is defined as follows: x.dropped_subsets(P) = {d | { x - c - d → c} ∈ P, where c ∈ CONTR(head(B)), d ⊆ body(B), d ≠ ∅ }. Given these preliminaries, we now present MinZoomUR. 4.3.2 Overview of the Discovery Strategy Unlike what was done in MinZoominUR, an itemset that generates a zoomin rule in MinZoomUR cannot always be deleted from subsequent consideration since it is possible for minimal zoomout rules to be derived from non-minimal zoomin rules. Consider the following example. For a belief a, b → x, let a, b, c → y and a, b, c, d → y be two zoomin rules. Though a, b, c, d → y is a non-minimal zoomin rule, the rule may result in a zoomout rule such as b, c, d → y which may belong to the minimal set of unexpected rules. Extending this example one more step, we observe that the zoomout rule b, c, d → y can, however, be guaranteed to be non-minimal if the first zoomin rule a, b, c → y resulted in a zoomout rule of the form p, c → y such that b, c, d |= p, c where p is a proper subset of the body of the belief. Examples of such p are {b} and {} corresponding to the zoomout rules b, c → y and c → y respectively (generated from a, b, c → y). However if the first zoomin rule generated only the zoomout rule a, c → y, it may still be possible for the zoomout rule b, c, d → y to be minimal since b, c, d |≠ a, c. 22 The discovery strategy of MinZoomUR is based on the following conditions under which some generated rules are guaranteed to be non-minimal and hence can be excluded from the minimal set. Theorem 4.3 below proves that these conditions do indeed exclude only non-minimal rules, hence we state these rules with only limited explanation here. The “exclusion rules” used in MinZoomUR are: 1. If x and y are two large itemsets such that x is a parent of y and x.rule = 1 then the zoomin rule generated from y cannot be minimal. Hence for any itemset y such that parentrules(y) is TRUE, the zoomin rule generated from y cannot be minimal. This is the only exclusion rule used previously in MinZoominUR. 2. If x is a large itemset that generates a zoomin rule and some zoomout rules, then the zoomin rule generated cannot be minimal. 9 3. If x is a large itemset that generates zoomout rules p and q and elem_p ∈ x.dropped_subsets(p) and elem_q ∈ x.dropped_subsets(q) and elem_p ⊂ elem_q then p cannot be minimal. For example, for a belief a, b → z, let itemset x = {a, b, c, y} be large where y ∈ CONTR(z). If x generates the zoomout rules p ("b, c → y") and q ("c → y") then elem_p is {a} and elem_q is {a, b}. Since elem_p ⊂ elem_q, the rule b, c → y cannot belong to the minimal set of unexpected rules since it can be inferred from c → y using the monotonicity assumption. 4. If x and y are two large itemsets such that x is a parent of y, zoomout rules generated from y generated by dropping any subset, p, from the body of the belief such that p is a subset of some element belonging to x.dropped_subsets cannot be minimal rules. For example, for a belief a, b, c → z, let itemset x = {a, b, c, d, m} be large where m ∈ CONTR(z). Let x generate the zoomout rule c, d → m. Hence {a, b} ∈ x.dropped_subsets. This exclusion rule states that from any large itemset y, generated in subsequent iterations from x, the zoomout rules derived from y by dropping either {a} or {b} cannot be minimal. For example, assume the large itemset y = {a, b, c, d, e, m}. The zoomout rules generated from y by dropping {a} or {b} are, respectively, b, c, d, e → m and a, c, d, e → m. Observe that neither of these two zoomout rules can be minimal since these can be derived under the monotonicity assumption from the prior zoomout rule c, d → m. MinZoomUR generates candidate itemsets in the same manner as in MinZoominUR. A main difference in the algorithms is that MinZoomUR considers zoomout rules also for a given itemset immediately after the itemset generates a zoomin rule. This is necessary because some of the exclusion rules applied to an unexpected rule generated depends on knowing the zoomout rules generated for that itemset and its parents. 9 For belief B and itemset x, x.dropped_subsets(p) where p is a single zoomout rule, will contain only one element which is the subset of body(B) that was dropped to create the zoomout rule. 23 Inputs: Beliefs Bel_Set, Dataset D, minwidth and maxwidth for all ordered attributes and thresholds min_support and min_conf Outputs: For each belief, B, MinUnexp(B) 1 forall beliefs B ∈ Bel_Set { 2 MinUnexp(B) = {}; k=0 3 C0 = {{x,body(B)} | x ∈ CONTR(head(B))}; C0’ = {{body(B)}}; 4 while (Ck != ∅ ) do { forall c ∈ Ck ∪ Ck’, compute support(c) Lk = {x| x ∈ Ck, support(x) ≥ min_support } 5 6 8 Lk’ = {x| x ∈ Ck’, support(x) ≥ min_support} forall (x ∈ Lk) { 9 x.rule = 0; Let a = x ∩ CONTR(head(B)); 10 rule_conf = support(x)/support(x-a) 11 if (rule_conf > min_conf) { 7 12 x.rules = 1 13 zoutrules = minzoomoutrules(x,B) 14 x.dropped_subs = dropped_subsets(zoutrules,B) 15 MinUnexp(B) = MinUnexp(B) ∪ zoutrules 16 if (zoutrules == parentrules(x) == NULL) MinUnexp(B) = MinUnexp(B) ∪ {x – a →a} 17 if ({body(B)} ∈ x.dropped_subs) Lk = Lk - x 18 19 } 20 } 21 k++ 22 Ck = generate_new_candidates(Lk-1, B) 23 Ck’ = generate_bodies(Ck , B) 24 25 } forall x ∈ MinUnexp(B) { 26 Other_unexp = Unexp(B)-x 27 if (∃ y ∈ Other_unexp | y |=M x) 28 29 MinUnexp(B) = MinUnexp(B) - {x} } 30 } Figure 4.3 Algorithm MinZoomUR 24 MinZoomUR is presented in Figure 4.3. The beginning (Steps 1 through 7) and the end (Steps 25 through 30) of the algorithm where the minimal filter is applied are the same as in MinZoominUR. We explain steps 8 through 24 next, where unexpected rules are generated from each large itemset and the potentially minimal ones stored in MinUnexp(B). Steps 8 through 12 consider whether a large itemset, x, generates a zoomin rule and sets the attribute x.rule accordingly. If a zoomin rule is generated, step 13 applies the procedure minzoomoutrules to consider potentially minimal zoomout rules generated from this itemset. This procedure first applied exclusion rule #3 to generate some potentially minimal zoomout rules. Step 14 initializes x.dropped_subsets but notice that these are actually computed in the process of applying exclusion rule #3 in the previous step. Then exclusion rule #4 is applied to this set of rules to filter out any guaranteed non-minimal rules. The resulting set of potentially minimal zoomout rules generated from this itemset are zoutrules. From this set of rules, Step 14 sets the attribute x.dropped_subsets to the set of subsets of the body of the belief dropped in any of the rules in zoutrules. Step 15 adds the rules zoutrules to the potentially minimal set. In order to decide whether the zoomin rule generated for x should be added to the potentially minimal set, Step 16 and 17 applies exclusion rules #1 and #2. Finally, for each large itemset, x, Step 18 applies a corrollary to exclusion rule #4 - if the entire body of a belief is dropped to generate a zoomout rule, then no children of x can generate any minimal rule. Hence, in this event the itemset x is deleted from subsequent consideration. Below we prove the completeness of MinZoomUR. Theorem 4.3. For any belief MinZoomUR discovers the minimal set of unexpected patterns. Proof. As proved in Theorem 4.1, steps 25 through 30 generate the minimal set of unexpected patterns from MinUnexp(B). Hence in order to prove the theorem, it just needs to be shown that the exclusion rules applied prior to step 25 exclude only the non-minimal rules. In the remainder of the proof we will consider each exclusion rule and demonstrate that the rule excludes only non-minimal rules. Exclusion rule #1. This rule is the exclusion rule used in MinZoominUR. Hence, according to Theorem 4.2, this rule excludes only rules guaranteed to be non-minimal. Exclusion rule #2 This rule immediately follows from the definitions. Exclusion rule #3. Let x be a large itemset that generates zoomout rule p by dropping elem_p from body(B) and generates zoomout rule q by dropping elem_q from body(B). Further, as in exclusion rule #3, let elem_p ⊂ elem_q. Let c ∈ x such that c ∈ CONTR(B). The zoomout rule p therefore is x - elem_p - c → c and the zoomout rule q is x - elem_q - c → c. Since elem_p ⊂ elem_q, we have x - elem_p - c ⊃ x - elem_q - c. Hence 25 x - elem_p - c |= x - elem_q - c and as a result x - elem_q - c → c |=M x - elem_p - c → c. Thus x - elem_p - c → c is not minimal and can be excluded. Exclusion rule #4. Let x and y be two large itemsets such that x is a parent of y and assume that zoomout rule zy,p is generated from y by dropping a subset, p, from the body of the belief such that p is a subset of q where q ∈ x.dropped_subsets. It needs to be shown that zy,p cannot be minimal. Since x is a parent of y, x = y - k, where k is some non-empty itemset and there exists c ∈ CONTR(B) such that c ∈ x and c ∈ y. Further since p is a subset of q, assume p = q - t, where t is some itemset. The rule zy,p therefore is y - p - c → c. Since q ∈ x.dropped_subsets there is a zoomout rule zx,q generated from x that is obtained by dropping a subset q from the body of the belief. The rule zx,q therefore is x - q - c → c. Substituting for x and q, zx,q is equivalent to (y - k) - (p + t) - c → c which is the rule y - p - c - (k + t) → c. Since k is also non-empty, y - p - c ⊃ y - p - c - (k + t) and hence ,y - p - c |= y - p - c - (k + t). Therefore, zx,q |=M zy,p. Hence zy,p is not minimal and can be excluded. In this section we presented methods to generate the minimal set of unexpected patterns. The strength of these methods is that they eliminate redundant unexpected patterns and therefore generate a smaller set of unexpected patterns. In the next section we present experiments from a real world consumer purchase dataset that these ideas can be used to discover orders of magnitude fewer patterns that ZoomUR [PT98] and Apriori [AMS+95] and yet find most of the truly unexpected ones. 5. Experiments To illustrate the usefulness of our approach to discovering patterns, in this section we consider an illustrative case study application of applying the methods to consumer purchase data from a major market research firm. We pre-processed this data by combining different data sets (transaction data joined with demographics), made available to us into one table containing 38 different attributes and 313409 records. These attributes pertain to the item purchased by a shopper at a store over a period of one year, together with certain characteristics of the store and demographic data about the shopper and his or her family. Some demographic attributes include age and gender of the shopper, occupation, income and marital status of the household head and the presence of children in the family and the size of the household. Some transactionspecific attributes include product purchased, coupon usage (whether the shopper used any coupons to get a lower price or not), the availability of store coupons or manufacturer’s coupons and presence of advertisements for the product purchased in the store. For simplicity in generating beliefs and in making comparisons to other techniques that generate association rules in these experiments we restrict our 26 consideration to rules involving discrete attributes only. An initial set of 28 beliefs was generated by domain experts after examining 300 rules generated from the data using methods described in [P99]. In this section we present results from applying MinZoomUR, ZoomUR [PT98, P99] and Apriori [AMS+95] to this dataset starting from the initial set of beliefs where applicable. In Section 5.1 we compare these methods in terms of the number of rules generated and provide some guidelines as to when each may be applicable and in Section 5.2 we discuss scalability of MinZoomUR and ZoomUR with respect to the size of the database and the initial set of beliefs. 5.1 Number of Patterns Generated. In this section we compare and contrast MinZoomUR, ZoomUR and Apriori in terms of the number of patterns generated and other criteria. We also present practical implications of this comparison in terms of guidelines for when each may be preferable. For a fixed minimum confidence level of 0.6, Figure 5.1 through 5.3 show the number of patterns generated by Apriori, ZoomUR and MinZoomUR for varying levels of minimum support thresholds. Apriori generated 50,000 to 250,000 rules even for reasonably high minimum support values. This is not surprising since the objective of Apriori is to discover all strong association rules. For reasonable values of support (5 to 10%), ZoomUR generates 50 to 5000 unexpected patterns. MinZoomUR on the other hand generated only 15 to 700 unexpected patterns even for extremely low values for minimum support. 300,000 Number of Rules 250,000 200,000 150,000 100,000 50,000 0 5.0% 6.0% 7.0% 8.0% 9.0% 10.0% 11.0% 12.0% 13.0% Minimum Support Figure 5.1. Number of rules generated by Apriori 27 6000 5000 # of rules 4000 3000 2000 1000 0 5.3% 6.3% 7.3% 8.3% 9.3% 10.3% Minimum Support Figure 5.2. Number of unexpected rules generated by ZoomUR 800 700 # of rules 600 500 400 300 200 100 0 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% Minimum Support Figure 5.3. Number of unexpected rules generated by MinZoomUR Figure 5.4 illustrates the comparison of the three methods in terms of the number of generated rules. Due to the order of magnitude difference in the number of generated rules, the graph plots the number of rules generated using a logarithmic scale for the Y axis. As we would expect, as the minimum support threshold is lowered, all the methods discover a greater number of rules. Despite this, MinZoomUR discovers orders of magnitude fewer patterns than both ZoomUR and Apriori. The graphs in Figures 5.1 – 5.3 also demonstrate that a majority of patterns generated by ZoomUR are redundant. Observe that as the support threshold is lowered, the number of patterns generated by both ZoomUR and Apriori seem to increase more than linearly. While this is the case for MinZoomUR in 28 some regions, MinZoomUR plateaued out for lower regions of support, as Figure 5.3 demonstrates. This plateau signifies that very few new minimal unexpected patterns are generated despite the fact that the number of unexpected patterns generated by ZoomUR keep increasing in that region. This observation coupled with the comparison in the number of rules generated indicate that MinZoomUR is indeed effective in removing redundant patterns, which represent a large majority of the set of all unexpected patterns. 1000000 100000 10000 # of rules (logarithmic scale) Apriori 1000 ZoomUR MinZoomUR 100 10 1 0 2 4 6 8 10 12 Minimum Support Figure 5.4. Comparison of number of rules generated by Apriori, ZoomUR and MinZoomUR Discussion. Based on these experiments we discuss below some possible tradeoffs between these methods and provide some guidelines to their usage. The clear advantage of MinZoomUR over ZoomUR is that it generates far fewer patterns and yet retains most of the truly interesting ones. Since ZoomUR generates all unexpected patterns for a belief and MinZoomUR generates the minimal set of unexpected patterns, MinZoomUR will always generate a subset of patterns that ZoomUR generates. As shown above, this subset can be extremely small (from 15 to a few hundred patterns for the entire set of beliefs). We would also like to note here that the classical notion of “minimality” often assumes that it is possible to reconstruct the set of all objects having certain property from the minimal set of objects having this property. In our case also, the set of all unexpected patterns can be reconstructed from the minimal set of unexpected patterns. However, this can only be done using a generate-and-test procedure (starting from the minimal set) that requires data access again. This limitation of our approach is the result of using efficient search algorithms that directly discover the minimal set of unexpected patterns without even examining all unexpected patterns. Moreover, this limitation can also be circumvented by letting the domain expert examine the set of minimal unexpected patterns (that is small), 29 select the most interesting minimal patterns, and refine them to discover all the unexpected patterns obtained from this selected set. The drawback of MinZoomUR compared to ZoomUR is that MinZoomUR makes an implicit assumption that minimal unexpected patterns are the most interesting patterns. From a subjective point of view this may not be necessarily true. Consider the following example of two unexpected patterns: • When coupons are available for cereals, they don't get used (confidence = 60%) • On weekends, when coupons are available for cereals they don't get used (confidence = 98%) MinZoomUR will not generate the second unexpected pattern since it is monotonically implied by the first pattern. However, the second unexpected pattern has a much higher confidence and may be considered "more unexpected" by some users, in the spirit of [SA96, BAG99]. In a more general sense, the criteria implied by monotonicity and confidence are just two methods to rank unexpected patterns. In general there may be other criteria, some of which even depending on other subjective preferences of a user. Hence, since ZoomUR generates all the unexpected patterns, it is guaranteed to contain all the unexpected patterns that are "most unexpected" from any specific definition of the term "most unexpected". In subsequent work, we will study the issue of generating the "most unexpected patterns" by characterizing the degree of unexpectedness for patterns along the lines of [ST96]. In the context of objective measures of interestingness, [BA99] discuss interesting approaches to finding the “most interesting” patterns. Given the relative advantages of the two methods to discovering unexpected patterns, a practical implication of the above is that ZoomUR can be used to generate unexpected patterns for larger support values and MinZoomUR can be used if patterns of very low support need to be generated. As shown in Figure 5.3, MinZoomUR generates a reasonable number of unexpected patterns even for extremely small values of minimum support, as low as even 0.5%. Also the support of some beliefs about a domain may be very low, perhaps reflective of some condition that occurs rarely. In such cases methods such as MinZoomUR that can find patterns at very low support values are necessary. Apriori on the other hand has the drawback of generating a very large number of patterns since the objective is to discover all strong rules. As Figure 5.1 shows, for very low support values, this could result in millions of rules. However there are two sides of the coin. Generating a very large number of patterns results in a data mining problem of a second order and is hence avoidable. At the same time, domain knowledge that ZoomUR and MinZoomUR start with will almost always be incomplete for most business applications. Hence it is possible that either of the two methods that seek unexpected patterns could miss other interesting 30 patterns that may be unrelated to the captured domain knowledge. However the set of patterns generated by Apriori can be guaranteed to have all the interesting patterns since it has all patterns. We believe that this tradeoff is in some sense unavoidable since the problem of generating all interesting patterns (not just “unexpected”) is a difficult problem to solve. 5.2 Scalability Issues In this section we experimentally examine the scalability of ZoomUR and MinZoomUR with respect to the size of the database and the number of initial beliefs. Scalability with the size of the database For a sample of 10 beliefs, we ran ZoomUR and MinZoomUR multiple times by varying the number of records in the dataset from 40,000 to 200,000. Figure 5.4 and Figure 5.5 show the execution times for ZoomUR and MinZoomUR respectively. The experiments indicate that the methods are scalable in the range considered. Figures 5.4 and 5.5 indicate that both ZoomUR and MinZoomUR seem to scale linearly with the size of the database. This is not surprising since these algorithms are based on Apriori, which as shown in [AMS+95] scales linearly. Figure 5.4. Execution time of ZoomUR as a function of database size 30 Time (minutes) 25 20 15 10 5 0 0 50000 100000 150000 200000 Database size 31 25 Time (minutes) 20 15 10 5 0 0 50000 100000 150000 200000 250000 Database size Figure 5.5. Execution time of MinZoomUR as a function of database size Scalability as a function of number of initial beliefs Since ZoomUR and MinZoomUR generate unexpected patterns for each belief individually, these methods will scale linearly with the number of beliefs. However, the time taken to generate unexpected patterns for each belief is dependent on the specific belief in consideration due to many reasons such as the support and confidence values of the belief on the data and the number of unexpected patterns that actually exist. To examine this in greater detail, we measured the percentage of execution time that was used to generate unexpected patterns for each belief. Figures 5.6 and 5.7 illustrate the percentage of time taken for each of the beliefs. Each sector represents the time taken for generating unexpected patterns for a specific belief. Figure 5.6. Proportion of execution time of ZoomUR for each belief 32 Figure 5.7. Proportion of execution time of MinZoomUR for each belief Each of the pie charts in Figures 5.6 and 5.7 has slices corresponding to each belief. The distribution of time is similar in both cases, but the disproportionate share of some beliefs was interesting and we discuss a reason here as to why this would be expected to occur. Assume that a belief A → B has support s and confidence c in the data. The upper bound for the support for any unexpected rule generated by ZoominUR is 10 the fraction of the records where A is true and B is not true. This upper bound is (1-c)*s/c . For any given minimum support value, for beliefs where this fraction is large, intuitively many more itemsets can be generated and hence the execution time would be greater. When we ranked the beliefs according to this criterion, the top six beliefs were the same six beliefs that contributed to most of the execution time. Hence the distribution shown in Figure 5.6 and Figure 5.7 is just indicative of the fact that the initial set of beliefs may vary in their support and confidence values on the data. In this section we presented results pertaining to the effectiveness of MinZoomUR and compared it to Apriori and ZoomUR. In this real world case study we demonstrate that MinZoomUR can be used to discover far fewer patterns than Apriori and ZoomUR yet finding most of the truly interesting patterns. 6. Conclusions In this paper we presented a detailed formal characterization of the minimal set of unexpected patterns and proposed three algorithms for discovering the minimal set of such patterns. In a real world application we demonstrate that the main discovery algorithm, MinZoomUR, discovers orders of magnitude fewer patterns 10 Support(A, ¬B) = support(A) – support(A, B) = (s/c) – s = (1-c)*(s/c) 33 than other comparable methods yet retains most of the truly unexpected patterns. We also discuss tradeoffs between various discovery methods and present some guidelines for their usage. The power of this approach lies in combining two independent concepts of unexpectedness and minimality of a set of patterns into one integrated concept that provides for the discovery of small but important sets of interesting patterns. Moreover, MinZoominUR and MinZoomUR are efficient since we focus directly on discovering minimal unexpected patterns rather than adopting any of the post-processing approaches, such as filtering. Finally, the approach is effective in supporting decision making in real business applications where unexpected patterns with respect to managerial intuition can be of great value. One of the key assumptions in our approach to unexpectedness is the construction of a comprehensive set of beliefs since few truly interesting patterns can be discovered for a poorly specified set of beliefs. In [P99] we describe a method of generating a comprehensive set of beliefs. We also utilized this method to generate the initial set of beliefs used in the experiments described in Section 5. As a future research, we plan to study how different sets of initial beliefs affect generation of the minimal sets of unexpected patterns generated from these beliefs. We also plan to study how unexpected patterns can be generated from multiple beliefs. References [AIS93] Agrawal, R., Imielinski, T. and Swami, A., 1993. Mining Association Rules Between Sets of Items in Large Databases. In Proc. of the ACM SIGMOD Conference on Management of Data, pp. 207-216. [AMS+95] Agrawal, R., Mannila, H., Srikant, R., Toivonen, H. and Verkamo,A.I., 1995. Fast Discovery of Association Rules. In Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., and Uthurusamy, R. eds., Advances in Knowledge Discovery and Data Mining. AAAI Press. [AT97] Adomavicius, G., and Tuzhilin, A., 1997. Discovery of Actionable Patterns in Databases: The Action Hierarchy Approach. In Proc. of the Third International Conference on Knowledge Discovery and Data Mining. [BA99] Bayardo, R. and Agrawal, R., 1999. Mining the Most Interesting Rules. In Proc. of the Fifth ACMSIGKDD International Conference on Knowledge Discovery and Data Mining. [BAG99] Bayardo, R., Agrawal, R. and Gunopulos, D., 1999. Constraint-Based Rule Mining in Large, Dense Databases. In Proceedings of ICDE, 1999. [BT98] Berger, G. and Tuzhilin, A., 1998. Discovering Unexpected Patterns in Temporal Data Using Temporal Logic. In Etzion, O., Jajodia, S. and Sripada, S. eds, Temporal Databases: 34 Research and Practice. Springer, 1998. [BM78] Buchanan, B.G. and Mitchell, T.M., 1978. Model Directed Learning of Production Rules. In Waterman and Hayes-Roth (eds.) Pattern-Directed Inference Systems, Academic Press, New York. [BMU+97] Brin, S., Motwani, R., Ullman, J.D., and Tsur, S., 1997. Dynamic Itemset Counting and Implication Rules for Market Basket Data. Procs. ACM SIGMOD Int. Conf. on Mgmt. of Data, pp.255-264. [CSD98] Chakrabarti, S., Sarawagi, S. and Dom, B. Mining Surprising Patterns Using Temporal Description Length. In International Conference on Very Large Databases, 1998. [F97] Forbes Magazine, Sep. 8, 1997. Believe in yourself, believe in the merchandise, pp.118-124. [FPS96] Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., 1996. From Data Mining to Knowledge Discovery: An Overview. In Fayyad, U.M.,Piatetsky-Shapiro, G., Smyth, P., and Uthurusamy, R. eds., Advances in Knowledge Discovery and Data Mining. AAAI/MIT Press. [HRM75] Hayes-Roth, M. and Mostow, D., 1975. An Automatically Compliable Recognition Network for Structured Patterns. In Proceedings of International Joint Conference on Artificial Intelligence-1975., pp. 356-362. [KMR+94] Klemettinen, M., Mannila, H., Ronkainen, P., Toivonen, H. and Verkamo, A.I., 1994. Finding Interesting Rules from Large Sets of Discovered Association Rules. In Proc. of the Third International Conference on Information and Knowledge Management, pp. 401-407. [LH96] Liu, B. and Hsu, W., 1996. Post-Analysis of Learned Rules. In Proc. of the Thirteenth National Conference on Artificial Intelligence (AAAI ’96), pp. 828-834. [LHC97] Liu, B., Hsu, W. and Chen, S, 1997. Using General Impressions to Analyze Discovered Classification Rules. In Proc. of the Third International Conference on Knowledge Discovery and Data Mining (KDD 97), pp. 31-36. [LHM99] Liu, B., Hsu, W., and Ma, Y., 1999. Pruning and Summarizing the Discovered Rules. In Proc. of the Fifth ACM-SIGKDD International Conference on Knowledge Discovery and Data Mining. [M77] Mitchell, T.M., 1977. Version Spaces: A Candidate Elimination Approach to Rule Learning. In Proceedings of International Joint Conference on Artificial Intelligence-1977., pp. 305-310. [M82] Mitchell, T., 1982. Generalization as Search. Artificial Intelligence, pp. 203-226. [MPM96] Matheus, C.J., Piatetsky-Shapiro, G. and McNeill, D., 1996. Selecting and Reporting What is Interesting: The KEFIR Application to Healthcare Data. In Advances in Knowledge Discovery and Data Mining. AAAI Press, 1996. 35 [MUB82] Mitchell, T.M., Utgoff, P.E., and Banerji, R.B., 1982. Learning Problem-Solving Heuristics by Experimentation. In Michalski et al. (Eds.) Machine Learning, Tioga Press, Palo Alto. [P70] Plotkin, G.D., 1970. A Note on Inductive Generalization. In Meltzer and Michie (eds.) Machine Intelligence, Edinburgh University Press, Edinburgh. [P99] Padmanabhan, B., 1999. Discovering Unexpected Patterns in Data Mining Applications. Doctoral Dissertation, New York University, May 1999. [PSM94] Piatetsky-Shapiro, G. and Matheus, C.J., 1994. The Interestingness of Deviations. In Procs. of the AAAI-94 Workshop on Knowledge Discovery in Databases, pp. 25-36. [PT98] Padmanabhan, B. and Tuzhilin, A., 1998. “A Belief-Driven Method for Discovering Unexpected Patterns.” In Proc. of the 4th International Conference on Knowledge Discovery and Data Mining, 1998. [PT99] Padmanabhan, B. and Tuzhilin, A., 1999. Unexpectedness as a Measure of Interestingness in Knowledge Discovery. In Decision Support Systems, (27)3 (1999) pp. 303-318 [S97] Stedman, C., 1997. Data Mining for Fool's Gold. Computerworld, Vol. 31,No. 48, Dec. 1997. [SA96] Srikant, R. and Agrawal, R., 1996. Mining Quantitative Association Rules in Large Relational Tables. In Proc. of the ACM SIGMOD Conference on Management of Data, 1996. [SLR+99] Shah, D., Lakshmanan, L.V.S, Ramamritham, K., and Sudarshan, S., 1999. Interestingness and Pruning of Mined Patterns. In Proceedings of the 1999 ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery (DMKD), Philadelphia, 1999. [ST95] Silberschatz, A. and Tuzhilin, A., 1995. On Subjective Measures of Interestingness in Knowledge Discovery. In Proc. of the First International Conference on Knowledge Discovery and Data Mining, pp. 275-281. [ST96] Silberschatz, A. and Tuzhilin, A., 1996. What Makes Patterns Interesting in Knowledge Discovery Systems. IEEE Transactions on Knowledge and Data Engineering. Special Issue on Data Mining, vol. 5, no. 6, pp. 970-974. [SVA97] Srikant, R., Vu, Q. and Agrawal, R. Mining Association Rules with Item Constraints. In Proc. of the Third International Conference on Knowledge Discovery and Data Mining (KDD 97), pp. 67-73. [Sub98] Subramonian, R. Defining diff as a data mining primitive. In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, 1998. [Suz97] Suzuki, E., 1997. Autonomous Discovery of Reliable Exception Rules. In Proc. of the Third International Conference on Knowledge Discovery and Data Mining, pp. 259-262. 36 [TKR+95] Toivonen, H., Klemetinen, M., Ronkainen, P., Hatonen, K. and Mannila, H., 1995. Pruning and Grouping Discovered Association Rules. In MLNet Workshop on Statistics, Machine Learning and Discovery in Databases, pp. 47-52. [V78] Vere, S.A., 1978. Inductive Learning of Relational Productions. In Waterman and Hayes-Roth (Eds.) Pattern-Directed Inference Systems, Academic Press, New York. [W75] Winston, P.H., 1975. Learning Structural Descriptions from Examples. In Winston (ed.) The Psychology of Computer Vision, McGraw Hill, New York. 37