advertisement

Knowledge Discovery in Databases Alaaeldin M. Hafez Department of Computer Science and Automatic Control Faculty of Engineering Alexandria University Abstract. Business information received from advanced data analysis and data mining is a critical success factor for companies wishing to maximize competitive advantage. The use of traditional tools and techniques to discover knowledge is ruthless and does not give the right information at the right time. Knowledge discovery is defined as ``the non-trivial extraction of implicit, unknown, and potentially useful information from data''. There are many knowledge discovery methodologies in use and under development. Some of these techniques are generic, while others are domain-specific. In this paper, we present a review outlining the state-ofthe-art techniques in knowledge discovery in database systems 1. Introduction Recent years have seen an enormous increase in the amount of information stored in electronic format. It has been estimated that the amount of collected information in the world doubles every 20 months and the size and number of databases are increasing even faster and the ability to rapidly collect data has outpaced the ability to analyze it. Information is crucial for decision making, especially in business operations. As a response to those trends, the term 'Data Mining' (or 'Knowledge Discovery') has been coined to describe a variety of techniques to identify nuggets of information or decisionmaking knowledge in bodies of data, and extracting these in such a way that they can be put to use in the areas such as decision support, prediction, forecasting and estimation. Automated tools must be developed to help extract meaningful information from a flood of information. Moreover, these tools must be sophisticated enough to search for correlations among the data unspecified by the user, as the potential for unforeseen relationships to exist among the data is very high. A successful tool set to accomplish these goals will locate useful nuggets of information in the otherwise chaotic data space, and present them to the user in a contextual format. An urgent need for creating a new generation of techniques is needed for automating data mining and knowledge discovery in databases (KDD). KDD is a broad area that integrates methods from several fields including statistics, databases, AI, machine learning, pattern recognition, machine discovery, uncertainty modeling, data visualization, high performance computing, optimization, management information systems (MIS), and knowledge-based systems. 1 The term “Knowledge discovery in databases” is defined as the process of identifying useful and novel structure (model) in data [1, 3, 4, 17, 31]. It could be viewed as a multistage process. Those stages are summarized as follows: Data gathering, e.g., databases, data warehouses, Web crawling. Data cleansing; eliminate errors, e.g., GPA = 7.3. Feature extraction; obtaining only the interesting attributes of data Data mining; discovering and extracting meaningful patterns. Visualization of data. Verification and evaluation of results; drawing conclusions. Data Gathering Databases Data Warehouses Data Cleansing Web Crawlers Feature Extraction Verification and evaluation Data Mining Visualization Data mining is considered as the main step in the knowledge discovery process that is concerned with the algorithms used to extract potentially valuable patterns, associations, trends, sequences and dependencies in data [1, 3, 4, 7, 22, 33, 34, 37, 39]. Key business examples include web site access analysis for improvements in e-commerce advertising, fraud detection, screening and investigation, retail site or product analysis, and customer segmentation. Data mining techniques can discover information that many traditional business analysis and statistical techniques fail to deliver. Additionally, the application of data mining techniques further exploits the value of data warehouse by converting expensive volumes of data into valuable assets for future tactical and strategic business development. Management information systems should provide advanced capabilities that give the user the power to ask more sophisticated and pertinent questions. It empowers the right people by providing the specific information they need. Data mining techniques could be categorized either by tasks or by methods. 1.1 Data mining tasks: Association Rule Discovery. 2 Classification. Clustering. Sequential Pattern Discovery. Regression. Deviation Detection Most researchers refer to the first three data mining tasks as the main data mining tasks. The rest are either related more to some other fields, such as regression, or considered as a sub-field in one of the main tasks, such as sequential pattern discovery and deviation detection. 1.2 Data Mining Methods: Algorithms for mining spatial, textual, and other complex data Incremental discovery methods and re-use of discovered knowledge Integration of discovery methods Data structures and query evaluation methods for data mining Parallel and distributed data mining techniques Issues and challenges for dealing with massive or small data sets Fundamental issues from statistics, databases, optimization, and information processing in general as they relate to problems of extracting patterns and models from data. Because of the limited space, we limit our discussion on the main techniques in data mining tasks. In section 2, we briefly discuss and give examples of the various data mining tasks. In sections 3, 4 and 5, we discuss the main features and techniques in association mining, classification and clustering, respectively. The paper is concluded in section 6. 2 Data Mining Tasks 2.1 Association Rule Discovery Given a set of records each of which contains some set of items from a given collection of items. Produce dependency rules that predict occurrences of one item based on the occurrences of some other items [1, 3,27, 28]. Transaction ID 1 2 3 4 5 6 Items Milk, Bread, Diaper Milk, Bread Coke, Milk, Bread, Diaper Coke, Coffee, Bread Bread, Coke Milk, Coke, Coffee 3 Some of the Discovered Rules are: Bread Milk Milk, Bread Diaper Example 2.1: (Supermarket shelf management) Identify items that are bought together by sufficiently many customers. Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items. Stack those items mostly likely to buy together next to each other. Example 2.2: (Inventory Management) A consumer appliance repair company wants to anticipate the nature of repairs on its consumer products and keep the service vehicles equipped with right parts to reduce on number of visits to consumer households. Approach: 2.2 Process the data on tools and parts required in previous repairs at different consumer locations and discover the co-occurrence patterns. Classification Given a collection of records (training set), each record contains a set of attributes [10, 12, 30, 32, 35], one of the attributes is the class (Classifier), find a model for class attribute as a function of the values of other attributes. previously unseen records should be assigned a class as accurately as possible. a test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it. Test Set Training Set Classifier Model 4 Example 2.3: (Marketing) Reduce cost of mailing by targeting a set of consumers likely to buy a new car. Approach: Use the data for a similar product introduced before. Define the class attribute, which customers decided to buy and which decided otherwise. This {buy, don’t buy} decision forms the class attribute. For all such customers, collect various lifestyle, demographic, and business related information such as salary, type of business, city, etc. Use this information as input attributes to learn a classifier model. Example 2.4: (Fraud Detection) Predict fraudulent cases in credit card transactions. Approach: Use credit card transactions and the information on its account-holder as attributes, such as, when does a customer buy, what does he buy, how often he pays on time, etc. Label past transactions as fraud or fair transactions. This forms the class attribute. Learn a model for the class of the transactions. Use this model to detect fraud by observing credit card transactions on an account. 2.3 Clustering Given a set of data points, each having a set of attributes, and a similarity measure among them [24, 25, 28, 41]. Find clusters such that Data points in one cluster are more similar to one another. Data points in separate clusters are less similar to one another. Where similarity measures are Euclidean Distance if attributes are continuous. Other Problem-specific Measures. 5 Example 2.5: (Market Segmentation) Subdivide a market into distinct subsets of customers where any subset may conceivably be selected as a market target to be reached with a distinct marketing mix. Approach: Collect different attributes of customers based on their geographical and lifestyle related information. Find clusters of similar customers. Measure the clustering quality by observing buying patterns of customers in same cluster vs. those from different clusters. Example 2.6: (Document Clustering) Find groups of documents that are similar to each other based on the important terms appearing in them. Approach: Identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster. Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents. 2.4 Sequential Pattern Discovery Given a set of event sequence, find rules that predict strong sequential dependencies among different events [4, 20, 37]. Rules are formed by first discovering patterns. Event occurrences in the patterns are governed by timing constraints. Example 2.7: (Telecommunications Alarm) In telecommunication alarm logs, Rectifier_Alarm Fire_Alarm 2.5 Regression Regression techniques are important in statistics and neural network fields [10]. Assuming a linear or nonlinear model of dependency, they are used in predicting a value of a given continuous valued variable based on the values of other variables. 6 Example 2.8: Predicting sales amounts of new product based on advertising expenditure. Predicting wind velocities as a function of temperature, humidity, air pressure, etc. Time series prediction of stock market indices. 2.6 Deviation Detection Deviation detection deals with discovering the most significant changes that are often infrequent, in data from previously measured values. Example 2.9: Outlier detection in statistics. 3. Association Rule Discovery Association rule discovery or association mining that discovers dependencies among values of an attribute was introduced by Agrawal et al.[1] and has emerged as a prominent research area. The association mining problem also referred to as the market basket problem can be formally defined as follows. Let I = {i1,i2, . . . , in} be a set of items as S = {s1, s2, . . ., sm} be a set of transactions, where each transaction si S is a set of items that is si I. An association rule denoted by X Y, where X,Y I and X Y = , describes the existence of a relationship between the two itemsets X and Y. Several measures have been introduced to define the strength of the relationship between itemsets X and Y such as support, confidence, and interest. The definitions of these measures, from a probabilistic model are given below. Support (X Y ) P( X , Y ) ; the percentage of transactions in the database that contain both X and Y. Confidence(X Y ) P( X , Y ) / P( X ) ; the percentage of transactions containing Y in transactions those contain X. Interest(X Y ) P( X , Y ) / P( X ) P(Y ) , represents a test of statistical independence. The problem of finding all association rules that have support and confidence greater than some user-specified minimum support and confidence out of database D, could be viewed as Find all sets of items (itemsets) that have transaction support above minimum support. The support for an itemset is the number of transactions that contain the itemset. Itemsets with minimum support are called large itemsets. An itemset of size k is a k-itemset. 7 Use the large itemsets to generate the desired association rules with minimum confidence. Most association mining algorithms concentrate on finding large itemsets, and consider generating association rules as a straightforward procedure. 3.1 Spectrum of Association Mining Techniques Many algorithms [1, 2, 3, 5, 7, 11, 22, 23, 33, 34, 37, 39], have been proposed to generate association rules that satisfy certain measures. A close examination of those algorithms reveals that the spectrum of techniques that generate association rules, has two extremes: A transaction data file is repeatedly scanned to generate large itemsets. The scanning process stops when there are no more itemsets to be generated. A transaction data file is scanned only once to build a complete transaction lattice. Each node on the lattice represents a possible large itemset. A count is attached to each node to reflect the frequency of itemsets represented by nodes. In the first case, since the transaction data file is traversed many times, the cost of generating large itemsets is high. In the later case, while the transaction data file is traversed only once, the maximum number of nodes in the transaction lattice is 2n , n is the cardinality of I, the set of items. Maintaining such a structure is expensive. In association mining, the subset relation of item-sets. Also, the subset relation itemsets. In other words, if B is a frequent itemset, then all subsets A B are also frequent. Association mining algorithms are different in the way they search the itemset lattice spanned by the subset relation. Most approaches use a level-wise or bottom-up search of the lattice to enumerate the frequent itemsets. If long frequent itemsets are expected, a pure top-down approach might be preferred. Some have proposed a hybrid search, which combines top-down and bottom-up approaches. 3.2 New Candidates Generation Association mining algorithms can differ in the way they generate new candidates. A complete search, the dominant approach, guarantees that we can generate and test all frequent subsets. Here, complete doesn’t mean exhaustive; we can use pruning to eliminate useless branches in the search space. Heuristic generation sacrifices completeness for the sake of speed. At each step, it only examines a limited number of “good” branches. Random search to locate the maximal frequent itemsets is also possible. Methods that can be used here include genetic algorithms and simulated annealing. Because of a strong emphasis on completeness, association mining literature has not given much attention to the last two methods. Also, association mining algorithms differ depending on whether they generate all frequent subsets or only the maximal ones. Identifying the maximal itemsets is the core 8 task, because an additional database scan can generate all other subsets. Nevertheless, the majority of algorithms list all frequent itemsets. 3.3 Association Mining Algorithms 3.3.1 The Apriori Algorithm The Apriori algorithm [1] is considered as the most famous algorithms in the area of association mining. Apriori starts with a seed set of itemsets found to be large in the previous pass, and uses it to generate new potentially large itemsets (called candidate itemsets). The actual support for these candidates is counted during the pass over the data, and non-large candidates are thrown out. The main outlines of the Apriori algorithm are described below. Apriori algorithm: First pass: counts item occurrences to determine the large 1-itemsets. Second and subsequent passes: for (k=2; Lk-1 not empty; k++) Ck = apriori-gen(Lk-1); // (New candidates) forall transactions t in D do Ct = subset(Ck,t) // (Candidates contained in t) forall candidates c in Ct do o c.count++ Lk = {c in Ck | c.count >= minsupport } Answer = Unionk(Lk) where Lk: Set of large k-itemsets (i.e. those with minimum support). Each member of set has an itemset and a support count. Ck: Set of candidate k-itemsets (potentially large). It has itemset and support count. apriori-gen accepts the set of all large (k-1)-itemsets Lk-1, and returns a superset of the set of all large k-itemsets. First, it performs a join of Lk-1 to itself to generate Ck, and then prunes all itemsets from Ck such that some (k-1)-subset of that itemset is not in Lk-1. In Apriori algorithm, the transaction data file is repeatedly scanned to generate large itemsets. The scanning process stops when there are no more itemsets to be generated. For large databases, the Apriori algorithm requires extensive access of secondary storage that can become a bottleneck for efficient processing. Several Apriori based algorithms have been introduced to overcome the excessive use of I/O devices. 9 3.3.2 Apriori Based Algorithms The AprioriTid algorithm[4] is a variation of the Apriori algorithm. The AprioriTid algorithm also uses the "apriori-gen" function to determine the candidate itemsets before the pass begins. The main difference from the Apriori algorithm is that the AprioriTid algorithm does not use the database for counting support after the first pass. Instead, the set <TID, {Xk}> is used for counting. (Each Xk is a potentially large k-itemset in the transaction with identifier TID.) The benefit of using this scheme for counting support is that at each pass other than the first pass, the scanning of the entire database is avoided. But the downside of this is that the set <TID, {Xk}> that would have been generated at each pass may be huge. Another algorithm, called AprioriHybrid, is introduced in [3]. The basic idea of the AprioriHybird algorithm is to run the Apriori algorithm initially, and then switch to the AprioriTid algorithm when the generated database (i.e. <TID, {Xk}>) would fit in the memory. 3.3.3. The Partition Algorithm In [34], Savasere et al. introduced the Partition algorithm. The Partition algorithm logically partitions the database D into n partitions, and only reads the entire database at most two times to generate the association rules. The reason for using the partition scheme is that any potential large itemset would appear as a large itemset in at least one of the partitions. The algorithm consists of two phases. In the first phase, the algorithm iterates n times, and during each iteration, only one partition is considered. At any given iteration, the function "gen_large_itemsets" takes a single partition and generates local large itemsets of all lengths from this partition. All of these local large itemsets of the same lengths in all n partitions are merged and then combined to generate the global candidate itemsets. In the second phase, the algorithm counts the support of each global candidate itemsets and generates the global large itemsets. Note that the database is read twice during the process: once in the first phase and the other in the second phase, which the support counting requires a scan of the entire database. By taking minimal number of passes through the entire database drastically saves the time used for doing I/O. 3.3.4. The Dynamic Itemset Counting Algorithm In the Dynamic Itemset Counting (DIC) algorithm [11], the database is divided into p equal-sized partitions so that each partition fits in memory. For partition 1, DIC gathers the supports of single items. Items found to be locally frequent (only in this partition) generate candidate 2-itemsets. Then DIC reads partition 2 and obtains supports for all current candidates—that is, the single items and the candidate 2-itemsets. This process repeats for the remaining partitions. DIC starts counting candidate k-itemsets while processing partition k in the first database scan. After the last partition p has been processed, the processing wraps around to partition 1 again. A candidate’s global support is known once the processing wraps around the database and reaches the partition where it was first generated. 10 DIC is effective in reducing the number of database scans if most partitions are homogeneous (have similar frequent itemset distributions). If data is not homogeneous, DIC might generate many false positives (itemsets that are locally frequent but not globally frequent) and scan the database more than Apriori does. DIC proposes a random partitioning technique to reduce the data partition skew. 3.4 New Trends in Association Mining Many knowledge discovery applications, such as on-line services and world wide web, are dynamic and require accurate mining information from data that changes on a regular basis. In world wide web, every day hundreds of remote sites are created and removed. In such an environment, frequent or occasional updates may change the status of some rules discovered earlier. Performance is the main consideration in data mining. Discovering knowledge is an expensive operation. It requires extensive access of secondary storage that can become a bottleneck for efficient processing. Two solutions have been proposed to improve the performance of the data mining process, Dynamic data mining Parallel data mining 3.4.1 Dynamic Data Mining Using previously discovered knowledge along with new data updates to maintain discovered knowledge could solve many problems, that have faced data mining techniques; that is, database updates, accuracy of data mining results, gaining more knowledge and interpretation of the results, and performance. In [33], a dynamic approach that dynamically updates knowledge obtained from the previous data mining process was introduced. Transactions over a long duration are divided into a set of consecutive episodes. Information gained during the current episode depends on the current set of transactions and the discovered information during the last episode. The approach discovers current data mining rules by using updates that have occurred during the current episode along with the data mining rules that have been discovered in the previous episode. SUPPORT for an itemset S is calculated as SUPPORT ( S ) F ( S ) F where F(S) is the number of transactions having S, and F is the total number of transactions. For a minimum SUPPORT value MINSUP, S is a large (or frequent) itemset if SUPPORT(S) MINSUP, or F(S) F*MINSUP. 11 Suppose we have divided the transaction set T into two subsets T1 and T2, corresponding to two consecutive time intervals, where F1 is the number of transactions in T1 and F2 is the number of transactions in T2, (F=F1+F2), and F1(S) is the number of transactions having S in T1 and F2(S) is the number of transactions having S in T2, (F(S)=F1(S)+F2(S)). By calculating the SUPPORT of S, in each of the two subsets, we get F (S ) F ( S ) and SUPPORT2 ( S ) 2 SUPPORT1 ( S ) 1 F2 F1 S is a large itemset if F1 ( S ) F2 ( S ) MINSUP F1 F2 , or F1 ( S ) F2 ( S ) ( F1 F2 )* MINSUP In order to find out if S is a large itemset or not, we consider four cases, S is a large itemset in T1 and also a large itemset in T2, i.e., F1 ( S ) F1 * MINSUP and F2 ( S ) F2 * MINSUP . S is a large itemset in T1 but a small itemset in T2, i.e., F1 ( S ) F1 * MINSUP and F2 ( S ) F2 * MINSUP . S is a small itemset in T1 but a large itemset in T2, i.e., F1 ( S ) F1 * MINSUP and F2 ( S ) F2 * min sup . S is a small itemset in T1 and also a small itemset in T2, i.e., F1 ( S ) F1 * MINSUP and F2(S)< F2*MINSUP. In the first and fourth cases, S is a large itemset and a small itemset in transaction set T, respectively, while in the second and third cases, it is not clear to determine if S is a small itemset or a large itemset. Formally speaking, let SUPPORT(S) = MINSUP + , where 0 if S is a large itemset, and 0 if S is a small itemset. The above four cases have the following characteristics, 1 0 and 2 0 1 0 and 2 0 1 0 and 2 0 1 0 and 2 0 S is a large itemset if F1 * ( MINSUP 1 ) F2 * ( MINSUP 2 ) MINSUP , or F1 F2 F1 * ( MINSUP 1 ) F2 * ( MINSUP 2 ) MINSUP * ( F1 F2 ) which can be written as F1 * 1 F2 * 2 0 Generally, let the transaction set T be divided into n transaction subsets Ti 's, 1 i n. S n is a large itemset if Fi * i 0 , where Fi is the number of transactions in Ti and i = i 1 SUPPORTi(S) - MINSUP, 1 i n. -MINSUP i 1-MINSUP, 1 i n. n For those cases where i 1 Fi * i 0 , there are two options, either 12 discard S as a large itemset (a small itemset with no history record maintained), or keep it for future calculations (a small itemset with history record maintained). In this case, we are not going to report it as a large itemset, but its formula F * n i 1 i i will be maintained and checked through the future intervals. 3.4.2. Parallel Association Mining algorithms Association mining is computationally and I/O intensive [2, 18]. Data is huge in terms of number of items and number of transactions, one of the main features needed in association ming is scalability. For large databases, sequential algorithms cannot provide scalability, in terms of the data size or runtime performance. Therefore, we must rely on high-performance parallel computing. In the literature, most parallel association mining algorithms [2, 18, 21] are based on their sequential counterparts. Researchers expect parallelism to relieve current association mining methods from the sequential bottleneck, providing scalability to massive data sets and improving response time. The main challenges include synchronization and communication minimization, workload balancing, finding good data layout and data decomposition, and disk I/O minimization. The parallel design space spans three main components: Memory Systems Type of Parallelism Load Balancing 3.4.2.1 Memory Systems Three techniques for using multiple processors have been considered [2, 18, 21, 27, 28]. Shared memory (where all processors access common memory). In shared memory architecture, many desirable properties could be achieved. Each processor has direct and equal access to all the system’s memory. Parallel programs are easy to implement on such a system. Although shared memory architecture offers programming simplicity, a common bus’s finite bandwidth can limit scalability. Distributed memory (where each processor has a private memory). In distributed memory architecture, each processor has its own local memory, which only that processor can access directly. For a processor to access data in the local memory of another processor, message passing must send a copy of the desired data elements from one processor to the other. A distributed-memory, message-passing architecture cures the scalability problem by eliminating the bus, but at the expense of programming simplicity. Combine the best of the distributed and shared memory approaches. The physical memory is distributed among the nodes but a shared global address space on each 13 processor is provided. Locally cached data always reflects any processor’s latest modification. 3.4.2.2 Type of Parallelism Task and data parallelism are the two main paradigms for exploiting algorithm parallelism. Data parallelism corresponds to the case where the database is partitioned among P processors— logically partitioned for shared memory architecture, physically for distributed memory architecture. Each processor works on its local partition of the database but performs the same computation of counting support for the global candidate itemsets. Task parallelism corresponds to the case where the processors perform different computations independently, such as counting a disjoint set of candidates, but have or need access to the entire database. In shard memory architecture, processors have access to the entire data, but for distributed memory architecture, the process of accessing the database can involve selective replication or explicit communication of the local portions. Hybrid parallelism, which combines both task and data parallelism, is also possible and perhaps desirable for exploiting all available parallelism in association mining methods. 3.4.2.3 Load Balancing Two approaches are used in load balancing, static load balancing, and dynamic load balancing In static load balancing, a heuristic cost function initially partitions work among the processors. Subsequent data or computation movement is not available to correct load imbalances. Dynamic load balancing takes work from heavily loaded processors and reassigning it to lightly loaded ones. Dynamic load balancing requires additional costs for work and data movement, and also for the mechanism used to detect whether there is an imbalance. However, dynamic load balancing is essential if there is a large load imbalance or if the load changes with time. Dynamic load balancing is especially important in multi-user environments with transient loads and in heterogeneous platforms, which have different processor and network speeds. These kinds of environments include parallel servers and heterogeneous clusters, meta-clusters, and super-clusters. All extant association mining algorithms use only static load balancing that is inherent in the initial partitioning of the database among available nodes. This is because they assume a dedicated, homogeneous environment. 14 4. Classification Classification is an important problem in the rapidly emerging field of data mining [10, 19, 30, 32]. The problem can be stated as follows. We are given a training dataset consisting of records. Each record is identified by a unique record id and consists of fields corresponding to the attributes. An attribute with a continuous domain is called a continuous attribute. An attribute with finite domain of discrete values is called a categorical attribute. One of the categorical at-tributes is the classifying attribute or class and the values in its domain are called class labels. Classification is the process of discovering a model for the class in terms of the remaining attributes. 4.1 Serial Algorithms for Classification Serial Algorithms for classification [10, 12, 19, 29, 30] are categorized as decision tree based methods and non-decision tree based methods. Non-decision tree based methods include neural networks, genetic algorithms, and Bayesian networks. 4.1.1. Decision Tree Based Classification The decision tree models [5] are found to be most useful in the domain of data mining. They yield comparable or better accuracy as compared to other models such as neural networks, statistical models or genetic models [30]. Many advantages could be considered such as o o o o Inexpensive to construct Easy to Interpret Easy to integrate with database systems Comparable or better accuracy in many applications Example 4.1: ID 1 2 3 4 5 6 7 8 9 10 Income 95K 120K 140K 80K 160K 100K 90K 75K 170K 125K Marital Status Married Single Single Married Divorced Married Single Divorced Divorced Single Refund Yes Yes No No Yes Yes Yes No No No Cheat NO NO YES NO NO NO NO NO YES YES 15 Refund Yes No NO Marital Status Single/Divorced Married NO Income >80K <=80K YES NO Many algorithms have been used in constructing decision trees. Hunt’s algorithm is one of the earliest versions that have been used in building decision trees. Many other algorithms such as, CART [10], ID3, C4.5 [32], SLIQ [29], and SPRINT [35], have followed. Two phases are needed for the structure of the classification tree, o o Tree Induction Tree Pruning 4.1.1.1 Decision Tree Induction For tree induction, records are split based on an attribute that optimizes the splitting criterion. Splitting could be either on a categorical attribute or on a continuous attribute. For categorical attributes, each partition has a subset of values signifying it. Two methods are use to form partition. In the simple method, it uses as many partitions as distinct values, while in the complex method, two partitions are used, and values are divided into two subsets; one subset for each partition. Marital Status Single Divorced Married Simple Method 16 Marital Status {Married} {Single, Divorced} Complex Method For continuous attributes, splitting is done either by using the static approach, where Apriori discretization is used to form a categorical attribute, or by using the dynamic approach, where decisions are made as algorithm proceeds. The dynamic approach is complex but more powerful and flexible in approximating true dependency. Dynamic decisions are made as follows: Form binary decisions based on one value; two partitions: A < v and A >= v Find ranges of values and use each range to partition. o Ranges can be found by a simple-minded bucketing, or by more intelligent clustering techniques Example: Salary in [0,15K), [15K, 60K), [60K, 100K), [100K,...] Find a linear combination of multiple variables, and make binary decisions or range-decisions based on it. Different splitting criterions are used. CART, SLIQ and SPRINT use GINI index as a splitting value. For n classes, the value of GINI index at node t is calculated as n GINI (t ) 1 ( p( j / t )) 2 j 1 where p(j/t) is the relative frequency of class j at node t. For node t, GINI index value is maximum (i.e., 1-1/n) when records are equally distributed among all classes, i.e., least interesting information. GINI index value is minimum (i.e., 0) when all records belong to one class, i.e., most interesting information. 17 Example 4.2: Count (Class1) = 0 Count (Class2) = 8 GINI = 0 Count (Class1) = 1 Count (Class2) = 7 GINI = 0.2187 Count (Class1) = 2 Count (Class2) = 6 GINI = 0.375 Count (Class1) = 3 Count (Class2) = 5 GINI = 0.4687 Count (Class1) = 4 Count (Class2) = 4 GINI = 0.5 Usually, when the GINI index is used, the splitting criterion is to minimize the GINI index of the split. When a node e is split into k partitions, the quality of the split is computed as k GINI split i 1 ni GINI (i ) n n is the number of records at node e, and ni is the number of records at node (partition) i. To compute the GINI index, the following steps are used. Use Binary Decisions based on one value. Choose the splitting value, e.g., number of possible splitting values = number of distinct values. Each splitting value has a count matrix associated with it, i.e., class counts in each of the partitions, A < v and A >= v Choose best v, for each v, scan the database to gather count matrix and compute its Gini index. This could be computationally inefficient (repetition of work.) For efficient computation: for each attribute, o Sort the attribute on values/ o Linearly scan these values, each time updating the count matrix and computing gini index. o Choose the split position that has the least GINI index. In C4.5 and ID3, another splitting criterion based on an information or entropy measure (INFO). INFO based computations are similar to GINI index computations. For n classes, the value of INFO at node t is calculated as n INFO(t ) p( j / t ) log( p( j / t )) j 1 where p(j/t) is the relative frequency of class j at node t. For node t, INFO value is maximum (i.e., log (n)) when records are equally distributed among all classes, i.e., least interesting information. 18 INFO value is minimum (i.e., 0) when all records belong to one class, i.e., most interesting information. Example 4.3: Count (Class1) = 0 Count (Class2) = 8 INFO = 0 Count (Class1) = 1 Count (Class2) = 7 INFO = 0.16363 Count (Class1) = 2 Count (Class2) = 6 INFO = 0.24422 Count (Class1) = 3 Count (Class2) = 5 INFO = 0.28731 Count (Class1) = 4 Count (Class2) = 4 INFO = 0.30103 When a node e is split into k partitions, the information gain is computed as k GAIN split INFO(e) i 1 ni INFO(i ) n n is the number of records at node e, and ni is the number of records at node (partition) i. The splitting criterion used with INFO is to choose the split that achieves the most reduction (maximize GAIN). The decision tree based classifiers that handle large datasets improve the classification accuracy. Some proposed classifiers SLIQ and SPRINT use entire dataset for classification and are shown to be more accurate as compared to the classifiers that use sampled dataset or multiple partitions of the dataset. The decision tree model is built by recursively splitting the training set based on a locally optimal criterion until all or most of the records belonging to each of the partitions bear the same class label. Briefly, there are two phases to this process at each node of the decision tree. First phase determines the splitting decision and second phase splits the data. The very difference in the nature of continuous and categorical attributes requires them to be handled in different manners. The handling of categorical attributes in both phases is straightforward. Handling the continuous attributes is challenging. An efficient determination of the splitting decision used in most of the existing classifiers requires these attributes to be sorted on values. The classifiers such as CART and C4.5 perform sorting at every node of the decision tree, which makes them very expensive for large datasets, since this sorting has to be done out-of-core. The approach taken by SLIQ and SPRINT sorts the continuous attributes only once in the beginning. The splitting phase maintains this sorted order without requiring to sort the records again. The attribute lists are split in a consistent manner using a mapping between a record identifier and the node to which it belongs after splitting. SPRINT implements this mapping as a hash table, which is built on-the-fly for every node of the decision tree. The size of this hash table is proportional to the number of records at the node. For the upper levels of the tree, this number is O(N),where N is the number of records in the training set. If the hash table does not fit in the main memory, then SPRINT has to divide the splitting phase into several stages such that the hash table for each of the phases fits in the memory. This requires multiple passes over each of the attribute lists causing expensive disk I/O. In the following table, we summarize the features and disadvantages of the some of the decision tree based classifiers. 19 Classification Technique C4.5 SLIQ Features Simple depth-first construction. Sorts Continuous Attributes at each node. Needs entire data to fit in memory. Unsuitable for Large Datasets; Needs out-of-core sorting. Classification Accuracy shown to improve when entire datasets are used! The arrays of the continuous attributes are pre-sorted. The classification tree is grown in a breadth-first fashion. Class List structure maintains the record-id to node mapping. Split determining Phase: Class List is referred to for computing the best split for each individual attribute. Computations for all nodes are clubbed for efficiency purposes (hence, breadth-first strategy). Splitting Phase: The list of this splitting attribute is used to update the leaf labels in class list. (no physical splitting of attribute lists among nodes) Class List is frequently and randomly accessed in both the phases of tree induction. So, it is required to be in-memory all the time for efficient performance. This limits the size of largest training set. SPRINT Disadvantages The arrays of the continuous attributes are pre-sorted. The sorted order is maintained during each split. The classification tree is grown in a breadth-first fashion. Class information is clubbed with each attribute list. Attribute lists are physically split among nodes. Split determining phase is just a linear scan of lists at each node. Hashing scheme used in splitting phase, tids of the splitting attribute are hashed with the tree node as the key, and lookup table, remaining attribute arrays are split by querying this hash structure. Size of hash table is O(N) for top levels of the tree. If hash table does not fit in memory (mostly true for large datasets), then build in parts so that each part fits. Multiple expensive I/O passes over the entire dataset. 4.1.1.2 Decision Tree Pruning Generalize the tree by removing statistical dependence on a particular training set. The subtree with least estimated error rate is chosen leading to compact and accurate representation o Separate sets for training and pruning. o Same dataset for tree building and tree pruning. Cross-validation (build tree on each subset and prune it using the remaining). Not suitable for large datasets. All training samples for pruning – Minimum Description Length (MDL) based pruning 20 MDL Based Tree Pruning Cost(Model,Data) = Cost(Data|Model) + Cost(Model) o Cost is the number of bits needed for encoding. o Search for a least cost model. Encode data in terms of number of classification errors. Encode tree (model) using node encoding (number of children) plus splitting condition encoding. A node is pruned to have less number of children if removing some or all children gives smaller Cost. 4.1.2 Non-Decision Tree Based Classification 4.1.2.1 Neural Networks A neural network [30] is an analytic technique that mimics the working of the human brain. Such a network consists of nodes (the neurons) and connections with a certain "weight". By feeding the network with a training data set it adjusts its weights to achieve sufficiently accurate predictions for expected future data sets. The advantage of neural networks over decision-trees is the speed at which it can operate. This increase in speed can be obtained by using the inherent parallel computations of the network. A major disadvantage of the neural network is the inability to backtrack the decision making process; a decision is calculated on the result of the initialisation of the network from the training set. This is opposed to decision-tree methods where the path leading to a decision is easily backtracked. Input 1 Input 2 Output (Class) Input 3 Input 4 Input 5 Hidden Layer 21 4.1.2.2 Bayesian Classifiers The Bayesian Approach is a graphical model that uses directed arcs exclusively to form a directed acyclic graph'' [13]. Although the Bayesian approach uses probabilities and a graphical means of representation, it is also considered a type of classification. Bayesian networks are typically used when the uncertainty associated with an outcome can be expressed in terms of a probability. This approach relies on encoded domain knowledge and has been used for diagnostic systems. The basic features of Bayesian networks are 4.2 Each attribute and class label are random variables. Objective is to classify a given record of attributes (A1, A2, …, An) to class C s.t. P(C | A1, A2, …, An) is maximal. Naive Bayesian Approach: o Assume independence among attributes Ai. o Estimate P(Ai | Cj) for all Ai and Cj. o New point is classified to Cj if P(Cj) Pi P(Ai| Cj) is maximal. Generic Approach based on Bayesian Networks: o Represent dependencies using a direct acyclic graph (child conditioned on all its parents). Class variable is a child of all the attributes. o Goal is to get compact and accurate representation of the joint probability distribution of all variables. Learning Bayesian Networks is an active research area. Parallel Formulations of Classification Algorithms The memory limitations faced by serial classifiers and the need of classifying much larger datasets in shorter times make the classification algorithm an ideal candidate for parallelization. The parallel formulation, however, must address the issues of efficiency and scalability in both memory requirements and parallel runtime [18, 26, 28, 36]. In constructing a parallel classifier, two partitioning philosophies should be considered, Partitioning of data only, where large number of classification tree nodes gives high communication cost. Partitioning of classification tree nodes which satisfies the natural concurrency, but o it loads imbalance as the amount of work associated with each node varies, o child nodes use the same data as used by parent node which leads to loss of locality and high data movement cost 22 Categorical attributes algorithms are classified into three different classes. The features of these three classes are summarized below. Class Synchronous Tree Construction Approach Features no data movement required high communication cost as tree becomes bushy. Partitioned Tree Construction Approach Hybrid Algorithm processors work independently once partitioned completely. load imbalance and high cost of data movement. combines good features of two approaches. adapts dynamically according to the size and shape of trees. For continuous attributes, three different approaches are used, Sort continuous attributes at each node of the tree, as in C4.5. Discretize continuous attributes, as in SPEC [36] (Srivastava, Han, Kumar, and Singh, 1997) . Use a pre-sorted list for each continuous attributes as in SPRINT [35](Shafer, Agrawal, and Mehta, VLDB’96), and ScalParC [26](Joshi, Karypis, and Kumar, IPPS’98). Many parallel formulations of decision tree based classifiers have been developed [28, 35, 36]. Among these, the most relevant one is the parallel formulation of SPRINT, as it requires sorting of continuous attributes only once. SPRINT’s design allows it to parallelize the split determining phase effectively. The parallel formulation proposed for the splitting phase, however, is inherently unscalable in both memory requirements and runtime. It builds the required hash table on all processors by gathering the record_id-tonode mapping from all the processors. For this phase, the communication overhead per processor is O(N), where N is the number of records in the training set. Apart from the initial sorting phase, the serial runtime of a classifier is O(N). Hence, SPRINT is unscalable in run-time. It is un-scalable in memory requirements also, because the memory requirement per processor is O(N), as the size of the hash table is of the same order as the size of the training dataset for the upper levels of the decision tree, and it resides on every processor. 5 Clustering Clustering is grouping some points in some space into a small number of clusters [18, 24, 38, 41], each cluster consisting of points that are “near" in some sense. 23 Example 5.1: Documents may be thought of as points in a high-dimensional space, where each dimension corresponds to one possible word. The position of a document in a dimension is the number of times the word occurs in the document (or just 1 if it occurs, 0 if not). Clusters of documents in this space often correspond to groups of documents on the same topic. Example 5.2: Many years ago, during a cholera outbreak in London, a physician plotted the location of cases on a map. The data indicated that cases clustered around certain intersections, where there were polluted wells, not only exposing the cause of cholera, but indicating what to do about the problem. 5.1 Distance Measures To consider whether a set of points is close enough to be considered a cluster, we need a distance measure D(x, y) to tell us how far points x and y are. The usual axioms for a distance measure D are: D(x, x) = 0, a point is distance 0 from itself. D(x, y) =D(y, x), D is symmetric. D(x; y) D(x; z) +D(z; y), the triangle inequality. Often, our points may be thought to live in a k-dimensional Euclidean space, and the distance between any two points, say x = [x1; x2; : : :; xk] and y = [y1; y2; : : :; yk] is given in one of the usual manners: Common distance (“L2 norm"): k (x i 1 Manhattan distance (“L1 norm"): i yi ) 2 i yi | k | x i 1 Max of dimensions (“L norm"): max ik1 | xi y i | In some cases, when there is no Euclidean space to place the points, we need to have other forms of distance measures. Example 5.3: In DNA sequences, two sequences may be similar even though there are some insertions and deletions as well as changes in some characters. For the two sequences, abcde and bcdxye (they don't have any positions in common, and don't even have the same length), we can define the distance function D(x,y) = |x| + |y| - 2 |LCS(x,y)|, where LCS stands for the longest common subsequence of x and y. In our example, LCS(abcde; bcdxye) is bcde, of length 4, so D(abcde, bcdxye) =5+6- 2X 4 =3; i.e., the strings are fairly close. 5.2 Approaches to Clustering Clustering algorithms [24, 25, 38] are divided into two broad classes: 24 Centroid approaches. The centroid or central point of each cluster is estimated, and points are assigned to the cluster of their nearest centroid. Hierarchical approaches. Starting with each point as a cluster by itself, nearby clusters are repeatedly merged, The clustering algorithms are classified according to: whether or not they assume a Euclidean distance, and whether they use a centroid or hierarchical approach. The following three algorithms are examples of the above classification. BFR: Centroid based; assumes Euclidean measure, with clusters formed by a Gaussian process in each dimension around the centroid. GRGPF: Centroid-based, but uses only a distance measure, not a Euclidean space. CURE: Hierarchical and Euclidean, this algorithm deals with odd-shaped clusters. In the following subsections, we give some of the popular clustering algorithms. 5.3 Centroid Clustering 5.3.1 The k-Means Algorithm The k-Means algorithm [24, 25, 38] is a popular main-memory algorithm. k cluster centroids are picked and points are assigned to the clusters by picking the closest centroid to the point in question. As points are assigned to clusters, the centroid of the cluster may migrate. Example 5.4: In the following figure, we have five points in the 2-dimensional space. For k=2, points 1 and 2 are assigned to the two clusters, and become their centroids for the moment. 5 c 1 3 b 2 a 4 For point 3, suppose it is closer to 2, so 3 joins the cluster of 2, whose centroid moves to the point indicated as a. Suppose that point 4 is closer to 1 than to a, so 4 joins 1 in its cluster, whose center moves to b. Finally, 5 is closer to a than b, so it joins the cluster { 2, 3}, whose centroid moves to c. 25 5.3.2 The BFR Algorithm Based on k-means, this algorithm [38] reads its data once, consuming a main-memoryfull at a time. The algorithm works best if the clusters are normally distributed around a central point, perhaps with a different standard deviation in each dimension. Figure 17 suggests what the data belonging to a typical cluster in two-dimensions might look like. A centroid, marked by +, has points scattered around, with the standard deviation in the horizontal dimension being twice what it is in the vertical dimension. About 70% of the points will lie within the 1 ellipse; 95% will lie within 2, 99.9% within 3, and 99.9999% within 4. 1 2 + A cluster consists of: A central core, the Discard set (DS). This set of points is considered certain to belong to the cluster. All the points in this set are replaced by some simple statistics, described below. Although called “discarded" points, these points in truth have a significant effect throughout the running of the algorithm, since they determine collectively where the centroid is and what the standard deviation of the cluster is in each dimension. Surrounding sub-clusters, the Compression set (CS). Each sub-cluster in the CS consists of a group of points that are sufficiently close to each other that they can be replaced by their statistics, just like the DS for a cluster is. However, they are sufficiently far away from any cluster's centroid, that we are not yet sure which cluster they belong to. Individual points that are not part of a cluster or sub-cluster, the Retained set (RS). These points can neither be assigned to any cluster nor can they be grouped into a sub-cluster of the CS. They are stored in main memory, as individual points, along with the statistics of the DS and CS. The statistics used to represent each cluster of the DS and each sub-cluster of the CS are: The count of the number of points, N. The vector of sums of the coordinates of the points in each dimension. The vector is called SUM, and the component in the ith dimension is SUMi . The vector of sums of squares of the coordinates of the points in each dimension, called SUMSQ. The component in dimension i is SUMSQi . 26 For k dimensions, 2k +1 are needed to compute important statistics of a cluster or subcluster. The mean and variance in each dimension are The coordinate i of the centroid of the cluster in dimension i is SUMi / N. SUMSQi SUM i 2 The variance in dimension i is ( ) N N 5.3.3 Fastmap Fastmap [25] picks k pairs of points (ai, bi), each of which pairs serves as the “ends" of one of the k axes of the k-dimension space. Using the law of cosines, we can calculate the “projection" x of any point c onto the line ab, using only the distances between points, not any assumed coordinates of these points in a plane. The diagram is c D(b, c) D(a, c) a x b D(a, b) And the formula is x D 2 ( a , c ) D 2 ( a ,b ) D 2 ( b , c ) 2 D ( a ,b ) Having picked a pair of points (a, b) as an axis, part of the distance between any two points c and d is accounted for by the projections of c and d onto line ab, and the remainder of the distance is in other dimensions. If the projections of c and d are x and y, respectively, then in the future (as we select other axes), the distance Dcurrent(c, d) should be related to the given distance function D by 2 Dcurrent ( c, d ) D 2 ( c, d ) ( x y ) 2 d D(c, d) Dcurrent(c, d) c a b x y 27 The Fastmap algorithm computes for each point c, k projections; c1, c2,…, ck onto the k axes, which are determined by pairs of points (a1, b1), (a2, b2), …, (ak, bk). For i = 1, 2, …, k, do the following: Using the current distance Dcurrent, pick ai and bi, as follows: o Pick a random point c. o Pick ai to be the point as far as possible from c, using distance Dcurrent. o Pick bi to be the point as far as possible from ai. For each point x, compute xi , using the law-of-cosines formula described above. Change the definition of Dcurrent to subtract the distance in the ith dimension as well as previous dimensions. That is D current (x; y) D 2 ( x, y) ( X ( j ) y ( j ) ) 2 j i 5.4 Hierarchical Clustering Hierarchical clustering [25, 38] is a general technique that could take, in the worst case, O(n2) time to cluster n points. The General outlines of this approach are as follows. Start with each point in a cluster by itself. Repeatedly select two clusters to merge. In general, we want to pick the two clusters that are closest, but there are various ways we could measure “closeness." Some possibilities: o Distance between their centroids (or if the space is not Euclidean, between their clustroids). o Minimum distance between nodes in the clusters. o Maximum distance between nodes in the clusters. o Average distance between nodes of the clusters. End the merger process when we have \few enough" clusters. Possibilities: o Use a k-means approach -- merge until only k clusters remain. o Stop merging clusters when the only clusters that can result from merging fail to meet some criterion of compactness, e.g., the average distance of nodes to their clustroid or centroid is too high. 5.4.1 The GRGPF Algorithm This algorithm [25, 38] assumes there is a distance measure D, but no Euclidean space. It also assumes that there is too much data to fit in main memory. The data structure it uses to store clusters is like an R-tree. Nodes of the tree are disk blocks, and we store different things at leaf and interior nodes: In leaf blocks, we store cluster features that summarize a cluster in a manner similar to BFR. However, since there is no Euclidean space, the \features are somewhat different, as follows: o The number of points in the cluster, N. o The clustroid: that point in the cluster that minimizes the rowsum, i.e., the sum of the squares of the distances to the other points of the cluster. 28 If C is a cluster, C’ will denote its clustroid. Thus, the rowsum of the clustroid is D(C ' , X ). x in c Notice that the rowsum of the clustroid is analogous to the statistic SUMSQ that was used in BFR. However, SUMSQ is relative to the origin of the Euclidean space, while GRGPF assumes no such space. The rowsum can be used to compute a statistic, the radius of the cluster that is analogous to the standard deviation of a cluster in BFR. The formula is radius = rowsum / N o The p points in the cluster that are closest to the clustroid and their rowsums, for some chosen constant p. o The p points in the cluster that are farthest from the clustroid. In interior nodes, we keep samples of the clustroids of the clusters represented by the descendants of this tree node. An effort is made to keep the clusters in each subtree close. As in an R-tree, the interior nodes thus inform about the approximate region in which clusters at their descendants are found. When we need to insert a point into some cluster, we start at the root and proceed down the tree, choosing only those paths along which a reasonably close cluster might be found, judging from the samples at each interior node. 5.4.2 CURE The outlines of the CURE algorithm [38] are: Start with a main memory full of random points. Cluster these points using the hierarchical approach. For each cluster, choose c “sample" points for some constant c. These points are picked to be as dispersed as possible, then moved slightly closer to the mean, as follows: o Pick the first sample point to be the point of the cluster farthest from the centroid. o Repeatedly pick additional sample points by choosing that point of the cluster whose minimum distance to an already chosen sample point is as great as possible. o When c sample points are chosen, move all the samples toward the centroid by some fractional distance, e.g., 20% of the way toward the centroid. As a result, the sample points need not be real points of the cluster, but that fact is unimportant. The net effect is that the samples are “typical" points, well dispersed around the cluster, no matter what the cluster's shape is. Assign all points, including those involved in steps (1) and (2) to the nearest cluster, where “nearest" means shortest distance to some sample point. Example 5.5: For c=6, 6 sample points are picked from an elongated cluster, and then moved 20% of the way toward the centroid. 29 3 2 5 + 6 1 4 6. Conclusions In this paper, we have reviewed various data mining techniques. The basic features that most data mining techniques should consider are: all approaches deal with large amounts of data, efficiency is required due to volume of data, and accuracy is an essential element in the data mining process. Many effective data mining techniques in association mining, classification and clustering have been discussed. Parallelism is an important trend in all data mining tasks. Also, it could show a promising reorganization of the potential benefit of involvement in the data mining step to accelerate the whole KDD process and to improve the results. Most parallel data mining techniques are derived from sequential data mining techniques. The methodologies used in these techniques could lead to less and shorter iterations within the knowledge discovery process loop. But, dealing with the data mining process in the parallel and distributed environment should be through a new generation of data mining techniques that are designed especially for the parallel and distributed environment. Another issue related to the nature of data should be considered. The process of choosing the used data mining technique should depend on some parameters related to the distribution and homogeneity of data. Some techniques could give excellent performance with some data volumes, while the same techniques would give a poor performance with other data volumes. References [1] [2] [3] [4] R. Agrawal, T. Imielinski and A.Swami, “Mining Association Rules between sets of items in large databases,” Proc. ACM-SIGMOD Int. Conf. On Management of Data, Washington, D.C. 1993. R. Agrawal and J.C. Shafer, “Parallel Mining of Association Rules,” IEEE Trans. On Knowledge and Data Eng., 8(6):962-969, December 1996. R. Agrawal and R. Srikant, “Fast Algorithms for mining association rules,” Proc. Of 20th VLDB Conference, 1994. R. Agrawal and R. Srikant, "Mining Sequential Patterns", In Proc. 11th Intl. Conf. On Data Engineering, Taipi, Taiwan, March 1995. 30 [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] M. Ankerst, M. Ester and H.P. Kriegel, "Visual Classification: An Interactive Approach to Decision Tree Construction", Proc. Int. Conf. on Knowledge Discovery and Data Mining, pp. 392-397, 1999. M. Ankerst, M. Ester, H.P. Kriegel, "Towards an Effective Cooperation of the User and the Computer for Classification", Proc. Int. Conf. on Knowledge Discovery and Data Mining, pp. 178-188, 2000. M. Berry and G. Linoff, Data Mining Techniques (For Marketing, Sales, and Customer Support), John Wiley & Sons, 1997. P. Bollmann-Sdorra, A.M. Hafez and V.V. Raghavan, “A Theoretical Framework for Association Mining based on the Boolean Retrieval Model,” DaWaK 2001, September 2001. R. Brachmann and T. Anand, "The Process of Knowledge Discovery in Databases: A Human-Centered Approach," Advances in Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, CA, pp. 37-58. L. Breiman, J.H. Friedman, R.A.Olshen, and C.J. Stone. Classification and Regression Trees. Wadsworth, Belmont, 1984. S. Brin et al., “Dynamic Itemset Counting and Implication Rules for Market Basket Data,” Proc. ACM SIGMOD Conf. Management of Data, ACM Press, New York, 1997, pp. 255–264. W. Buntine, "A Guide To The Literature On Learning Probabilistic Networks From Data." IEEE Transactions on Knowledge and Data Engineering 8, 2 (Apr. 1996), 195-210. W. Buntine, Graphical Models For Discovering Knowledge. In Advances In Knowledge Discovery And Data Mining, eds. V. Cherkassky and F. Mulier, Learning from Data, John Wiley & Sons, 1998. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth and R. Uthurasamy (eds.), Advances in Knowledge Discovery and Data Mining, AAAI Press/ The MIT Press, 1996. U. Fayyad, G. Piatetsky-Shapiro and P. Smyth, "From Data Mining to Knowledge Discovery: An Overview," Advances in Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, CA, pp.1-30. U. Fayyad, G. Piatetsky-Shapiro and P. Smyth, "The KDD process for extracting useful knowledge from volumes of data," Communications of the ACM, 39(1). A. Freitas and S. Lavington, Mining Very Large Databases with Parallel Processing, Kluwer Academic Publishers, 1998. N. Friedman, D. Geiger and M. Goldszmidt, ‘Bayesian Network Classifiers,” Machine Learning 29:131-163, 1997. AM. Hafez, “A Dynamic Approach for Knowledge Discovery of Web Access Patterns”, ISMIS 2000, pp. 130-138. A.M. Hafez, “Association mining of dependency between time series,” Proceedings of SPIE Vol. 4384, SPIE AeroSense, April 2001. A.M. Hafez and V.V. Raghavan, "A Matrix Approach for Association Mining," the ISCA 10th International Conference on Intelligent Systems, June 13-15, 2001. E.H. Han, G. Karypis and V. Kumar, “Scalable Parallel Data Mining for Association Rules,” Proc. 1997 ACM-SIGMOD Int. Conf. On Management of Data, Tucson, Arizona, 1997. 31 [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] E.H. Han, G. Karypis, V.Kumar and B. Mobasher, “Clustering Based On Association Rule Hypergraphs,” SIGMOD’97 Workshop on Research Issues on Data Mining and Knowledge Discovery. A.K. Jain and R.C. Dubes, Algorithms for Clustering Data, Prentice Hall, 1988. M. Joshi, G. Karypis and V. Kumar, “ScalParC: A New Scalable and Efficient Parallel Classification Algorithms for Mining Large Datasets,” Proc. 12th International Parallel Processing Symposium (IPPS), Orlando, 1998. M. Joshi, G. Karypis and V. Kumar, “Parallel Algorithms for Sequential Associations: Issues and Challenges,” Minisymposium Talk at Ninth SIAM International Conference on Parallel Processing (PP’99), San Antonio, 1999. V. Kumar, A. Grama, A. Gupta and G.Karypis, Introduc-tion to Parallel Computing: Algorithm Design and Analysis. Benjamin-Cummings/Addison Wesley, Redwood City, CA, 1994. M. Mehta, R. Agarwal and J. Rissanen, “SLIQ: A fast scalable classifier for data mining,” In Proc. of 5th In-ternational Conference on Extending Database Technology (EBDT), Avignon, France, March 1996. D. Michie, D.J. Spiegelhalter and C. C. Taylor. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994. D. Pyle, Data Preparation for Data Mining, Morgan Kaufmann, 1999. J. R. Quinlan, C4.5: Programs for Machine Learning, Morgan Kaufmann, 1993. V.V. Raghavan and A.M. Hafez, “Dynamic Data Mining,“ IEA/AIE 2000, pp.220-229, 2000. A. Savasere, E. Omiecinski and S. Navathe, "An Efficient Algorithm for Mining Association Rules in large Databases," Proc. 21st Int'l Conf. of Very Large Data Bases, 1995 J. Shafer, R. Agarwal and M. Mehta, “SPRINT: A scalable parallel classifier for data mining,” In Proc. of 22nd Interna-tional Conference on Very Large Databases, Mumbai, India, September 1996. A. Srivastava, E.H. Han, V. Kumar and V. Singh, “Parallel Formulations of Decision-Tree Classification Algorithms,” Proc. 12th International Parallel Processing Symposium (IPPS), Orlando, 1998. R. Srikant and R. Agrawal, “Mining Sequential Patterns: Generalizations and Performance Improvements,” Proc. of 5th Int. Conf. On Extending Database Technology (EDBT), Avignon, France, 1996. A.K. Tung, J. Han, L.V. Lakshmanan and R.T. Ng, "Constraint-based Clustering in Large Databases," Proc. Int. Conf. on Database Theory, pp. 405-419, 2001. K. Wang, Y. He and J. Han, "Mining Frequent Itemsets Using Support Constraints," Proc. 26th Int. Conf. On very Large Data Bases, pp. 43-52, 2000. M. Ware, E. Frank, G. Holmes, M. Hall and I.H. Witten, "Interactive Machine Learning Letting Users Build Classifiers," http://www.cs.waikato.ac.nz/ ml/ publications.html S. M. Weiss and N. Indurkhya, Predictive Data Mining (a practical guide), Morgan Kaufmann Publishers,1998. P.C. Wong, "Visual Data Mining," IEEE Computer Graphics and Applications, Vol. 19(5), pp. 20-12, 1999. 32