Data Mining Course
Chapter 1
Introduction to Data Mining
Data Mining Course, Sharif University of Technology
1
Introduction to Data Mining
• Examples of Data Mining
–
–
–
–
–
Bank of America
13 million contact Bank of America’s call center each month
In past, customers listened to same marketing message
Whether relevant to customer or not
Kelly, VP Database Marketing states, “...we want to be as
relevant as possible to each customer”
– Customer profiles available to service representatives
– May suggest applicable products or services
– Data mining helps identify marketing approach based on
customer’s profile
Data Mining Course, Sharif University of Technology
2
Introduction to Data Mining
(cont’d)
– Homeland Security
– Shortly after 9/11/2001 events, FBI announced identification of
five terrorists in consumer database records
– One had 30 credit cards with $250,000 debt
– Another had 12 different addresses
– Former President Clinton concluded data should be proactively
searched
– Clinton said, “...they have 12 homes, they’re either really rich or
up to no good...shouldn’t be hard to figure out which.”
Data Mining Course, Sharif University of Technology
3
Introduction to Data Mining
(cont’d)
–
–
–
–
–
–
Gene Expression Database
In children, brain tumors represent deadly form of cancer
3,000 cases diagnosed per year
Children’s Memorial Hospital building gene expression database
Goal is developing more effective treatment
Bremer, Director of Brain Research, uses Clementine as initial
step in tumor identification
– Classification identifies one of 12 different tumor types
Data Mining Course, Sharif University of Technology
4
Trends leading to Data Flood
• More data is generated:
– Bank, telecom, other
business transactions ...
– Scientific data:
astronomy, biology, etc
– Web, text, and ecommerce
Data Mining Course, Sharif University of Technology
5
Big data examples
• Europe's Very Long Baseline Interferometry
(VLBI) has 16 telescopes, each of which
produces 1 Gigabit/second of astronomical
data over a 25-day observation session
– storage and analysis a big problem
• AT&T handles billions of calls per day
– so much data, it cannot be all stored -- analysis has
to be done “on the fly”, on streaming data
Data Mining Course, Sharif University of Technology
6
Largest databases in 2003
• Commercial databases:
– Winter Corp. 2003 Survey: France Telecom has
largest decision-support DB, ~30TB; AT&T ~ 26 TB
• Web
–
–
–
–
Alexa internet archive: 7 years of data, 500 TB
Google searches 4+ Billion pages, many hundreds TB
IBM WebFountain, 160 TB (2003)
Internet Archive (www.archive.org),~ 300 TB
Data Mining Course, Sharif University of Technology
7
Data growth rate
• Twice as much information was created in
2002 as in 1999 (~30% growth rate)
• Other growth rate estimates even higher
• Very little data will ever be looked at by a
human
• Knowledge Discovery is NEEDED to make
sense and use of data.
Data Mining Course, Sharif University of Technology
8
What is Data Mining?
• “…the process of discovering meaningful new correlations, patterns, and
trends by sifting through large amounts of data…” (Gartner Group)
• “…the analysis of observational data sets to find unsuspected relationships
and to summarize data in novel ways…” (Hand et al.)
• “Extraction of interesting (non-trivial, implicit, previously unknown and
potentially useful) patterns or knowledge from huge amount of data”, (Han
et al.)
• “…is an interdisciplinary field bringing together techniques from machine
•
learning, pattern recognition, statistics, databases, and visualization…”
(Cabana et al.)
Alternative names:
– Knowledge discovery (mining) in databases (KDD), knowledge
extraction, data/pattern analysis, data archeology, data dredging,
information harvesting, business intelligence, etc.
Data Mining Course, Sharif University of Technology
9
What is Data Mining?
(cont’d)
• Data mining chosen as one of top 10 emerging
technologies
(MIT Technology Review)
• “Data mining expertise is most sought after...” (1999
Information Week Survey)
• Brown, from BridgeGate LLC said, “Many companies
have implemented a data warehouse...starting to look at
what they can do with all that data”
Data Mining Course, Sharif University of Technology
10
What is Data Mining?
(cont’d)
• How widespread is data mining?
–
–
–
–
–
–
–
–
Boston Celtics listed employment position in 12/2003
Statistics Intern: Work with Basketball Operations
“Responsibilities include: ...data mining, etc.”
New York Nicks already using IBM’s Advanced Scout data mining
software
Software includes NBA’s game data in form of “events”
Each game includes statistics such as shots, passes, points,
rebounds, etc.
Against Chicago Bulls, software discovered pattern coaching staff
missed
16 of 29 NBA teams have turned to Advanced Scout to mine
play-by-play data
Data Mining Course, Sharif University of Technology
11
Why Data Mining?
• “...we are drowning in information but starved for
•
•
knowledge.” (Naisbitt, author Megatrends)
Not enough trained analysts available to translate data
into knowledge
Data mining fueled by several factors
–
–
–
–
Explosive growth in data collection
The storage of enterprise-wide data in data warehouses
Increased availability of Web clickstream data
The tremendous growth in computing power and storage
capacity
– Development of off-the-shelf commercial data mining software
products
Data Mining Course, Sharif University of Technology
12
Potential applications
• Data analysis and decision support
– Market analysis and management
• Target marketing, customer relationship management (CRM),
market basket analysis, cross selling, market segmentation
• Forecasting, customer retention, improved underwriting,
quality control, competitive analysis
– Fraud detection and detection of unusual patterns
• Other applications
– Text mining (news group, email, documents) and Web
mining
– Stream data mining
– DNA and bio-data analysis
Data Mining Course, Sharif University of Technology
13
Market Analysis and Management
•
Where does the data come from?
–
•
•
Target marketing
–
Find clusters of “model” customers who share the same characteristics: interest,
income level, spending habits, etc.
–
Determine customer purchasing patterns over time
Cross-market analysis
–
•
•
Associations/co-relations between product sales, & prediction based on such
association
Customer profiling
–
•
Credit card transactions, loyalty cards, discount coupons, customer complaint calls,
plus (public) lifestyle studies
What types of customers buy what products (clustering or classification)
Customer requirement analysis
–
identifying the best products for different customers
–
predict what factors will attract new customers
Provision of summary information
–
multidimensional summary reports
–
statistical summary information (data central tendency and variation)
Data Mining Course, Sharif University of Technology
14
Corporate Analysis & Risk Management
• Finance planning and asset evaluation
– cash flow analysis and prediction
– contingent claim analysis to evaluate assets
– cross-sectional and time series analysis (financial-ratio, trend
analysis, etc.)
• Resource planning
– summarize and compare the resources and spending
• Competition
– monitor competitors and market directions
– group customers into classes and a class-based pricing
procedure
– set pricing strategy in a highly competitive market
Data Mining Course, Sharif University of Technology
15
Fraud Detection & Mining Unusual Patterns
• Approaches: Clustering & model construction for frauds, outlier analysis
• Applications: Health care, retail, credit card service, telecomm.
– Auto insurance: ring of collisions
– Money laundering: suspicious monetary transactions
– Medical insurance
• Professional patients, ring of doctors, and ring of references
• Unnecessary or correlated screening tests
– Telecommunications: phone-call fraud
• Phone call model: destination of the call, duration, time of day or week. Analyze
patterns that deviate from an expected norm
– Retail industry
• Analysts estimate that 38% of retail shrink is due to dishonest employees
– Anti-terrorism
Data Mining Course, Sharif University of Technology
16
Other Applications
• Sports
– IBM Advanced Scout analyzed NBA game statistics (shots blocked, assists, and
fouls) to gain competitive advantage for New York Knicks and Miami Heat
• Astronomy
– JPL and the Palomar Observatory discovered 22 quasars with the help of data
mining
• Internet Web Surf-Aid
– IBM Surf-Aid applies data mining algorithms to Web access logs for marketrelated pages to discover customer preference and behavior pages, analyzing
effectiveness of Web marketing, improving Web site organization, etc.
Data Mining Course, Sharif University of Technology
17
Data Mining: A KDD Process
Pattern Evaluation
– Data mining—core of
knowledge discovery
Data Mining
process
Task-relevant Data
Data Warehouse
Selection
Data Cleaning
Data Integration
Databases
Data Mining Course, Sharif University of Technology
18
Steps of a KDD Process
• Learning the application domain
•
•
•
•
•
•
•
•
– relevant prior knowledge and goals of application
Creating a target data set: data selection
Data cleaning and preprocessing: (may take 60% of effort!)
Data reduction and transformation
– Find useful features, dimensionality/variable reduction, invariant
representation.
Choosing functions of data mining
– summarization, classification, regression, association, clustering.
Choosing the mining algorithm(s)
Data mining: search for patterns of interest
Pattern evaluation and knowledge presentation
– visualization, transformation, removing redundant patterns, etc.
Use of discovered knowledge
Data Mining Course, Sharif University of Technology
19
The Need for Human Direction of
Data Mining
– Some early data mining definitions described process as
“automatic”
– “…this has misled many people into believing data mining is
product that can be bought rather than a discipline that must be
mastered.” (Berry, Linoff)
– Automation no substitute for human input
– Data mining is easy to do badly
– Understanding statistical and mathematical model structures of
underlying software required
– Humans need to be actively involved in every phase of data
mining process
– Task of data mining should be integrated into human process of
problem solving
Data Mining Course, Sharif University of Technology
20
Cross Industry Standard Process:
CRISP-DM
• Cross-Industry Standard Process for Data Mining (CRISPDM) developed in 1996
–
–
–
–
–
Contributors include DaimlerChrysler, SPSS, and NCR
Developed to fit data mining into general business strategy
Process vendor and tool-neutral
Non-proprietary and freely available
Data mining projects follow iterative, adaptive life cycle
consisting of 6 phases
– Phase sequences are adaptive
– Next, Figure 1.1 illustrates CRISP-DM lifecycle
Data Mining Course, Sharif University of Technology
21
Cross Industry Standard Process:
CRISP-DM (cont’d)
– Iterative CRIP-DM process
shown in outer circle
– Most significant
dependencies between
phases shown
Business / Research
Understanding Phase
Deployment Phase
– Next phase depends on
results from preceding
phase
Evaluation Phase
Data Understanding
Phase
Data Preparation
Phase
Modeling Phase
– Returning to earlier phase
possible before moving
forward
Data Mining Course, Sharif University of Technology
22
Cross Industry Standard Process:
CRISP-DM (cont’d)
• (1) Business Understanding Phase
– Define business requirements and objectives
– Translate objectives into data mining problem definition
– Prepare initial strategy to meet objectives
• (2) Data Understanding Phase
– Collect data
– Assess data quality
– Perform exploratory data analysis (EDA)
• (3) Data Preparation Phase
– Cleanse, prepare, and transform data set
– Prepares for modeling in subsequent phases
– Select cases and variables appropriate for analysis
Data Mining Course, Sharif University of Technology
23
Cross Industry Standard Process:
CRISP-DM (cont’d)
• (4) Modeling Phase
– Select and apply one or more modeling techniques
– Calibrate model settings to optimize results
– If necessary, additional data preparation may be required
• (5) Evaluation Phase
– Evaluate one or more models for effectiveness
– Determine whether defined objectives achieved
– Make decision regarding data mining results before deploying to
field
Data Mining Course, Sharif University of Technology
24
Cross Industry Standard Process:
CRISP-DM (cont’d)
• (6) Deployment Phase
– Make use of models created
– Simple deployment: generate report
– Complex deployment: implement additional data mining effort in
another department
– In business, customer often carries out deployment based on
model
• See http://www.crisp-dm.org for more information
Data Mining Course, Sharif University of Technology
25
Case Study 1
• Analyzing Automobile Warranty Claims
– Business Understanding
– Objectives include improving customer satisfaction and reducing
costs
– Manufacturing engineers consulted to formulate business
problems
– Data mining techniques used to uncover possible issues:
• Warranty claim interdependencies?
• Past claims associated with future claims?
• Association between claim and repair facility?
Data Mining Course, Sharif University of Technology
26
Case Study 1
(cont’d)
– Data Understanding
– 40GB QUIS database containing 7 million vehicle records used
– Vehicle records include manufacturing location, warrant claims,
and additional codes
– Database unintelligible to non-domain experts
– Costly effort to consult with domain experts from different
departments
– Data Preparation
– QUIS discovered to have limited SQL access
– Cases and variables manually extracted
– Additional variables derived for modeling phase
Data Mining Course, Sharif University of Technology
27
Case Study 1
(cont’d)
–
–
–
–
–
Proprietary data mining software used
Data format requirements varied for different algorithms
Resulted in exhaustive pre-processing of data
Modeling Phase
Applied Bayesian networks and association rules to uncover
dependencies between warranty claims
– Discovered specific combination of construction specifications
doubles probability of electrical cable claim
– Investigated whether some garages had more claims than others
– Remaining results confidential
Data Mining Course, Sharif University of Technology
28
Case Study 1
(cont’d)
–
–
–
–
–
–
–
Evaluation
Researchers disappointed in results
Association rules could not be generalized
Rules “not interesting” according to domain experts
Data models fell short of business objectives
Legacy databases not suited to data mining
Proposal suggested database redesign for future data mining
efforts
– Deployment
– Foregoing effort identified as pilot project, models not deployed
– Future data mining efforts planned to integrate more closely to
database systems at DaimlerChrysler
Data Mining Course, Sharif University of Technology
29
Case Study 1
(cont’d)
–
–
–
–
–
–
–
–
Summary
Uncovering hidden nuggets very difficult
During each phase, researchers encountered roadblocks
Applying new data mining effort problematic
Data mining effort requires management support
Substantial human participation required at every stage
Installation, configuration, and data mining modeling not magic
Wrong analysis leads to possibly expensive policy
recommendations
– No guarantee data mining effort delivers actionable results
– However, used properly, data mining may provide profitable
results
Data Mining Course, Sharif University of Technology
30
Fallacies of Data Mining
• Four Fallacies of Data Mining (Louie, Nautilus Systems, Inc.)
–
–
–
–
–
–
–
Fallacy 1
Set of tools can be turned loose on data repositories
Finds answers to all business problems
Reality 1
No automatic data mining tools solve problems
Rather, data mining is process (CRISP-DM)
Integrates into overall business objectives
– Fallacy 2
– Data mining process is autonomous
– Requires little oversight
Data Mining Course, Sharif University of Technology
31
Fallacies of Data Mining
(cont’d)
–
–
–
–
Reality 2
Requires significant intervention during every phase
After model deployment, new models require updates
Continuous evaluative measures monitored by analysts
–
–
–
–
–
Fallacy 3
Data mining quickly pays for itself
Reality 3
Return rates vary
Depending on startup, personnel, data preparation costs, etc.
– Fallacy 4
– Data mining software easy to use
Data Mining Course, Sharif University of Technology
32
Fallacies of Data Mining
(cont’d)
– Reality 4
– Ease of use varies across projects
– Analysts must combine subject matter knowledge with specific
problem domain
• Two Additional Fallacies (Larose)
–
–
–
–
–
Fallacy 5
Data mining identifies causes of business problems
Reality 5
Knowledge discovery process uncovers patterns of behavior
Humans interpret results and identify causes
Data Mining Course, Sharif University of Technology
33
Fallacies of Data Mining
–
–
–
–
–
–
(cont’d)
Fallacy 6
Data mining automatically cleans data in databases
Reality 6
Data mining often uses data from legacy systems
Data possibly not examined or used in years
Organizations starting data mining efforts confronted with huge
data preprocessing task
Data Mining Course, Sharif University of Technology
34
What Tasks Can Data Mining
Accomplish?
• Six common data mining tasks
–
–
–
–
–
–
Description
Estimation
Prediction
Classification
Clustering
Association
• (1) Description
– Describes patterns or trends in data
– For example, pollster may uncover patterns suggesting those
laid-off less likely to support incumbent
– Descriptions of patterns, often suggest possible explanations
Data Mining Course, Sharif University of Technology
35
What Tasks Can Data Mining
Accomplish? (cont’d)
– For example, those laid-off now less financially secure;
therefore, prefer alternate candidate
– Data mining models should be transparent
– That is, results should be interpretable by humans
– Some data mining methods more transparent than others
– For example, Decision Trees (transparent) <-> Neural Networks
(opaque)
– High-quality description accomplished using Exploratory Data
Analysis (EDA)
– Graphical method of exploring patterns and trends in data
Data Mining Course, Sharif University of Technology
36
What Tasks Can Data Mining
Accomplish? (cont’d)
• (2) Estimation
– Similar to Classification task, except target variable numeric
– Models built from complete data records
– Records include values for each predictor field and numeric
target variable in training set
– For new observations, estimate of target variable made
– For example, estimate a patient’s systolic blood pressure, based
on patient’s age, gender, body-mass index, and sodium levels
– Here, estimation model built from training set records
– Model then estimates value for new case
Data Mining Course, Sharif University of Technology
37
What Tasks Can Data Mining
Accomplish? (cont’d)
– Estimation Tasks in Business and Research:
– Estimate amount of money, family of four will spend on back-toschool shopping
– Estimate percentage decrease in rotary movement sustained to
NFL player with knee injury
– Estimate number of points basketball player scores when
double-teamed in playoffs
– Estimate GPA of graduate student, based on student’s
undergraduate GPA
Data Mining Course, Sharif University of Technology
38
What Tasks Can Data Mining
Accomplish? (cont’d)
– Figure 1.2 shows scatter plot of graduate GPA against
undergraduate GPA
– Linear regression finds line (blue) best approximating
relationship between two variables
– Regression line estimates student’s graduate GPA based on their
undergraduate GPA
Data Mining Course, Sharif University of Technology
39
What Tasks Can Data Mining
Accomplish? (cont’d)
– Minitab statistical software produces regression
equation ŷ = 1.24 + 0.67x
– Therefore, estimated student’s graduate GPA = 1.24 plus 0.67
times their undergraduate GPA
–
–
–
–
For example, suppose student’s undergraduate GPA = 3.0
According to estimation model
Estimated student’s graduate GPA = 1.24 + 0.67(3.0) = 3.25
Point (x = 3.0, ŷ = 3.25) lies on regression line
– Statistical Analysis uses several estimation methods: point
estimation, confidence interval estimation, linear regression and
correlation, and multiple regression
Data Mining Course, Sharif University of Technology
40
What Tasks Can Data Mining
Accomplish? (cont’d)
• (3) Prediction
– Similar to classification and estimation, except results lie in the
future
– Prediction Tasks in Business and Research:
– Predict price of stock 3 months into future, based on past
performance
?
Stock
Price
?
?
Q1
Q2
Q3
Q4
Data Mining Course, Sharif University of Technology
41
What Tasks Can Data Mining
Accomplish? (cont’d)
– Predict percentage increase in traffic deaths next year, if speed
limit increased
– Predict whether molecule in newly discovered drug leads to
profitable pharmaceutical drug
– Methods used for classification and estimation applicable to
prediction
– Includes point estimation, confidence interval estimation, linear
regression and correlation, and multiple regression
Data Mining Course, Sharif University of Technology
42
What Tasks Can Data Mining
Accomplish? (cont’d)
• (4) Classification
– Classification requires categorical target variable such as Income
Bracket
– Three values include “High”, “Middle”, “Low”
– Data model examines records containing input fields and target
field
– Table shows several records from data set
Subject
Age
Gender
Occupation
Income Bracket
001
47
F
Software Engineer
High
002
28
M
Marketing Consultant
Middle
003
35
M
Unemployed
Low
…
…
…
…
…
Data Mining Course, Sharif University of Technology
43
What Tasks Can Data Mining
Accomplish? (cont’d)
– Records of persons in data set used to “train” classification
model
– First, Model built from data records, where value of categorical
target variable (Income Bracket) already known
– Algorithm “first learns about” which combinations of input fields
are associated with Income Bracket values in training set
– For example, algorithm may determine that older females
associated with high income
– Next, trained model examines new records
– Information regarding Income Bracket not available
Data Mining Course, Sharif University of Technology
44
What Tasks Can Data Mining
Accomplish? (cont’d)
– Based on classifications in training set, new records classified
– For example, 63-year old female professor might be classified in
“High” income bracket
– Classification Tasks in Business and Research:
– Determine whether credit card transaction fraudulent
– Assessing mortgage application to determine “good” or “bad”
credit risk
– Diagnosing whether particular disease present
Data Mining Course, Sharif University of Technology
45
What Tasks Can Data Mining
Accomplish? (cont’d)
– Determine if will written by deceased, or fraudulently by
someone else
– Identify whether certain financial behavior represents terrorist
threat
– Scatter plot shows Na/K ratio against Age for 200 patients
– For example, classify drug type to prescribe based on patient’s
age and sodium/potassium ratio
Data Mining Course, Sharif University of Technology
46
What Tasks Can Data Mining
Accomplish? (cont’d)
– Actual drug type prescribed symbolized by shade (light, medium,
dark) of points
– Suppose prescription of new patient based on this data set?
– Prescribe which drug for young patient with high Na/K ratio?
– Young patients plotted on left
– High Na/K plotted on upper-half
– Quadrant of graph shows light points
– Recommended drug = Y (corresponds to light points)
– Prescribe which drug for older patient with low Na/K ratio?
– Lower-right half of graph shows patients prescribed different
drug types
Data Mining Course, Sharif University of Technology
47
What Tasks Can Data Mining
Accomplish? (cont’d)
– Definitive classification cannot be made
– More information required to make decision
– Examples show graphs are helpful for understanding twodimensional data
– However, classification often requires many input attributes
– More sophisticated methods of classification required
– Commonly used algorithms for classification include k-Nearest
Neighbor, Decision Trees, and Neural Networks
Data Mining Course, Sharif University of Technology
48
What Tasks Can Data Mining
Accomplish? (cont’d)
• (5) Clustering
– Refers to grouping records into classes of similar objects
– Clustering algorithm seeks to segment data set into
homogeneous subgroups
– Where similarity of records in clusters maximized, and similarity
to records outside clusters minimized
– Target variable not specified
– For example, Claritas, Inc. PRIZM software clusters demographic
profiles for different geographic areas according to zip code
Data Mining Course, Sharif University of Technology
49
What Tasks Can Data Mining
Accomplish? (cont’d)
– Table shows 62 distinct “lifestyle” types used by PRIZM
01 Blue Blood Estates
02 Winner's Circle
03 Executive Suites
04 Pools & Patios
05 Kids & Cul-de-Sacs
06 Urban Gold Coast
07 Money & Brains
08 Young Literati
09 American Dreams
10 Bohemian Mix
11 Second City Elite
12 Upward Bound
13 Gray Power
14 Country Squires
15 God's Country
16 Big Fish, Small Pond
17 Greenbelt Families
18 Young Influentials
19 New Empty Nests
20 Boomers & Babies
21 Suburban Sprawl
22 Blue-Chip Blues
23 Upstarts & Seniors
24 New Beginnings
25 Mobility Blues
26 Gray Collars
27 Urban Achievers
28 Big City Blend
29 Old Yankee Rows
30 Mid-City Mix
31 Latino America
32 Middleburg Managers
33 Boomtown Singles
34 Starter Families
35 Sunset City Blues
36 Towns & Gowns
37 New Homesteaders
38 Middle America
39 Red, White & Blues
40 Military Quarters
41 Big Sky Families
42 New Eco - topia
43 River City, USA
44 Shotguns & Pickups
45 Single City Blues
46 Hispanic Mix
47 Inner Cities
48 Smalltown Downtown
49 Hometown Retired
50 Family Scramble
51 Southside City
52 Golden Ponds
53 Rural Industria
54 Norma Rae-Ville
55 Mines & Mills
56 Agri - Business
57 Grain Belt
58 Blue Highways
59 Rustic Elders
60 Back Country Folks
61 Scrub Pine Flats
62 Hard Scrabble
Data Mining Course, Sharif University of Technology
50
What Tasks Can Data Mining
Accomplish? (cont’d)
– What do the clusters mean?
– According to PRIZM, Clusters for Beverly Hills, CA 90210 include:
•
•
•
•
Cluster
Cluster
Cluster
Cluster
01:
10:
02:
08:
Blue Blood Estates
Bohemian Mix
Winner’s Circle
Young Literati
– Description of Cluster 01, “...’old money’ heirs that live in
America’s wealthiest suburbs...accustomed to privilege and live
luxuriously...”
Data Mining Course, Sharif University of Technology
51
What Tasks Can Data Mining
Accomplish? (cont’d)
– Clustering Tasks in Business and Research:
– Target marketing niche product for small business that does not
have large marketing budget
– For accounting purposes, segment financial behavior into benign
and suspicious categories
– Use as dimensionality-reduction tool for data set having several
hundred inputs
– Determine gene expression clusters, where many genes exhibit
similar behavior or characteristics
– Clustering often used as preliminary step in data mining
– Resulting clusters used as input to different technique
downstream, such as neural networks
Data Mining Course, Sharif University of Technology
52
What Tasks Can Data Mining
Accomplish? (cont’d)
• (6) Association
– Find out which attributes “go together”
– Market Basket Analysis commonly used in business applications
– Quantify relationships in the form of Rules
IF antecedent THEN consequent
– Rules measured using support and confidence
– For example, discover which items in supermarket are purchased
together
– Thursday night 200 of 1,000 customers bought diapers, and of
those buying diapers, 50 purchased beer
– Association Rule: “IF buy diapers, THEN buy beer”
– Support = 200/1,000 = 5%, and confidence = 50/200 = 25%
Data Mining Course, Sharif University of Technology
53
What Tasks Can Data Mining
Accomplish? (cont’d)
– Association Tasks in Business and Research:
– Investigating proportion of subscribers to cell phone plan
responding positively to service upgrade offer
– Predicting degradation in telecommunication networks
– Discovering which items in supermarket purchased together
– Determining proportion of cases where administering new drug
exhibits serious side effects
– Two commonly-used algorithms for generating association rules
– A Priori and Generalized Rule Induction (GRI)
Data Mining Course, Sharif University of Technology
54
Case Study 2
• Predicting Abnormal Stock Market Returns
– Business Understanding
– Alan Safer reports stock market trades by insiders have
abnormal returns
– Profits by outsiders can be increased, by using legal insider
trading information
– Safer attempts to predict abnormal stock price returns
– Data Preparation
– Rank of company insiders not considered important
– Also, company insiders omitted where not involved in company
decisions
Data Mining Course, Sharif University of Technology
55
Case Study 2
–
–
–
–
(cont’d)
Modeling
Data split training = 80% and validation = 20%
Neural Network model uncovered results:
(a) Several groups had most predictable abnormal returns:
•
•
•
•
Electronic equipment, excluding computer equipment
Chemical products
Transportation equipment
Business services
– (b) Predictions looking farther into future increased ability to
predict unusual variations
– (c) Abnormal stock returns of small companies easier to predict
Data Mining Course, Sharif University of Technology
56
Case Study 2
(cont’d)
– Evaluation
– Multivariate Adaptive Regression Spline (MARS) also applied to
data
– Uncovered similar findings to Neural Network model
– Confluence of results is powerful method of evaluating validity of
model
– This increases confidence in results
– Deployment
– Safer published findings in Intelligent Data Analysis
Data Mining Course, Sharif University of Technology
57
Case Study 3
• Mining Association Rules from Legal Databases
– Business Understanding
– Ivkovic, Yearwood, and Stranieri mine association rules from
large database of applicants for government-funded legal aide in
Australia
– Legal data highly unstructured
– Goal is to improve delivery of legal services
– Data Understanding
– Data provided by Victoria Legal Aid (VLA)
– Contains 380,000 applications with 300 attributes
Data Mining Course, Sharif University of Technology
58
Case Study 3
(cont’d)
– Domain experts consulted in effort to reduce dimensionality
– Researchers selected seven of the most important inputs for
inclusion in data set
– Data Preparation
– VLA data set relatively clean
– VLA database administration system responsible for high-quality
data
– Modeling
– Rules restricted to having both single antecedent and
consequent
– Many interesting and uninteresting rules uncovered
Data Mining Course, Sharif University of Technology
59
Case Study 3
(cont’d)
– Researchers adopted premise that interesting rules spawn
interesting hypotheses
– For example, possible reasons for rule “If place of birth =
Vietnam, then law type = criminal law” include:
• Vietnamese applicants applied for criminal law assistance only
• Vietnamese applicants committed more crimes than other groups
• Perhaps high proportion of males applied, and males more closely
•
associated with criminal activity
Vietnamese didn’t have access to VLA promotional material
– Researchers concluded first hypothesis most likely
– Note intense human activity required for data mining process
Data Mining Course, Sharif University of Technology
60
Case Study 3
(cont’d)
– Evaluation
– Three domain experts estimated confidence level of 144 rules
– These estimates compared to confidence level reported by
software
– Deployment
– Web-based application developed (WebAssociator)
– Non-specialists able to access rule-building engine
– Researchers suggest WebAssociator deployment to enhance
judicial system
– May identify unjust processes
Data Mining Course, Sharif University of Technology
61
Case Study 4
• Predicting Corporate Bankruptcies using Decision Trees
– Business Understanding
– Recent economic crisis has spawned many corporate
bankruptcies in East Asia
– Sung, Chang, and Lee developing models predicting
bankruptcies that maximize interpretability of results
– Therefore, decision trees used for modeling
– Data Understanding
– Data included two groups of Korean companies
– Those that went bankrupt 1991-1995 during stable period
– Companies that went bankrupt during crisis years 1997-1998
Data Mining Course, Sharif University of Technology
62
Case Study 4
(cont’d)
–
–
–
–
–
–
–
–
29 firms identified, mostly manufacturing
Financial data collected from Korean Stock Exchange
Data Preparation
56 financial ratios identified through literature search
16 dropped due to duplication
Measures included growth, profitability, etc.
Modeling
Separate decision tree models were applied under “normal” and
“crisis” conditions
– Normal-conditions rules uncovered:
• If productivity of capital > 19.65, then predict non-bankrupt with
86% confidence
Data Mining Course, Sharif University of Technology
63
Case Study 4
(cont’d)
• If productivity of capital <= 19.65 and ratio of cash flow to total
assets <= -5.65, then predict bankrupt with 84% confidence
– Crisis-conditions rules uncovered:
• If productivity of capital > 20.61, predict non-bankrupt with 95%
•
confidence
If ratio of cash flow to liabilities > 2.54, predict non-bankrupt with
85% confidence
– “Cash flow” and “productivity of capital” important predictors,
regardless of economic conditions
– Evaluation
– Panel of domain experts concluded “productivity of capital” most
important attribute for differentiating firms at risk
– Domain experts verified results of decision tree
Data Mining Course, Sharif University of Technology
64
Case Study 4
(cont’d)
– Group confirmed results would generalize to population of
Korean manufacturing firms
– Discriminant analysis determined many of the 40 financial ratios
were important predictors
– Deployment
– No specific deployment took place
– However, financial institutions in Korea became more aware of
important predictors of bankruptcy
Data Mining Course, Sharif University of Technology
65
Case Study 5
• Profiling the Tourism Market using k-Means Clustering
Analysis
– Business Understanding
– Hudson and Richie were interested in studying intra-province
tourism behavior in Alberta, Canada
– Goal was development of marketing campaign for tourism in
Alberta (sponsored by Travel Alberta)
– Models created in effort to quantify factors for choosing
vacations in Alberta
– Data Understanding
– Data collected from 13,445 Albertans using phone survey in
1999
– Only 3,071/13,445 records included in modeling
Data Mining Course, Sharif University of Technology
66
Case Study 5
(cont’d)
– Data Preparation
– One question asked respondents to indicate which of 13 factors
most influenced their travel decisions
– Factors included accommodations, weather conditions, etc.
– Modeling
– Between two and six clusters explored with k-Means
– Five-cluster solution chosen with profile names:
•
•
•
•
•
Young Urban Outdoor Market
Indoor Leisure Traveler Market
Children-first Market
Fair-weather-friends Market
Older, Cost-conscious Traveler Market
Data Mining Course, Sharif University of Technology
67
Case Study 5
(cont’d)
–
–
–
–
–
Evaluation
Discriminant analysis verified “reality” of clusters
Classified 93% correctly
Deployment
Findings resulted in launch of new campaign, “Alberta, Made to
Order”
– More than 80 projects launched
– Travel Alberta found increase of 20% in number of Albertans
considering Alberta “top-of-the-mind” travel destination
Data Mining Course, Sharif University of Technology
68
Summary
• Data mining: discovering interesting patterns from large amounts of data
• A natural evolution of database technology, in great demand, with wide
applications
• A KDD process includes data cleaning, data integration, data selection,
transformation, data mining, pattern evaluation, and knowledge
presentation
• Mining can be performed in a variety of information repositories
• Data mining functionalities: characterization, discrimination, association,
classification, clustering, outlier and trend analysis, etc.
• Data Mining tasks
• Case studies
Data Mining Course, Sharif University of Technology
69
Readings
• Chapter 1 Larose book
• Chapters 1 Han and Kamber book
• Intro to Data Mining and Knowledge
Discovery article
• DataMining-1.html
• From Data Mining to Knowledge Discovery
in Databases article
• Data Mining an overview article
Data Mining Course, Sharif University of Technology
70
A Brief History of Data Mining Society
• 1989 IJCAI Workshop on Knowledge Discovery in Databases (Piatetsky-Shapiro)
–
Knowledge Discovery in Databases (G. Piatetsky-Shapiro and W. Frawley, 1991)
• 1991-1994 Workshops on Knowledge Discovery in Databases
–
Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R.
Uthurusamy, 1996)
• 1995-1998 International Conferences on Knowledge Discovery in Databases and Data
Mining (KDD’95-98)
–
Journal of Data Mining and Knowledge Discovery (1997)
• 1998 ACM SIGKDD, SIGKDD’1999-2001 conferences, and SIGKDD Explorations
• More conferences on data mining
–
PAKDD (1997), PKDD (1997), SIAM-Data Mining (2001), (IEEE) ICDM (2001), etc.
Data Mining Course, Sharif University of Technology
71
Where to Find References?
• Data mining and KDD (SIGKDD: CDROM)
–
Conferences: ACM-SIGKDD, IEEE-ICDM, SIAM-DM, PKDD, PAKDD, etc.
–
Journal: Data Mining and Knowledge Discovery, KDD Explorations
• Database systems (SIGMOD: CD ROM)
–
Conferences: ACM-SIGMOD, ACM-PODS, VLDB, IEEE-ICDE, EDBT, ICDT, DASFAA
–
Journals: ACM-TODS, IEEE-TKDE, JIIS, J. ACM, etc.
• AI & Machine Learning
–
Conferences: Machine learning (ML), AAAI, IJCAI, COLT (Learning Theory), etc.
–
Journals: Machine Learning, Artificial Intelligence, etc.
• Statistics
–
Conferences: Joint Stat. Meeting, etc.
–
Journals: Annals of statistics, etc.
• Visualization
–
Conference proceedings: CHI, ACM-SIGGraph, etc.
–
Journals: IEEE Trans. visualization and computer graphics, etc.
Data Mining Course, Sharif University of Technology
72
Recommended Reference Books
•
R. Agrawal, J. Han, and H. Mannila, Readings in Data Mining: A Database Perspective, Morgan
Kaufmann (in preparation)
•
U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy. Advances in Knowledge Discovery
and Data Mining. AAAI/MIT Press, 1996
•
U. Fayyad, G. Grinstein, and A. Wierse, Information Visualization in Data Mining and Knowledge
Discovery, Morgan Kaufmann, 2001
•
J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann, 2001
•
D. J. Hand, H. Mannila, and P. Smyth, Principles of Data Mining, MIT Press, 2001
•
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference,
and Prediction, Springer-Verlag, 2001
•
T. M. Mitchell, Machine Learning, McGraw Hill, 1997
•
G. Piatetsky-Shapiro and W. J. Frawley. Knowledge Discovery in Databases. AAAI/MIT Press, 1991
•
S. M. Weiss and N. Indurkhya, Predictive Data Mining, Morgan Kaufmann, 1998
•
I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java
Implementations, Morgan Kaufmann, 2001
Data Mining Course, Sharif University of Technology
73