Data Mining
The Mining Analogy
• Data mining gains its name to some degree its popularity, by playing
off a meaning that the data you have stored is much like a ‘mountain’
and that buried within the mountain (just as buried within your data)
are certain ‘gems’ of great value
• The problem is that there are also lots of non-valuable rocks and rubble
in the mountain that need to be mined through and discarded in order
to get that which is valuable.
• The trick is that both for mountains of rock and mountains of data you
need some power tools to unearth the value of the data
• For rock, this means earthmovers and dynamite; for data, this means
powerful computers and data mining software
What Data Mining Isn’t
• Statistic
– Statistical tools take longer to run
– Less robust on messy real world data
– Must often be wielded by a master craftsman
• OLAP (Online Analytical Processing)
– OLAP provides a tool for looking quickly anywhere
within the mountain
– What it doesn’t tell you is What is valuable and What
isn’t
• Data Warehousing
Data Mining has come of age
• What is data mining? And why are so many people talking about it in
both the computer industry and in direct marketing?
• The answer is simple: data mining helps end users extract useful
business information from large databases.
• What is so new about extracting information from data to make your
business run better?
• The allure of data mining is that it promise to fix the problem of
miscommunication between you and your data, and allow you to ask
complex questions of your data such as
– What has been going on?
– What is going to happen next and how can I profit?
Data Mining has come of age
(cont’d)
• What has been going on?
– To answer this question can be provided by the data warehouse and
multidimensional database technology that allow the user to easily
navigate and visualize the data
• What is going to happen next and how can I profit?
– The answer to this question can be provided by data mining tools
built on some of the latest computer algorithms: Decision Tree
(CART, CHAID, AID), Neural Networks, Nearest Neighbor, and
Rule Induction
Learning from Your Past
Mistakes
• Those who cannot remember the past are
condemned to repeat it [G.Santayana]
• How does data mining work?
• It works the same way as a human being does.
• It uses historical information (experience) to learn
from the past.
• The trick to building a successful predictive model
is to have some data in your database that
describes what has happened in the past. (see
details at page. 96)
Measuring Data Mining Effectiveness –
Accuracy, Speed, Cost
• To make the right choice of the data mining tool, they need
to evaluate it in comparison to existing statistical
techniques and also compare among the large number of
new data mining products that are currently on the market.
• Data mining technology is actually quite similar to
statistics in the way it builds a predictive model from data.
• Often, the accuracy of that prediction depends more on the
correct deployment of the technology and the quality of the
data than it does in the technology itself.
• The choice of data mining should be driven by the
advantages that it brings to the bottom line of the entire
business process – not just the statistical predictive
accuracy.
Measuring Data Mining Effectiveness –
Accuracy, Speed, Cost (cont’d)
• The other way that data mining techniques are often
measured is by speed.
• The reasoning is that the faster the tools runs, the larger is
the data set to which it can be applied. The larger the
database is, the better the accuracy of the predictive model
will be.
• To truly determine which technologies are best, it is
helpful to look at the big picture, which includes a much
larger business process than just data analysis. The full
process includes data collection, data analysis (data
mining), predictive model visualization, and the launching
of a marketing program against a customer set
Discovery versus Prediction
•
Discovery – finding something that you weren’t looking for
– One of the obvious things about real mining is that when you come
across a diamond or vein of gold, you know that you have found it
– You can recognize the important properties of diamonds or gold because they have
been discovered before and you know what they look like and feel like
– The main idea of how these system work is by making three measurements:
• How strong is the association?
• How unexpected is it?
• How ubiquitous is it?
– The first rule requires that the pattern in the data be a strong pattern (for example,
that it occurs 90% of the time)
– The second measurement ensures that the pattern is interesting to the user and not
obvious.
– The fulfillment of the third ensures that the pattern occurs often enough that it is
useful.
Discovery versus Prediction (cont’d)
• Prediction
– With prediction, you as the end user have a very specific event or attribute
that you want to find a pattern in association with.
– For instance, suppose you want to predict customer attrition.
– One of the most important parts of predicting customer attrition is having
historical information in your database about which customers have
attrited in the past.
– There may be many interesting patterns in your database – say, between
the age of your customers and their buying habits – that you might like to
discover, but in this case, you know very well that attrition is costing you
a lot of money.
State of the Industry
• The current offering in data mining software products
emphasize different important aspects of the algorithms
and their usage. The different emphases are driven because
of differences in the targeted user and the types of
problems being solved. There are four main categories of
products:
–
–
–
–
Targeted Solutions
Business Tools
Business Analyst Tools
Research Analyst Tools
Data Mining Methodology
• The first of these is the concept of finding a
pattern in the data
• The second of these is that of sampling or not
having to use all of the data in order to make
significant conclusions about what might be
happening with other parts of the data
• The third, validating the predictive models that
arise out of data mining algorithm
What is a pattern? What is a model?
• Although there are many ways to define patterns and
models, here is what they mean in the context of data
warehousing and data mining:
– Model. A description of the original historical database from which
it was built that can be successfully applied to new data in order to
make predictions about missing values or to make statements about
expected values.
– Pattern. An event or combination of events in a database that
occurs more often than expected. Typically, this means that its
actual occurrence is significantly different than what would be
expected by random chance
• What is the difference between a pattern and a model?
Visualizing a pattern
Figure 5-2 A Graphical Representation of Number Sequence
Visualizing a pattern
Figure 5-2 A Graphical Representation of Complex Number Sequence
appears much more understandable thanthe raw data
A note on terminology
•
•
•
•
•
•
Database
Record
Field
Predictor
Prediction
Value
A note on terminology (cont’d)