Data Warehouse Fundamentals Rabie A. Ramadan, PhD 2 What did you do in Your Assignment ? For an airlines company, how can strategic information increase the number of frequent flyers? Discuss giving specific details. You are a Senior Analyst in the IT department of a company manufacturing automobile parts. The marketing heads are complaining about the poor response by IT in providing strategic information. Draft a proposal to them explaining the reasons for the problems and why a data warehouse would be the only viable solution. 2 What did you do in the Project ? Egypt Election System • Governorates’ database system • Summarization System • Data Warehouse Server • Web page with query based system • Multiple databases on Multiple Servers • Meta data 3 http://www.inf.unibz.it/dis/teaching/DWDM/index.html 4 Definitions & Motivations Why Data Mining? Explosive Growth of Data: from terabytes to petabytes Data Collections and Data Availability • Crawlers, database systems, Web, etc. Sources • • • Business: Web, e-commerce, transactions, etc. Science: Remote sensing, bioinformatics, etc. Society and everyone: news, YouTube, etc. 5 Why Data Mining? Problem: We are drowning in data, but starving for knowledge! Solution: Use Data Mining tools for Automated Analysis of massive data sets 6 What is Data Mining? Data mining (knowledge discovery from data) • Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data 7 What is Data Mining? Alternative names • Knowledge discovery (mining) in databases (KDD), • knowledge extraction, • data/pattern analysis, • data archeology, • Data dredging, • information harvesting, • business intelligence, • etc. 8 Knowledge Discovery (KDD) Process 9 Knowledge Discovery (KDD) Process 10 Typical Architecture of a Data Mining System 11 Confluence of Multiple Disciplines 12 Why Confluence of Multiple Disciplines? Tremendous amount of data • Scalable algorithms to handle terabytes of data (e.g., Flickr had 5 billion images in September, 2010 [http://blog.flickr.net/en/2010/09/19/5000000000/]) High dimensionality of data • Data can have tens of thousands of features (e,g., DNA microarray) 13 Why Confluence of Multiple Disciplines? 14 Different Views of Data Mining Data View • Kinds of data to be mined Knowledge view • Kinds of knowledge to be discovered Method view • Kinds of techniques utilized Application view • Kinds of applications 15 Data to Mined In principle, data mining should be applicable to any data repository We will have examples about: • Relational databases • Data warehouses • Transactional databases • Advanced database systems 16 Relational Databases 17 Data Warehouses 18 Transactional Databases 19 Advanced Database Systems(1) 20 Advanced Database Systems(2) 21 Knowledge to be Discovered 22 Characterization and Discrimination 23 Characterization and Discrimination (1) 24 Class Activity • Differentiate between Data Mining and Data warehousing? Data warehousing is merely extracting data from different sources, cleaning the data and storing it in the warehouse. Where as data mining aims to examine or explore the data using queries What are the Different problems that “Data mining” can solve? Data mining can be used in a variety of fields/industries like marketing, advertising of goods, products, services, AI, government intelligence. How does the data mining and data warehousing work together? Data warehousing can be used for analyzing the business needs by storing data in a meaningful form. Using Data mining, one can forecast the business needs. Data warehouse can act as a source of this forecasting. 25 Frequent Patterns, Associations, Correlations 26 Classification and Prediction 27 Cluster Analysis 28 Outlier Analysis 29 Evolution Analysis 30 Techniques Utilized 31 Applications Adapted 32 Major Challenges in Data Mining 33 Summary 34