DataWarhouse rješenje za banke - IBM Netezza Leader in Data Warehouse Appliances Robert Božič robert.bozic@si.ibm.com © 2013 IBM Corporation Information Management The IBM Netezza Appliance: Revolutionizing Analytics What is Netezza? © 2013 IBM Corporation Information Management Appliances make it simple, completely transforming the user experience. Dedicated device Optimized for purpose Complete solution Fast installation Very easy operation Standard interfaces Low cost 3 © 2013 IBM Corporation Information Management The IBM Netezza Appliance: Revolutionizing Analytics Purpose-built analytics engine Integrated database, server & storage Standard interfaces Low total cost of ownership Speed: 10-100x faster than traditional systems Simplicity: Minimal administration Scalability: Peta-scale user data capacity Smart: High-performance advanced analytics © 2013 IBM Corporation Information Management Some of the customers Zavarovalnica Maribor Zavarovalnca Triglav Telekom Slovenije Tuš Mobil Petrol Informatika NLB MKZ Fabrika Duvana Sarajevo © 2013 IBM Corporation Information Management Digital Media Financial Services Government Health & Life Sciences Retail / Consumer Products Telecom Other Page 6 © 2013 IBM Corporation Information Management Business case – financial institution Evolution of DWH environment Year 2000 – first “real” data warehouse – 1 power user, 20 common users, 1 IT developer Years 2001 – 2005 – Number of users increased – 3 IT professionals, 100 common users, 10 power users – Enlargment of the data warehouse by factor 3 – DWH/BI becomes invloved in all crucial processes Years 2005 – 2007 – Enlargment of the data warehouse by 100 – Increasing the number of common users to 250 – By end of 2007 DWH/BI declared as a key business process at ZM © 2013 IBM Corporation Information Management Users expectations High demand for information from the DWH system – the users expect that the system contains everything “Immediate” responsiveness • Daily or even shorter frequencies for data loading • Real time BI • Sophisticated iterative analysis: marketing analysis, product or customer profitability analysis etc. • Quick ad-hoc reporting for various purposes © 2013 IBM Corporation Information Management Main issues with existing DWH • More and more administration: database servers, application servers, BI tools, BI applications... • Higher and higher ownership costs (HW; SW licences...) • Smaller and smaller time window for ETL • More and more end users • Continously increased complexity of the reports and analysis - killer queries which can only be run in the evening hours , “freeze” the server, even on DWH server. © 2013 IBM Corporation Information Management Main challenges with building fully mature DWH/BI infrastructure • Cost boost • Entering in the upper cost-range of DWH servers • Unfavorable licensing policy for database SW (number of users, CPU licensees) • Complex administration of database servers • Required 1 - 4 employees to manage DWH © 2013 IBM Corporation Information Management Benefits • Considered by the management as one of the best IT investments in the last 10 years • The same DWH/BI team is capable to manage even larger BI infrastructure. • Vast simplification of many complex queries and batch jobs • Development of new BI solutions • Users completely changed the way of thinking what they can get out of DWH © 2013 IBM Corporation Information Management Smart Predicts what shoppers are likely to buy in future visits Coupon redemption rates as high as 25% “Because of (Netezza’s) in-database technology, we believe we'll be able to do 600 predictive models per year (10X as many as before) with the same staff." Eric Williams, CIO and executive VP © 2013 IBM Corporation Appliance Simplicity © 2013 IBM Corporation Information Management Managing The Netezza Appliance No software installation No storage administration No database tuning Less DBA drudgery, More applications © 2013 IBM Corporation Information Management OLE-DB The Netezza Appliance – Loading Data Integration Ab Initio JDBC Business Objects/SAP Composite Software Expressor Software GoldenGate Software (Oracle) Data In IBM Information Server Sunopsis (Oracle) ODBC Informatica SQL WisdomForce © 2013 IBM Corporation Information Management OLE-DB The Netezza Appliance – Querying © 2013 IBM Corporation ODBC Data Out SQL Actuate Business Objects/SAP Cognos (IBM) Information Builders Kalido KXEN MicroStrategy Oracle OBIEE QlikTech Quest Software SAS SPSS (IBM) Unica (IBM) JDBC Reporting & Analysis Information Management Simple to Deploy and Operate Operations Simply load and go .… it’s an appliance Installation to Business Value in ~2 days Ease of Evaluation and Perform As Advertised BI Developers Data model agnostic No configuration or physical modeling No indexes or tuning – out of the box performance Focus on business value, not physical design ETL Developers Faster load and transformation times No aggregate tables needed – simpler ETL logic In-database transformation – ‘ELT’ Business Analysts On-Stream processing by 100’s of nodes Train of thought analysis – 10 to 100x faster True ad hoc queries Lower latency – load & query simultaneously 17 © 2013 IBM Corporation Information Management Appliance Architecture © 2013 IBM Corporation Information Management IBM Netezza True Appliance Architecture SOLARIS AIX TRU64 HP-UX WINDOWS LINUX Client Database Server Storage DATA SQL ETL Server DBA CLI Source Systems 3rd Party Apps SQL I/O CACHE I/O CACHE CACHE Data High Performance Loader © 2013 IBM Corporation Information Management IBM Netezza True Appliance Architecture Client SOLARIS AIX TRU64 HP-UX Database WINDOWS Storage Server LINUX ODBC 3.X JDBC Type 4 SQL-92 SQL-99 Analytics ETL Server DBA CLI Source Systems CACHE 3rd Party Apps I/O CACHE Database, Server,I/O Storage - in one High Performance Loader © 2013 IBM Corporation CACHE Information Management IBM Netezza True Appliance Architecture Optimized Hardware+Software Streaming Data Purpose-built for high performance analytics; requires no tuning Hardware-based query acceleration for blistering fast results True MPP Deep Analytics All processors fully utilized for maximum speed and efficiency Complex analytics executed in-database for deeper insights 21 © 2013 IBM Corporation Information Management IBM Netezza True Appliance Massively Parallel Processing Client SOLARIS AIX TRU64 HP-UX WINDOWS 1 LINUX S-Blade Processor & ODBC 3.X JDBC Type 4 OLE-DB SQL/92 streaming DB logic SQL Compiler 2 S-Blade Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade Processor & streaming DB logic Optimize Admin ETL Server High-Speed Loader/Unloader 960 DBA CLI Source Systems Front End DBOS High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & streaming DB logic 3rd Party Apps SMP Host High Performance Loader © 2013 IBM Corporation Network Fabric Massively Parallel Intelligent Storage Information Management IBM Netezza True Appliance Massively Parallel Processing™ Client SOLARIS AIX TRU64 HP-UX WINDOWS 1 LINUX SQL SQL Compiler S-Blade Processor & 2 Snippets 1 2 3 3 streaming DB logic 2 S-Blade 2 3 Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade 2 3 Processor & streaming DB logic Optimize ETL Server High-Speed Loader/Unloader Admin SQL 960 DBA CLI Source Systems Front End DBOS High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & 2 3 streaming DB logic 3rd Party Apps SMP Host High Performance Loader © 2013 IBM Corporation Network Fabric Massively Parallel Intelligent Storage Information Management IBM Netezza True Appliance Massively Parallel Processing™ SOLARIS AIX TRU64 HP-UX Client WINDOWS 1 LINUX S-Blade Processor & 2 Consolidate 3 streaming DB logic SQL Compiler 2 S-Blade 2 3 Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade 2 3 Processor & streaming DB logic Optimize Admin ETL Server High-Speed Loader/Unloader 960 DBA CLI Source Systems Front End DBOS High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & 2 3 streaming DB logic 3rd Party Apps SMP Host High Performance Loader © 2013 IBM Corporation Network Fabric Massively Parallel Intelligent Storage Information Management Our Secret Sauce select DISTRICT, PRODUCTGRP, sum(NRX) from MTHLY_RX_TERR_DATA where MONTH = '20091201' and MARKET = 509123 and SPECIALTY = 'GASTRO' FPGA Core Uncompress Project CPU Core Restrict, Visibility Complex ∑ Joins, Aggs, etc. Slice of table MTHLY_RX_TERR_DATA (compressed) select DISTRICT, where MONTH = '20091201' PRODUCTGRP, and MARKET = 509123 sum(NRX) and SPECIALTY = 'GASTRO' © 2013 IBM Corporation sum(NRX) Information Management Appliance family for data life-cycle management 26 Skimmer N1001/N2001 Cruiser Dev & Test System Data Warehouse High Performance Analytics Queryable Archiving Back-up / DR 1 TB to 10 TB 1 TB to 1.5 PB 100 TB to 10 PB © 2013 IBM Corporation Advanced Analytics © 2013 IBM Corporation Information Management Advanced Analytics the Netezza Way SPSS complex analytics SAS, SPSS, R, Java, etc implicit parallelism petabyte scalability appliance simplicity SQL Demand Forecasting Fraud Detection R, S+ SQL © 2013 IBM Corporation Pre-Built In-Database Analytics Statistics Descriptive Statistics+ Distance Measures* Hypothesis Testing* Chi-Square & Contingency Tables* Univariate & Multivariate Distributions+ Transformations Data Profiling / Descriptive Statistics+ General Diagnostics Time Series Autoregressive+ Forecasting* Statistics+ Sampling Data prep Monte Carlo Simulation* Data Mining Predictive Mathematical Basic Math* Permutation and Combination* Greatest Common Divisor and Least Common Multiple* Conversion of Values* Exponential and Logarithm* Gamma and Beta Functions Matrix Algebra+ Area Under Curve* Interpolation Methods* Geospatial Association Rules+ Linear Regression+ Geospatial Data Type Clustering+ Logistic Regression+ Geometric Functions Feature Extraction+ Classification Geometric Analysis Discriminant Analysis* Bayesian Sampling Model Testing © 2013 IBM Corporation * Fuzzy Logix DB Lytix capabilities + Netezza Analytics and Fuzzy Logix DB Lytix capabilities IBM Netezza: bold claims, backed up © 2013 IBM Corporation Information Management Bold Claims, but...We Prove Them! We prove we are simpler We prove we deliver performance We prove we work within your environment We prove we integrate with your 3rd party tools We prove we are “easy to do business with” We prove we have the lowest TCO We prove business value © 2013 IBM Corporation