Informix Warehouse Accelerator
- A Look at its First Year and Beyond
Fred Ho
IBM
Content:
• Informix Warehouse & IWA Roadmap
• IWA Customers in its 1st Year
• IWA Partners
• Sample Customer Profiles
• Other products to Consider when Deploying IWA
• Competition
• Upcoming New Features
• When Should You Consider IWA?
• Q&A
vNext
Informix Warehouse Roadmap
11.70xC2
Warehouse
Accelerator
11.70
Star Join Optimization
Multi-index Scan
New Fragmentation
Fragment Level Stats
Storage Provisioning
11.5xC6
11.5xC5
11.5xC4
External Tables
11.5xC3
Informix Warehouse
Feature
- SQW
- Data Modeling
- ELT/ETL
3
Informix Warehouse with
Storage
Optimization/Compression
IWA Phase 2
Cognos integration
- Native Content Store on Informix
SQL Merge
IWA Roadmap
vNext
3GDB
2012 IIUG
12.1xC1
Trickle Feed
Union queries
OAT Integration
SQL/OLAP and SQL
enhancement in IDS
11.7xC5
11.7xC4
11.7xC2
IWA 1st Release
On SMP
4
Partition Refresh
Load from Secondary
Solaris on Intel
11.7xC3
Workload Analysis Tool
More Locales
Data Currency
IGWE
IWA on Blade
Server
IWA Product Summary
Milestones:
Key Product Characteristics:
-Initial Product Launch of Informix
Ultimate Warehouse Edition at end of Q1
2011
-Only IBM offering for in-memory columnar
database designed for warehousing and
analytics
-Added Informix Growth Warehouse
Edition at end of Q3 2011
-Exceeded an aggressive sales target in
its 1st year
-Customers in 4 different continents
across multiple industries
-Deploys on commodity Linux-Intel
hardware with Informix running mixedworkloads
-Transparency of queries from BI tools
-Extreme speed (e.g. 100x) with no
administration and no tuning, e.g. no
indexes, partitioning, summary tables,
cubes, etc.
-Competitive pricing, especially for SMB
customers
© Copyright IBM Corporation, 2012
IWA Customers At a Glance
• Customers in Europe, Latin America, U.S. and Africa
• Spans different industries such as Retail, Insurance, Transportation/Logistics, and
Public Sector
• Informix partners having great success resulting in 70% of deals
• Partner solutions based on Informix Warehouse Accelerator
• Customers with existing warehouse deployments as well as new deployments
• Existing Informix customers and new customers
• Beating competition in performance and much lower cost of ownership including
Teradata, SQL Server, Greenplum and Vectorwise
© Copyright IBM Corporation, 2012
IWA Partners
MC Software, Ltda
IWA in Action
Informix IWA at a
Retail Company
Store Managers &
Home Office Managers
across thousands of
stores want to analyze
promotional items
IWA with 24 cores &
1TB RAM on single
Linux Intel box
< 10 secs average response
with 500 concurrent users
1/10 of the cost of
competitor system
Data set is ~200GB
6-10 times faster
performance
Current database
unable to provide quick
enough turnaround
Ability for retailer to react to
promotional items
Challenge
Solution
Result
IWA at an E&U Agency
Interface to DM & IWA
Client
Apps
Server Prod Principal (1)
HDR
Réplicati
on
Data Warehouse Server (3)
Client
Apps
HDR Secondary (2)
BI
Queries
Remote Secondary (4)
Supplier to city for E&U, Water
and Drainage
Starting Smart Metering project
1 TB database, 1,200 OLTP
Users, 100 BI Users
Challenge
OLTP data
replicated to
Remote Secondary
Informix servers for
OLTP, High Availability
and Data Warehousing
across 4 servers
Solution
All Informix architecture
built to handle current
needs and future growth
Result
IWA at a Government Agency
• Officer contacts Dispatch
• Dispatch manually types into
disparate systems
• Slow, error-prone, lengthy
queue wait times
State
County
Sheriff
Dept
Specialty
Livermore Police “Link Analysis”
County
Daily-to-Hourly Loads
CAD/R
MS
Web query-pattern analyzer
application, extremely
lightweight results
Verizon 3G Link
•
State
•
•
IDS 11.7/IWA in single
Linux server
IWA loaded daily from IDS
Load hourly using pingponged paired instances
IWA at a Tax Collection Agency
Public independent institution of the Ministry of Economy and Finances in charge of
administering , applying tax collection processes, and controlling the National Government
taxes and Social Security contributions for a Latin American country.
Operational Data Store (ODS), with both historic and recent data
•
•
Size: ~3 TB (of which: 1.6+ TB of raw data, ~1.2TB in indexes)
•
Daily Refresh/Update
•
ETL: ~20 million rows (4GB) / day
•
Adding new electronic billing system data, expected growth of ~1.5TB/year
•
Data Warehouse expected to be ~10 TB large in 5 years
Business Users: 1800 total, 100 concurrent
(2% complex queries, 98% simple queries)
•
Won over Sybase after PoC consisting of:
8 tables - 1 Fact (406M rows), 8 dimensions (ranging from 7M to 49M rows),
9 query set
Summary of IWA Profiles
• Spans multiple industries including Retail, Insurance, ERP, E&U,
Hospitality, Logistics/Transportation, etc.
• Size of databases ranging from 200 GB to 1 TB+
• Size of systems from 4 cores and up
• Query performance is the key buying factor
• Ease of maintenance (lack of tuning, cube building, etc) also
important
• Integration with other Informix features an important factor, e.g.
Time-Series, High Availability
• Partner involvement key to providing solution and support
Why is IWA So Important?
According to Gartner, IDC, and Forrester:
• Cost is driving interest in alternative architectures for Data
Warehousing.. notably a strong interest in in-memory data mart
deployments
•Gartner estimates that IMDBMS will replace 25% of traditional data
warehouse and OLTP systems by 2016
•IMDBMS technology also introduces a higher probability that
analytics and transactional systems can share the same database.
IWA Changes the World
2/28/12
15
IWA Changes the BI Value Proposition
• BI has historically offered:
– Reporting, usually inflexible, to answer a priori questions
– Long cycle times for analytical results
– Little to no (affordable) opportunity for optimization
– Incredibly expensive
• IUWE/IWA meets the need for real-time BI analytics
– Supermarket checkout offers,
– Market-current financial analytics,
– Police stops & Investigations,
– Security screening,
– Health care usage analytics,
– Merger integration analysis
IWA Technology Innovations
Number of
Occurrences
64-bit processor
RAM in
TB
Frequency
Partitioning
Common
Values
…
…
11111
0
&1111
0
01001
0
== 1110
0
…
Compressed Predicate
Evaluation
Rare
values
…
A1 D 1
G1
A2 D 2
G2
A3 D 3
G3
A4 D 4
G4
SIMD
How Fast is Fast?
• 100 to 1000 times faster than alternatives
• That’s changing Hours/Minutes to Seconds
Who
Existing DB
What
How much
faster?
German Agency
XPS – 30GB
437 query set
US Agency
2TB warehouse
8 representative
queries
127x
Large Apparel
Retailer
Informix - 150GB
>30 queries
330x
Global Retailer
25 million row fact
table
Representative set
of queries
300x
90x
Some Amazing Results
• Some results are simply astonishing:
Size
Query
Original
IWA
2TB
Top 100 Entities
1:28:22
0:01:28
2TB
Top 100 Members
1:22:32
0:01:05
2TB
Summarize by State & County
1:34:37
0:00:14
2TB
Summarize by State, County, City,
State, Zip, Program, Program
Year, Commodity and Fiscal Year
1:48:58
0:00:41
Other Products to Consider when Using IWA
ETL
SQW
DataStage/Informatica
SSIS
Included in IUWE/IGWE
Many sites already have licenses
Common with Windows sites
BI Tools
Cognos Enterprise or Express
Hyperion/MicroStrategy/
Crystal Reports/Business Objects
Pentaho/Jaspersoft/BIRT
Lots of use cases
Standard ODBC/JDBC
SPSS
Pentaho
Internal tests
Partner tested
Predictive
Analytics
Partner based solutions
Competition to IWA
DW Appliance
Columnar Database
DataAllegro (Microsoft)
Calpont
Dataupia
Exasol
Greenplum (EMC)
Infobright
Kognito
ParAccel
Netezza (IBM)
Sand Technology
In-Memory OLAP Tools
QlikTech/QlikView
Applix TM-1 (IBM-Cognos)
Exalytics (Oracle)
PALO
Vertica (HP)
Sybase IQ (SAP)
In-Memory Data
Warehouse
HANA (SAP)
IWA (IBM)
Comparison with Teradata
Comparison with GreenPlum
SQL
# of Records Greenplum
Greenplum
Returned
- Row mode - Column mode
Ingres
Vectorwise
IWA
Select count(*) from salecost;
1
70s
7.5s
5.5s
0.1s
select
sum(qty*cost),sum(salevaluediscountvalue) from salecost
where shopid='S102' and sdate
between '20100101' and
'20101231';
1
71s
20s
13s
1s
select
sum(qty*cost),sum(salevaluediscountvalue) from salecost
where shopid='S102';
1
71s
18s
25s
2s
Comparison with SQL Server
Report Name
SQL in Report
SQL Server 2008
IDS+IWA
Speed-Up
TestBericht_ChMa
2 SQL tables
27.36s
3.93s
6.9x
HZB-Einschrankung
2 SQL tables
43.14s
2.76s
15.6x
Immatrikulierte Studierende
4 SQL tables
11.07s
0.81s
13.6x
Zusammenhang
Hochschulsemester und
Fachsemester
3 SQL tables
11.81s
2.7s
4.3x
Beziehungszahlen –
Hochschulen –
Studienanfangerinnen
3 SQL tables
3.59s
0.48s
7.5x
Comparison with Oracle Using Oracle
Database Gateway
Oracle DBMS
Oracle client
- native
DWH Development
- Design
- Data processing
Oracle
Database
Gateway
Informix
Ultimate
Edition
Source: Michael Koester (IBM Germany)
Informix
Warehouse
Accelerator
DWH Evaluation
- Analysis
- Reporting
Standard client
- Cognos
- MicroStrategy
- BO
Intel / AMD 64 Bit Linux
Bladeserver
Oracle Gateway Test Database
Test System
Series 1
Series 2
Series 3
Tables
#Rows
#Rows
#Rows
tf_ast_201111
10,000,000
30,000,000
57,000,000
td_dwh_status
4
4
4
td_dwh_geschlecht
2
2
2
50
50
50
6
6
6
10,000
10,000
10,000
1,000
1,000
1,000
100
100
100
10
10
10
td_dwh_alter
td_dwh_altersgruppe
td_dwh_gemeinde
td_dwh_kreis
td_dwh_reg_bez
td_dwh_bundesland
Series 1: appx. 300MB
Series 2: appx. 1.2GB
Series 3: appx. 2GB
Series
1 Oracle
Recording 1
Recording 2
Recording 3
Recording 4
Recording 5
11.422s
11.524s
11.416s
11.365s
11.426s
1 Oracle DG
0.803s
0.802s
0.785s
0.802s
0.775s
1 IWA
0.295s
0.311s
0.334s
0.295s
0.311s
31.493s
31.715s
31.473s
31.398s
31.639s
2 Oracle DG
0.905s
0.904s
0.854s
0.879s
0.869s
2 IWA
0.502s
0.446s
0.475s
0.450s
0.442s
58.928s
59.687s
58.874s
59.232s
59.081s
3 Oracle DG
1.088s
1.046s
1.029s
1.097s
1.078s
3 IWA
0.607s
0.623s
0.608s
0.611s
0.603s
2 Oracle
3 Oracle
3 Oracle (FS)
3m15.259s
Oracle recordings fully cached in bufferpool,
Oracle (FS) recording read from disk
Measurements:
Oracle:
time cat ora_q.sql | sqlplus / as sysdba
Oracle DG: time cat ora_tg_q.sql | sqlplus / as sysdba
IWA:
time cat iwa_q.sql | dbaccess dm_ast
Performance gain factor Oracle DG over Oracle:
Series 1:
Series 2:
Series 3:
Series 3 Oracle (FS):
avg. Factor 14,41
avg. Factor 35,77
avg. Factor 55,45
Factor 179,47
Summary of Competitive Analysis
• Many competitors from traditional DBMS vendors and purpose-built open
source vendors
• IWA can compete, in performance with any of them head-to-head
• Cost is significantly lower even compared to open source vendors
• Ability for Informix/IWA to provide OLTP-DW-Acceleration on a single
platform is unique
• Future release promises unlimited scalability with in-memory performance
IWA Moving Forward
More in 2012:
• Incremental Partition Refreshes
• Supporting FlexGrid configurations for loading to IWA
• Further Integration with Cognos and other BI tools
Informix Warehouse Offerings
Ultimate Warehouse Edition
Growth Warehouse Edition
(IUWE)
(IGWE)
*High-End
*Mid-Range
Fully Featured
Includes IWA & Storage
Optimization/Compression
Includes IWA
Unlimited scalability
Informix: 16 cores, 16GB RAM max
IWA: 16 cores, 48GB RAM max
4 64-bit Platforms;
Informix: 4 64-bit Platforms;
IWA runs on Linux-Intel 64
IWA runs on Linux-Intel 64
Sizing Guidelines - IUWE
T-shirt size
Raw data *
Main Memory
(GB)
Number of Intel cores
(X7560)
XL
>1.5 TB to 3 TB
1024
24-32
L
>750 GB to 1.5 TB
512
20-24
M
> 400 GB to 750 GB
256
16-20
S
> 250 GB to 400 GB
192
12-16
* Raw data represents only table data and excludes any indices, temp table space etc
Important Considerations
T-shirt sizes are a reference guideline only and are not officially available configurations.
Sizing Guidelines - IGWE
T-shirt size
Raw data *
Main Memory (GB)
Number of Intel cores (X3690)
L
≥ 100 GB to 250 GB
48
12-16
S
Up to 100 GB
32
8-12
With IGWE, IWA can be run on any 64-bit Linux-Intel system that has upto 16 cores. IWA will utilize a
maximum of 48GB of RAM to hold potentially upto 250 GB of raw table data. Customers can take advantage
of the affordable IBM System x3690 configuration for hardware.
IGWE is a third of the price of IUWE and offers the same benefits:
- Orders of magnitude performance gains
- Up to 5 times of data compression in memory
* Raw data represents only table data and excludes any indices, temp table space etc
Important Considerations
T-shirt sizes are a reference guideline only and are not officially available configurations.
When Do You Consider IWA?
•… performance issues on analytics
and business reports ?
•Reports taking too long to run
•Ad-hoc queries with unpredictable
response times
•… cost and flexibility for mixed
workloads?
•Unable to optimize on a single
platform
•… ongoing warehouse
maintenance and administration?
•Constant tuning
•Building/Maintaining cubes
•Constant storage optimization
•… leaving you at a competitive disadvantage ?
•This is an example text. Go ahead and replace it with your own text. It is meant to give
you a feeling of how the designs looks including text.
33
What we can offer?
• Complete in-depth training with hands-on experience
BP
• Enables Partners to do their own PoCs
Worksho
p
• Proof of Technology on the Cloud
• Before after comparison with and without IWA
Demo on
• Immediate demonstration of value to the customer
the Cloud
• In-depth analysis vs competition
Competitiv • Don’t hesitate to ask; competition throws out a lot of “FUD”
e Analysis
• Still being worked on; Goal is to be able to get H/W for PoCs
Try and Buy • Will enable partners to request and obtain h/w for PoCs
Program *
34
Q&A
Fred Ho
Program Director, Informix
IBM
hof@us.ibm.com