ON-DEMAND VISIBILITY
Data Virtualization an Overview
Ian Pestell
Director EMEA Operations
Copyright © 2005 Composite Software, Inc. All Rights Reserved.
Is Your Data an Asset or a Liability?
“Managers spend two hours a day looking for
information they need, and almost half the
data is useless once they get it”
Source: Accenture
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
2
Data within SOA
SOA
 SOA Architectures implement
Business Process
 Through Application Logic
 Operating on Data
Business
Business
Process
Process
Application
Application
Logic
Logic
 Data from many sources
 Existing Legacy systems
 Databases, Web Services,
Files
 Complicated by package
applications, ERP and CRM
Data
 Data Services are often
addressed late in SOA
Legacy
Data
Application
Data
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
3
Data Integration Options
Business
Solutions
Constant
Change
Dashboards
Data
Integration
Challenge
Reporting
SOA
Code within the Application?
File Extracts / Database Replications?
ETL / Physical Warehouses, Marts, Stores?
EII / Virtual?
Existing
Data
Packaged
Apps
RDBMS
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Data
Warehouses
4
Web
Services
Flat Files
Siloed
&
Complex
Data Consolidation / ETL
 Physical data movement via
scripts
Application
 Data is extracted from source
systems
 Transformed
 Data Cleansing, OLTP to OLAP
Data
Warehouse
 Loaded into target system
 Physical Data Warehouse,
Mart, Operational Data Store
 Very large volumes of data
 High degree of transformation
 Data is only as current as last
load
MDM
Hub
Data Mart
/ ODS
ETL
Server
Files
Databases
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
5
Applications
Data Federation / EII
 Virtual data views via modelling
 Data is left in-situ within data
sources
 EII server federates and
transforms data on demand
Application
 No Data Warehouse required
 High degree of flexibility in adding
new data sources
 Lower development & operating
costs
cache
EII
Server
Databases
Applications
 Moderate to high data volumes
 Low to medium transformations
 Data is ‘near’ real-time
Files
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
6
Solutions to Data Integration – Application
Integration
Application
Logic
 Integration performed in the
application logic
 Data sources require WS API
 Vendor – bespoke
 Access to WS API via ESB
Enterprise Service Bus
 Supported by standards
 ws-orchestration, ws-choreography
 Disadvantages
WS API
Legacy
Data
WS API
WS API
WS API
Application
Data
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
7






Complex logic with Application
High develop time
Maintenance of change
Enabling WS APIs for data
High load on ESB
Latency of ESB
Solutions to Data Integration – Data Services
Layer
Application
Logic
 Implement Data Services
Layer
 Provides set of reusable
data services
 WS access into ESB
 Connects to data sources
using native interfaces
 Integration / optimization
only done within data layer
 Reduces load
Enterprise Service Bus
Data Services Layer
Legacy
Data
Application
Data
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
8
You Can Integrate Data Differently
Business
Solutions
Real-time
Information
Dashboards
Reporting
Applications
Positions
Virtual
Data
Layer
Orders
POs
Virtualizes
Trades
Vendors
Abstracts
Invoices
Customers
GLs
Employees
Federates
Existing
Data
Packaged
Apps
RDBMS
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Data
Warehouses
9
Web
Services
Flat Files
Siloed
&
Complex
Composite’s Mission
Business
Applications
Dashboards
Reporting
SOA
 Composite helps IT teams who need to build new applications faster,
 by providing data services middleware that lets them access any
data in the enterprise… and deliver it on demand.
Existing
Data
Packaged
Apps
RDBMS
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Data
Warehouses
10
XML Docs
Flat Files
What We Do
Operational Reporting
Access
Combine
Deliver
>>> On Demand <<<
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
 Single Source for all Data
 Standardized Access
 Reusable Data Services
Data Services
(SOA)
NA Sales
Data Engine
For
Operational BI
 Drug Discovery
 Order Management
 Portfolio Management
 Inventory Management
Information
Infrastructure
App
11
Composite Software at a Glance

Privately-held, Silicon Valley firm
 Strong Business Model & Team

Best-of-Breed Virtual Data Integration / EII
 Discover, Virtualize, Abstract & Federate

IE 2008
Editor’s Award
Proven Business and IT Value
 Increased Flexibility
 Greater Agility
 Lower Costs
Financial Services
Top
5
Inv
Ban
k
Top 5
Comm
Bank
Top 5
I-Bank
Top 5
I-Bank
Top 5
I-Bank
Mutual
Fund
Life Sciences
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Technology
Energy and Utility
Consumer/Retail
Life Sciences
Public Sector
12
How we do it – Data Views
Composite View
Step 1 - Development:
Build and Publish a View
(A View contains Metadata
on the actual data source –
format, type, location,
access, etc.)
Data Source 1
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Data Source 2
13
How we do it – Data Views
Reporting
Analytics
Step 3 – Runtime:
Deliver Information
(Combined data creates
meaningful information)
Composite
Applications
Composite View
Step 1 - Development:
Build and Publish a View
(A View contains Metadata
on the actual data source –
format, type, location,
access, etc.)
1
A
B
a
b
2
C
D
c
d
3
E
F
e
f
4
G
H
g
h
Step 2 – Runtime:
Access and Combine
(Queries are optimized
for speed and efficiency)
Data Source 1
Data Source 2
1
A
B
1
a
b
2
C
D
2
c
d
3
E
F
3
e
f
4
G
H
4
g
h
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
14
Composite Lets You Easily Construct Data Services
Access Data
Request Results
Combine Data/
Model
Access
Data
Auto-Generated
Deliver
Results
WSDL
Combine
Data/
Model
Request Results
Deliver Results
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Auto-Generated WSDL
15
Composite Information Server Architecture
Easily Works Within Complex Environments
Client and Enterprise Applications
Web Services (SOAP/JMS)
Relational SQL (ODBC/JDBC/ADO.NET)
Composite Studio
COMPOSITE INFORMATION SERVER
Modeler
Composite Web
Manager
Administration API
Manager
Security
Data Services
Pre-Built
•S.
Customer-Built
(Views, SQL Script, XQuery, Java)
Cache
Query Processing Engine
Security
Relational (SQL) WS (SOAP/JMS)
RDBMS
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Excel
Java
APIs
MF Adapter
Web Service Flat File Custom Application
16
Mainframe
Why Composite Application Data Services™
Excel
BI Tools
Composite
Apps
SQL / WS
Complex API
Adaptor
Application
Views
Enterprise Data
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
17
Data Services are a Critical SOA Component
Composite Applications
1) Data services stand alone
2) Transaction services stand
alone
3) Data services integrated
with transaction services
4) Data services with
orchestration using a
business process
management tool or ESB
5) Data services WSDL stored
in a UDDI registry
Orchestration
ESB / BPM
Registry
Services
DataData
Services
(Info Server)
“Over 60% of SOA
services will be
data services”
Transaction Services
Info Server
App Server
Source: IDC
Packaged Apps
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
SOAP/HTTP
XML Files
18
XML/HTTP
Files
Custom
Apps
NY Investment Bank – New e-business
application
 Implementation of new ebusiness application
 Initial estimate 2 man
years
 1 year application
 1 data integration
 Using EAI approach
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
19
NY Investment Bank – New e-business
application
 Implementation of new e-business
application
 Initial estimate 2 man years
 1 year application
 1 data integration
 Using EAI approach
 Using Composite Data services
 Reduces Data integration from 12
Month to 4 Months
 Created optimized and re-usable
data services
 Introduced abstraction layer
between application and data
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
20
Data Services At Work - New York Bank
Multiple Projects
1.
2.
3.
4.
Business and IT Benefits




Prime Brokerage Reporting
Risk Management
Back Office Reconciliation
Portfolio Balancing
Data Required (100%)
Reuse (nearly 40%)
Lower Impact (75% or more)
Faster Response (up to 80%)
Data
Services
Layer
Existing
Data
Packaged
Apps
RDBMS
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
Data
Warehouses
21
XML Docs
Flat Files
Single View of Payroll and Variances
Husasmidjan, Inc.
 Situation & Requirements:
 Scandinavian construction materials
retailer/wholesaler, 25 stores
 Business requirement: detect and address
variances between planned and actual payroll
expense
 Business user: Office of the CFO
 Technical requirement: Expose and combine
SAP R/3 and in-house budget data
Composite
Information
Server
 Solution:
Rollup View
1. Create views to SAP payroll information
2. Create views to other payroll information
3. Build roll-up view for analysis
Local views normalize
across different stores
and apps
4. Expose view into budget system
 Benefits:
 More accurate variance information sooner
 Rapid application development
Payroll data sources
Legacy
Files
Copyright © 2006 Composite Software, Inc. All Rights Reserved.
22