Information Server: Upgrade Workshop
Maj 2010
1
© 2010 IBM Corporation
Upgrade Workshop
What Is New In InfoSphere DataStage
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
●
●
●
●
Three releases of DataStage since the 7.5.x series
Each release has introduced new features
targeted specifically for the data integration user
Enhancements extend across a variety of product
areas (as shown to the left)
The following slides will:

introduce each category

list some specific features in each set

dive more deeply on a few
Connectivity
2
2
© 2010 IBM Corporation
Upgrade Workshop
Developer Usability
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
Connectivity
3
3
Features that simplify tasks within the user interface
tasks for both basic and advanced use cases
●
Designer Performance
●
Function Expansion
●
Designer Graphical & Function Upgrades
●
Multiple User Environment – Locking/Read Only
●
Find/Search for Objects in the Repository
●
Graphical Impact Analysis
●
Job, Table or Routine Difference
●
Job Deployment with Information Server Manager
●
Balanced Optimization – leverage the power of the DBMS !
●
DSX Export Improvements
●
Globalization – UI and messages translated to 9 different languages
© 2010 IBM Corporation
Upgrade Workshop
Designer – New Repository Tree
New tree view for display of Repository contents
– New folder model
– In-place “Find”
– Expandable view
New folder model
– Replaces previous Category model
– No restrictions on where objects live in the
folder structure
•
•
4
Jobs can live in the same folder as Table
Definitions, Routines, Transforms etc.
Allows user to configure Repository content in
the way that suits their application e.g. task
based vs class based structure
© 2010 IBM Corporation
Upgrade Workshop
Quick Find - Basic
●
Find item in Repository tree
– In-place find
– Find by Name (Full or Partial)
– Wild card support
– Find next…
– Filter on type
5
© 2010 IBM Corporation
Upgrade Workshop
Impact Analysis – Graphical View
Impact Analysis:
- Find dependencies …What does this item depend on?
- Find where used
…Where is this item used?
Results shown using the
Advanced Find window
6
© 2010 IBM Corporation
Upgrade Workshop
Job Difference – Integrated report
Difference report
displayed
in Designer - jobs
opened
automatically
from report hot links
Options available to:
- Print report
- Simple “Find” in
report
- Launch external diff
tool for more indepth diff of
textual
properties, e.g.
Routine source
7
© 2010 IBM Corporation
Upgrade Workshop
General Enhancements

Design Time Performance
– Significant Performance improvement in
Job Open, Save, Compile etc.

Function Expansion
– New Utility, String and Date/Time functions
eg: IsValidTime, NthWeekdayFromDate, DecimalToTimestamp
8
Parallel Only
© 2010 IBM Corporation
8
Upgrade Workshop
Functional Stages
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
New stage types that introduce new off the shelf data
integration functions or expand existing ones
●
New XML Stage
●
Vertical Pivot
●
Transformer Enhancements
●
Slowly Changing Dimension
●
Enhanced Surrogate Key Stage
●
Multi Format Flat File Support
●
Range Lookup
●
Horizontal Parallel Pivot Stage
●
Checksum Stage
Connectivity
9
9
© 2010 IBM Corporation
Upgrade Workshop
XML Pack
 Improved environment for mapping complex XML documents from one schema to another
in single or multiple steps
 Support for Schemas (XSD XML Schema 1.0, WSDL 1.1 )
 Deliver Support for complex XML Transformations with need for shredding the XML
– Hierarchical Join, Relational join, Filter, Switch, Sort, Union, Regroup, RowToColumns,
ColumnsToRows, Aggregate, Distinct
 Support for multiple input and
output links, including
reference, reject
 Shall support partitioning, multi
threaded and stream
processing of large
XML documents
 Performance and volume
improvements
– reduced memory
requirements,
increased throughput
– Remove restrictions on
document size
10
10
© 2010 IBM Corporation
Upgrade Workshop
Customer XML Job – pre 8.5 vs 8.5
Ran in 11% of the time
without introducing any of
the new parallelization
features
11
11
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
Vertical Pivot
•
Enhanced Pivot stage to
support - Vertical Pivoting
•
mapping multiple input rows
with a common key, to a
single output row containing
multiple columns
•
Coves three basic
requirements: key based
groups, columnar pivot and
aggregate functions
12
12
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
Transformer Enhancements
•
Deliver Looping in the
Transformer
•
Allow multiple output rows to be
produced from a single input row
•
Support for End of Data Flag to
support Key Break Logic
•
New Input Cache
•
•
SaveInputRecord()
•
GetSavedInputRecord()
New System variables, function
•
@ITERATION, Loop Count
•
LastRow(), End of data flag for last row
•
LastRowInGroup(InputColumn), automate
change detection
•
Stage, Loop Variables support for
Nullability
•
More options for Null Handling
13
13
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
SCD Stage Functionality
14

A single stage that encapsulates all of the functionality required to target a star
schema model

“Wizard”-style features that enable the user to enter only the minimum amount
of information needed

Support for columns of SCD Types 1 and 2 in the same dimension table

Initial and incremental population of fact and dimension tables

Support for
surrogate key
management
across job runs
© 2010 IBM Corporation
Upgrade Workshop
Functional Stages
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
Simplify and expand the functionality available for
administrators of the tool
●
New Suite Installer
●
Increased Availability
●
Information Server Manager
●
Source Code Control System Integration
●
Audit Logging
●
Web Logging Console
●
New Administration and Super Operator Roles
●
Support Assistant
Connectivity
15
15
© 2010 IBM Corporation
Upgrade Workshop
New Suite Installer
 Web enabled (no XWindows
configuation requirement)
 Prerequisite checker confirms
system meets basic requirements
as well as selected installation
options
 Installer supports resuming an
installation that failed for
unexpected reasons
 Streamlined feature selection
including adding additional
Information Server products and
additional tiers to the system
 Trust based licensing is now default
 Install utility also handles
patches/updates including unique
Patch Merging Installation.
16
© 2010 IBM Corporation
Upgrade Workshop
Increased Availability





17
Delivering Horizontal and Vertical
scaling / load balancing of the
Domain and DB Tiers
Delivering higher levels of Availability
for Production and Development
environments
Cluster support for Application Server
Tier
– WAS 6.1 and 7.0 ND
Cluster support for Repository Tier
– DB2 HADR / Cluster, Oracle
RAC
Improved Failover support for Engine
Tier
© 2010 IBM Corporation
Upgrade Workshop
What is the Information Server Manager?

New Information Server application
 Manages DataStage & QualityStage
objects’ lifecycles
 Move objects from development
environments to test and production
environments
 Handle updates to previously deployed
objects
 New import/export capabilities
 GUI available on windows platforms
(client or server)
 Command line interface available on
all client and server platforms

18
Is a replacement for, NOT a
reimplementation of, the pre-8.0
DataStage Version Control application
© 2010 IBM Corporation
Upgrade Workshop
Source Code Control System Integration
 Leveraging the Eclipse
Platform for Team
Development
 Integration with codemanagement (SCCS)
providers supported
through Eclipse Plugins
 Support for ClearCase,
CVS
– Other providers
configurable via
Eclipse Team Plugins
 Functions to interact with
the SCCS invoked from
the Information Server
Manager
19
19
© 2010 IBM Corporation
Upgrade Workshop
ISA Lite - Support Assistant
1. System Requirements
Installation prerequisites checker
(general IS and SAP)
2. Data Collection
General IS data files (passive) and
specific component collections incl
SAP. Option to collect IS repository
records.
3. Diagnostics (Health)
General IS checks as well as
component specific checks.
4. Utilities
DS project cleanup tool
20
© 2010 IBM Corporation
Upgrade Workshop
Audit Logging
 Deliver an audit log of security-related events.
 Delivers SOX and Security Compliance
 The following groups of audit events are logged
– User and group management
creation and removal of users and groups, user group membership changes, and user
credential changes
– User, group, and project security role assignments
creation or deletion of a security role, assignment and removal of security roles to users
or groups, and assignment or removal of users or groups and roles to a project.
– Engine credential mapping
assignment and removal of credentials to IBM InfoSphere DataStage® suite users and
assignment of default credentials for an IBM InfoSphere Information Server engine when
mapping credentials using the Engine Credentials panel of the IBM InfoSphere
Information Server Web console.
– User session management
user login and logout, direct session termination, and session expiration
– Audit configuration
auditing properties file location, audit file configuration settings, and audit event settings.
21
21
© 2010 IBM Corporation
Upgrade Workshop
Operations & Run Time
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
Increased support for the management and
performance of the run time environment
●
New Platform Support
●
Run-time Performance Improvements
●
Job Parameter Sets
●
Runtime Optimizations
●
Machine Resource Estimation
●
Job Performance Data Analysis
●
Serviceability Tools
●
Documentation
●
Server-side import/export (via new istool command line utility)
Connectivity
22
22
© 2010 IBM Corporation
Upgrade Workshop
Platform Support











Red Hat Enterprise Linux 5,6 (64 bit)
SUSE Linux Enterprise Server 9,10 (64 bit)
Windows Server 2008 64bit (32-bit app)
AIX 5.3, 6.1 (64 bit)
Solaris 9,10 (64 bit)
HP-UX Itanium (64 bit)
Red Hat Enterprise Linux for System Z (64 bit)
SUSE Linux Enterprise Server for System Z (64 bit)
Windows Server 2003 (32-bit)
Red Hat Enterprise Linux 5, 6 (as 32-bit app)
SUSE Linux Enterprise Server 9, 10 (as 32-bit app)
 Clients – Windows XP, Vista and 7 (32 & 64 bit)
 Repository – DB2 9/5, 9.7, Oracle 10g, 11g, SQL Server 2005,
2008
23
23
© 2010 IBM Corporation
Upgrade Workshop
Parameter Sets
•
Job Parameter Sets
• New object in repository
that contains the names
and values of job
parameters.
• A Job Parameter set can
be referenced by one or
more jobs enabling easier
deployment of jobs across
machines and also
enabling easy propagation
of a changed job
parameter value
24
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
Run Time Optimizations
●
Buffer Optimization
– Improved buffer placement algorithm
–
●
Improved Job Startup Time
–
●
25
E.g., Removed unnecessary buffer before parallel sort in some instances
Startup time improvements allow efficient use of EE against smaller data
sets
Adaptive Job Monitoring
–
The primary function of the Adaptive Job Monitoring feature is to detect
when CPU utilization by the conductor reaches 80% and throttle the
volume of jobmon data by sending control messages to the players to
reduce the output rate
–
When 80% CPU utilization by the conductor is reached, a warning
message will be issued to the user
–
Note: only monitor messages will be throttled, metadata and summary
messages are not affected
© 2010 IBM Corporation
Parallel Only
Upgrade Workshop
Machine Resource Estimation
●
Three modes:
– Estimate
– Re-Estimate
●
– Run
Estimate Mode
– For a given job, provides
estimates for the disk space
required and CPU utilization.
– Two models:
•
•
26
Static – provides disk space
estimates based on schema
and job design
Dynamic – provides
calculated estimates by node
based on a run of the job
against a sample of the data
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
Job Performance Data Analysis
• Provides a graphical display
of job performance and
utilization based on a job run
• The type of data includes
CPU time, system time,
elapsed time, memory
(heap) size, processed
number of records
• The data is presented as
sub-phases and phases for
each of the stage/operator
execution in the job
• Can view all nodes in the job
or specific nodes in the job
27
© 2010 IBM Corporation
Upgrade Workshop
Connectivity
Developer
Usability
Functional
Stages
Administrator
Features
Operations
& Runtime
Connectivity
28
28
Maximize the reach across the organization to easily
access various types of data
●
New Connectors (available on Server canvas in 8.5)
●
Local transaction support
●
z/OS File Stage
●
Distributed Transaction Stage
●
CDC integration through DTS
●
Connection Objects - Meta Data Reuse
●
SQL Builder Enhancements
●
Netezza Enterprise Stage
●
iWay Enterprise Stage
●
WebSphere II Federation
●
Stored Procedure Plug-in Support (SQL Server & Teradata added)
© 2010 IBM Corporation
Upgrade Workshop
Common Connectors
●
One component to access data from same
source/target
–
●
●
Combines & extends features of existing
connectivity stages
Ease of Use - improved and consistent
interfaces
●
Extended Functionality
●
Better Performance
●
DBMS Version Management
●
29
Supports DataStage, QualityStage, & Information
Analyzer
Common Connectors can co-exist. Supersede
DS SE plug-ins and DS EE Operators, yet, they
can co-exist. Where IBM is investing and
adding new capabilities.
Connectors

DB2

ODBC

WebSphere MQ

Oracle

Teradata


DTS – Distributed
Transaction Stage
(XA)
Migration Tool to
convert existing jobs
to use new
Connectors
© 2010 IBM Corporation
Upgrade Workshop
Connection Migration Tool






30
30
Modifies jobs that use legacy plugin/operator stages to use newer
Connectors
Migrates all compatible stages
GUI and command line (batch) modes
Server and Parallel jobs
Backup, clone or replace jobs
Jobs are annotated with information
about the migration
© 2010 IBM Corporation
Parallel Only
Upgrade Workshop
Distributed Transaction Stage
 The Distributed Transaction Stage (DTS) utilizes the
WebSphere MQ Transaction Manager to enable distributed
two-phase transactions across multiple resources.
 It works in collaboration with the WebSphere MQ Connector
to move source messages to database targets.
 Currently supports MQ (Source), DB2. Adding support for
MQ (Target), Teradata, Oracle and ODBC
Source
Queue
Target
Database
Source data arrives on
MQ queue from some
external application
31
31
Business logic transforms
the data to construct target
actions
DTS updates target and
deletes message from MQ
source queue.
© 2010 IBM Corporation
Upgrade Workshop
Multiple Input Link Support for Connectors
●
Support Local Transaction Grouping
– Insert/Update/Delete
– Commit all rows, on all links, or fail / roll back
32
●
Both batch and real-time
●
Support for SQL Error code and reject links by link
© 2010 IBM Corporation
Upgrade Workshop
Parallel Only
z/OS File stage (already enabled for V8.1)
•
•
•
33
33
New Native Support for Accessing Mainframe files from Distributed platforms and
Linux for System Z
– Through a new stage called z/OS File stage
– VSAM files - KSDS, ESDS, RRDS
– Sequential files - QSAM, Sequential read of BDAM/BSAM, PDS members,
GDG files.
Initial release
– Read/Write for Sequential files and read only for VSAM.
– Fixed and variable-length records
– single or multi record type format files will be supported
Leveraging InfoSphere Classic Federation
© 2010 IBM Corporation
Upgrade Workshop
Tack för uppmärksamheten
Peter Bjelvert
IBM Software Group
InfoSphere Services
peter.bjelvert@se.ibm.com
www.ibm.com/software/data/infosphere
34
© 2010 IBM Corporation