Informatica Productivity Pack
Save Time and Money while Increasing the
Quality of Your PowerCenter Deployment
Louis Hausle
1
Are these issues you have run into?
• Best Practices Violations even though you have
them documented!
• Abuse and misuse of SQL overrides, log
verbosity or commit/cache/dtm buffer size?
• Zombie Workflows?
2
Where is Time Spent in a PowerCenter Environment?
(source Informatica World 2012 Poll)
• Development
accounts for less
than 50% of time
Code Review
and Best Practices
Enforcement
10%
Monitoring of
P0werCenter
21%
Data
Validation and
Testing
29%
Development
38%
• More Time and Money
Spent on Testing and
Monitoring
• You can shrink the
TCO of PowerCenter
by automated Testing
and Monitoring
3
What Exactly is the Productivity Pack?
( it is a Sales Bundle of Three Products)
DEV
TEST
PROD
PowerCenter Productivity Pack
Proactive
Proactive
Data
Monitoring for
Monitoring for
Validation
PowerCenter
PowerCenter
Option
- Governance
- Operations
4
Agenda
• Development – Proactive Monitoring for
PowerCenter – Governance
• Testing – The Informatica Data Validation
Option – DVO
• Production – Proactive Monitoring for
PowerCenter - Operations
5
Every PowerCenter Customer Has Best Practices,
Nearly Every One Struggles to Enforce Them!
6
Challenges in Data Integration & Governance
• Best Practice Enforcement
• Most companies have no way to
enforce best practices
• Production Impact
• Unchecked Development Practices
and Choices can lead to Fatal
Mistakes in Production
• Measure and Quantify
Development Work
• How can you track many developers in
different locations?
7
1. Enforcement of Best Practices
Customer Challenges
•
How do you monitor many developers in many locations, all the time?
•
How do make your Best Practices Actionable and enforceable as they
happen?
Mappings are
created and saved
without comments
Next Developer
has no idea of
what logic is doing
Time is wasted
and Mistakes can
be made
8
2. Production Impact of Development Options
Customer Challenges
•
There are hundreds of development flags and options on Mappings,
Workflows, Sessions and Transformations
• Customers can have dozens, hundreds and thousands of Workflows
A developer uses the
‘truncate target table’
option for testing in
development
He forgets to change
the flag, and the
workflow is migrated
into production
Data is lost, and
recovery steps can take
time and introduce new
data problems
9
Agenda
• Development – Proactive Monitoring for
PowerCenter – Governance
• Testing – The Informatica Data Validation
Option – DVO
• Production – Proactive Monitoring for
PowerCenter - Operations
11
The Data Validation and Testing Dilemma
Informatica has over 4,000 PowerCenter
Customers who believe using a
metadata driven, graphical tool to move
data is better than writing SQL and
hand coding.
15 years later, 99% of those 4,000
Customers still use SQL and Excel to
test the data they have moved in data
integration products.
There has to be a better way…
12
Current Approach: Like a Photo Hunt
13
Data Validation Simple Illustration
Source A
Sales
Orders
Sales
Amount
30
$150,000
Source B
Sales
Orders
Sales
Amount
23
$150,000
Target
(DW or Operational
System
Data Integration
Or Movement
Sales
Orders
Sales
Amount
52
$290,000
DVO - tells you 1 Order and $10,000 is Missing
In a few clicks it does set, aggregation and value tests.
14
DVO: Test Creation
Leverage your PowerCenter Metadata!
GUI with Pre-Built Test Operators with NO SQL
15
DVO: Test Results
Summary of Test Results
16
Agenda
• Development – Proactive Monitoring for
PowerCenter – Governance
• Testing – The Informatica Data Validation
Option – DVO
• Production – Proactive Monitoring for
PowerCenter - Operations
17
Informatica built these solutions to address three
common customer concerns…
1. Customers asked for a “Complete, PowerCenter
Centric” Monitoring Solution
2. Customers want to become “Proactive” in seeing issues
before they turn into problems. Send email alerts instead
of logging in to a monitor.
3. Scalable way to identify anomalies such as Workflows
running longer/shorter than normal or returning
more/less rows than normal.
18
What is Proactive PowerCenter Monitoring?
 Prebuilt Rules to Monitor
Informatica PowerCenter
 Delivers Alerts to people,
dashboards and systems
 Complete Operational and
Best Practices
 Extensible – build unlimited
number of new rules
19
Solution Overview
Operational
Monitoring Monitoring
via
Proactive
PowerCenter
Continuous Correlation
(Sample Rules)
SQL Sources to
Expose Repository
Features and Allow
Easy Rule and
Report Building
Environmental
Monitoring
Custom:
Environmental
(CPU, Memory..)
DB (SQL)
Runtime Statistics
JMSvia Web
Receiver
Web
Services
Services
Hub
Ping PowerCenter
Services
•
•
Operational
Design Time
1
Workflow/session was successful but zero
(0) records loaded
2
Rejected Records in Session
3
Disproportionate elapsed time between
Workflow and Session
4
Session/Workflow Failures
5
Correlate session changes to workflow
failures
SLA Violations: Workflow/Folder/Repository
Level
Significant increase in Workflow/Session
Elapsed Times
6
7
Notification
Framework for
Alerts
Prebuilt EMAIL
Responses
Prebuilt Dashboard
Alerts
Alert Enrichment
PowerCenter
Workflow Response
to Execute
Commands
Web Services and more…
Significant decrease in applied row counts
8 DBs,
9
Workflows
missing
schedules
Alert
Editing,
Creation
and Management
10
Too many concurrent workflows
11
Automatic Restart of Zombie Workflows
12
Ping Domain/Integration Service/Repository
Service
•
•
•
Start/Stop Workflows
Create Help Desk Ticket
Update Control Tables
20
Proactive Monitoring for PowerCenter
Illustration of Components that are Monitored
PowerCenter
1. The PowerCenter Domain, Integration
Services and Repository Services are
Monitored
Database
Domain
Repository
Service
3. The PowerCenter Server OS is
Monitored for Memory and CPU Usage
Administrator
Integration
Service
Server
Application Server
2. The PowerCenter Repository Database
is Monitored for Up/Down and
Tablespace Usage
4. Running Workflows and Sessions are
Monitored
5. Over 500 Attributes can be used in rules
21
DEMONSTRATION of
PROACTIVE MONITORING FOR
GOVERNANCE
• Check for Commit size violations
• Check On Rollback Transactions on Errors
• Naming Conventions and Check for
Comments
22