Integrated Data Management
@db2Dean
facebook.com/db2Dean
www.db2Dean.com
dcomphe@us.ibm.com
Data Security & Privacy for iSeries
⦁Dean Compher
⦁Click
to add text
⦁Big Data Portfolio Technical Sales Specialist
© 2009 IBM Corporation
Integrated Data Management
Perimeter Defenses No Longer Sufficient
“A fortress mentality will not work in cyber. We
cannot retreat behind a Maginot Line of firewalls.”
- William J. Lynn III,
U.S. Deputy Defense Secretary
Insiders
(DBAs, developers,
outsourcers, etc.)
Outsourcing
Stolen
Credentials
(Zeus, etc.)
Web-Facing Apps
Legacy App
Integration/SOA
Employee Self-Service,
Partners & Suppliers
2
© 2009 IBM Corporation
Integrated Data Management
Addressing the Full Lifecycle of Database Security &
Compliance
© 2009 IBM Corporation
Integrated Data Management
Agenda
• Data Security – Guardium Database Activity Monitoring
• Alert on Access Policy Violations
• Audit and Report Activity
• Data Privacy – Otpim Test Data Management
• Mask Data Copied to Test
• Create Subsets
• Automate Test Data Refresh
• Improve Security with Better Testing
© 2009 IBM Corporation
Integrated Data Management
Guardium Database Activity Monitoring
© 2009 IBM Corporation
Integrated Data Management
Real-Time Database Monitoring with InfoSphere
Guardium
Host-based Probes
(S-TAPs)
•
•
•
Collector
Non-invasive architecture
• Outside database
• Minimal performance impact (13%)
• No DBMS or application changes
Cross-DBMS solution
100% visibility including local DBA
access
•
•
•
–
•
Enforces separation of duties
Does not rely on DBMS-resident logs
that can easily be erased by attackers,
rogue insiders
Granular, real-time policies & auditing
Who, what, when, how
Automated compliance reporting, signoffs & escalations (SOX, PCI, NIST,
etc.)
© 2009 IBM Corporation
Integrated Data Management
Scalable Multi-Tier Architecture
iSeries
•Integration with LDAP,
IAM, IBM Tivoli SIEM,
IBM TSM, Remedy, …
© 2009 IBM Corporation
Integrated Data Management
Extend real-time Data Activity Monitoring to also protect sensitive data in
data warehouses, Big Data Environments and file shares
DATA
Big Data
Environments
NEW
InfoSphere
BigInsights
Integration with
LDAP, IAM,
SIEM, TSM,
Remedy, …
© 2009 IBM Corporation
Integrated Data Management
Extended data security platform coverage
• Providing complete and native data security solution for System I (DB2 6.1, 7.1)
System i
S-TAP for
System i
S-TAP for System i
• Monitors privileged user activity in real
time
• Enables complete separation of duties
• Helps satisfy auditor’s requirements and
ensure compliance
Protect sensitive data on
your System i deployments,
ensuring compliance to
mandates like PCI easily
and cost effectively
© 2009 IBM Corporation
Integrated Data Management
3 Types of Rules
Exception (ie. Invalid table)
3
Result Set
2
1
SQL Query
Database
Database Server
There are three types of rules:
1. An access rule applies to client requests
2. An extrusion rule evaluates data returned by the server
3. An exception rule evaluates exceptions returned by the server
© 2009 IBM Corporation
Integrated Data Management
Fine-Grained Policies with Real-Time Alerts
Application
Server
10.10.9.244
Database
Server
10.10.9.56
© 2009 IBM Corporation
Integrated Data Management
2. Extrusion Definition to Alert on Unauthorized Results Set
• Monitor 10.10.9.248
• SQL Server database
• Not user Bill
• Send Alert per match
© 2009 IBM Corporation
Integrated Data Management
Monitoring Data Extrusion
•Should my customer service rep view 99 records in an
hour?
•Is this normal?
© 2009 IBM Corporation
Integrated Data Management
3. Policy Exception Rule - Preventing Attacks
Rogue users know
what they’re looking
for, but...
They don’t always
know where to find it!
SQL injection leads
to SQL errors!
Brute force attacks
result in failed
logins!
Guardium: 100% visibility with real-time alerts …
© 2009 IBM Corporation
Integrated Data Management
Identifying Fraud via Application-Layer Monitoring
•Joe
•Marc
• Issue: App server uses generic service account
to access DB -- which doesn’t identify WHO
initiated transaction (connection pooling)
• Solution: Track access to application user
associated with specific SQL commands
• Deterministic identification vs. time-based “best guess”
•AppUser
• Out-of-the-box support for all major enterprise apps (Oracle
EBS, PeopleSoft, SAP, Siebel, Business Objects, Cognos,
etc.)
• Plus custom apps (WebLogic, WebSphere, Oracle AS, etc.)
•Application
Server
•Database
Server
• No changes to applications
© 2009 IBM Corporation
Integrated Data Management
Workflow Automation
• Schedule & automate tasks
• Compliance reporting
• Automatically generate reports
• Distribute to oversight team
• Track electronic sign-offs
• Escalate when required
• Store process trail in secure
repository
• Demonstrates oversight
process for auditors
© 2009 IBM Corporation
Integrated Data Management
Accelerators
• Software modules harnessing Guardium's extensive capabilities to
address the requirements of security mandates
• Customizable mandate-specific reports, policies, tools and
workflows
• Greatly improve security and streamline audit preparation
• Increased operational efficiency through automation of compliance
• Simplified validation of broad ranges of requirements
SarbanesOxley
23
Base II
HIPAA
GLBA
PCI
© 2009 IBM Corporation
Integrated Data Management
Protect data in real-time and ensure compliance in unstructured
Hadoop big data environments
Big data environments help organizations:
Process, analyze and derive maximum value from these new data
formats as well as traditional structured formats in real-time
Make more informed decisions instantaneously and cost effectively
•Turn 12 terabytes of Tweets into improved product sentiment analysis
• Monitor 100’s of live video feeds from surveillance cameras to identify security threats
Big data brings big security challenges
As big data environments ingest more data, organizations will face
significant risks and threats to the repositories in which the data is kept
NEW
Introducing Hadoop Activity Monitoring
Monitor and Audit Hadoop activity in real-time to support compliance requirements and protect data
•
•
•
•
Real time activity monitoring of HDFS, MapReduce, Hive and HBASE data sources
Automated compliance controls
Fully integrated with InfoSphere Guardium solution for database activity monitoring
View Hadoop systems with other data sources
© 2009 IBM Corporation
Integrated Data Management
Expand system openness and integration with Universal Feed
Universal Feed opens InfoSphere Guardium system, enabling all
capabilities to be applied to custom applications and niche data sources
• Open InfoSphere Guardium protocol (agent to Collector)
integration to clients and 3rd party companies
 Provides a means of supporting fragmented segments of the market:
custom applications, niche databases, etc.
 Data auditing model; not a SIEM
• Customer/partner responsible for developing interface to system
to be integrated (e.g. S-TAP equivalent)
 Open industry standard protocol used to simplify development
• Supports full capabilities, or subset of InfoSphere Guardium
capabilities
 Monitoring and protection
 Real-time
 Secure audit trail, compliance workflow automation, etc.
© 2009 IBM Corporation
Integrated Data Management
Universal Feed Overview
Agent
developer
Universal
Feed Agent
Guardium
Toolkit
Guardium
Appliance
Sending Audit Data via
Guardium messages
-- - --- - - - - - - --- -- - - -
Capturing
Events
Sending Information
Receiving &
processing
Universal Feed Agent
 Agent developer for universal feed agent
 Partner
 Customer
 3rd Party
 Responsible for capturing events with audit
interest
 Responsible for sending the audit data
using Guardium defined messages
 Responsible for receiving and processing
Guardium messages (policies, pings, etc)
-- - --- - - - - - - --- -- - - -
Send
Alert
Guardium Collector
 Accepting connections from the Universal
Feed Agent
 Processing and storing audit data
 Sending information to Universal Feed
agent (policy, pings, etc)
 Alerting if Universal Feed Agent doesn’t
send heart beat
© 2009 IBM Corporation
Integrated Data Management
InfoSphere Optim Data Privacy
& Test Data Management
© 2009 IBM Corporation
Integrated Data Management
InfoSphere Optim: Intelligent Move of Structured Data
Production
Data Privacy
for Test Data
Production Archive or
Development Test Data
Extract
Retrieved
Source
Data
Reference Data
Contextual Data
Restore
Current
SQL access to
Archived Data
Universal Access to Archived Data
Application
ODBC / JDBC
XML
Report Writer
IBM Mashup
Intelligent Move of Structured Data is a process that captures contextual
source data for the purpose of Archiving and Accessing historical data
or Populating Test Databases with privatized data
© 2009 IBM Corporation
Integrated Data Management
Supporting Enterprise Environments
Discovery
Test Data Management
Data Privacy
Data Growth
Application Retirement
Organization environments are diverse, yet interrelated therefore what
you use to manage the data MUST support across your environment
© 2009 IBM Corporation
Integrated Data Management
Our Unique Capability: The Complete Business Object
Business view
“reference snapshot”
of business activity
DBA view
Referentially-intact
subset of data
Federated
access to data
and metadata
Related LUW
Files or
Documents
Oracle
DB2
Sybase
Adabas
© 2009 IBM Corporation
Information
Management
Integrated
Data Management
Example: JD Edwards Accounts Payable Archiving
Ledger Tag
F0911T
Account Ledger
Company Master
Tax Area
F4008
F0911
Tax table
F0018
F0901
Account
Balances
Account
Master
F0006
AP Ledger
AP Header
F0411
F0413
F0101
F0005
UDC
F0008
AP Details
F0013
F0909
- Reference Only
- Archive Only
- Archive & Delete
AAI’s
F0025
LT Master
F0414
Currency Codes
F0014
31
F0010
Fiscal Date Pattern
F0902
Chart of A/C
AB Master
F0012
Batch Control
F0011
BU Master
Payment Terms
F0401
Supplier Master
F0015
Currency Ex. Rate
F00151
Currency Exchange
Rate Header
F11151
F0004
UDC Types
F1113
Currency
Restatement
Rate
Currency Ex.
Rate Calculation
31
© 2009 IBM Corporation
Integrated Data Management
A Word About Relationships...
Optim
DIRECTORY
Tables
Relationships
Access
Definitions
OPTIM
Referential
Integrity
Rules
DB Aliases
Stored in Database
- Catalog
- System Tables
- Data Dictionary
Maps
•
DB Relationships are automatically derived from database
RI rules
•
Application Specified Relationships
• Can be defined individually to Optim
• Can be imported into Optim from DDL
• Can be automatically discovered by InfoSphere Discovery
•
Shared by all Optim components
© 2009 IBM Corporation
Integrated Data Management
Automate Discovery and Accelerate Information Understanding
• Significant Acceleration of Information
Agenda projects
• Application/Data Consolidation, Migration
& Retirement
• Data Growth Management
• Master Data Management and Data
Warehousing
• Test Data Management
• Sensitive Data De-identification
• Why is this Different?
• Data-based discovery
• Automate discovery of business entities,
cross-source business rules &
transformation logic
• Evaluate multiple data sources
simultaneously
• Identify & remediate cross-system rules
and inconsistencies
© 2009 IBM Corporation
Integrated Data Management
InfoSphere Optim Deep Dive:
Test Data Management
© 2009 IBM Corporation
Integrated Data Management
Drivers for Test Data Management Projects
• Quality
•
•
•
•
Bad data
Unidentified test cases
Test Automation approach (Rational Borland MI…)
Verification of test results
• Parallelism (Multiple Sandboxes)
• Tunnel effect
• Multi project testing
• Storage
• Reduce storage
• Include into a cost control project
• Data Privacy / Compliance
© 2009 IBM Corporation
Integrated Data Management
How Does Test Data Management Impact Storage Cost?
Before
TDM
With
TDM
Production
500GB
500GB
Training
500GB
25GB
Unit Test
500GB
25GB
System Test
500GB 500GB
UAT
500GB
25GB
Integration
500GB
25GB
2.5TB
0.6TB
Training
Unit Test
Production
Integration
System
Test
UAT
Test Data
© 2009 IBM Corporation
Integrated Data Management
InfoSphere Optim Test Data Management Solution
Create/Modify
Application
Copy Production Data
for Testing
Correct Errors in
Production Data
Relational Extract
Relational Edit
Subset and Privatize
Relational Edit
Inspect and Add Data
to Test Error Routines
Archive Old Data
Optim Archive
TEST
Go Production !!!
Refresh Test Data
Compare Before/After
Data
Relational Extract
37
Relational Compare
© 2009 IBM Corporation
Integrated Data Management
The Relational Extract Facility
NewDB
CUST
-- ---- ---- ---- ------- ----
Extract a relationally intact subset
from production database(s)
ORD
Create
-- ---- ---- ---- ------- ----
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
CUSTOMERS
CUSTOMERS
-- ---- ---- ---- ------- ------- ------- ------- ------- ------------- -------- ---- ---- ---- ------- ----
INSERT/
UPDATE
DETAILS
DETAILS
CUST
-- ---- ---- ---- ------- ----
ORD
ORDERS
ORDERS
-- -- ------ -- --------- ------- ---- ----------- ---- ----------------- ---------- ---- ----------- ---- ----------------- ---------- ---- ----------- ---- ----------------- -------- -- ------ -- --------- ----
TESTDB
-- ---- ---- ---- ------- ----
Extract
File
-- ---- ---- ---- ------- ------- ------- ------- ------- ------------- ---------- ------- ------- ------- ----------------------- ------- ------- ------- ------------- -------- ---- ---- ---- ------- ----
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
QADB
Load
Files
CUST
-- ---- ---- ---- ------- ----
ORD
-- ---- ---- ---- ------- ----
LOAD
• Extract data and/or object definitions
• From multiple tables (files) that are related
• From multiple tables (files) that are not related
• From single tables (files)
• All data or subset
• Define a new set of test tables
• Populate Target databases
• Refresh Target databases
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
New_DB
CUST
-- ---- ---- ---- ------- ----
Create
ORD
-- ---- ---- ---- ------- ----
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Saves:
Programmer/DBA time
Disk space utilization
Testing interference
© 2009 IBM Corporation
Integrated Data Management
Traditional vs. Relational Tools
Single Table Editors

One table/view at a time

No edit of related data
from multiple tables
FIND DETAILS
NOTE INFO
EXIT TABLE
The Relational Editor

Simultaneous browse/edit
of related data from
multiple tables
CUSTOMERS
FIND ORDERS
NOTE INFO
EXIT TABLE
FIND CUSTOMER
NOTE INFO
EXIT TABLE
ORDERS
........................
........................
........................
........................
........................
DETAILS
Speeds time to create
boundary test cases.
Simplifies edit process.
© 2009 IBM Corporation
Integrated Data Management
Optim’s Relational Compare Facility
Optim
COMPARE
FILE
SOURCE 1
........................
........................
........................
........................
........................
Interactive Browse
COMPARE
PROCESS
SOURCE 2
•
Single-table or multi-table compare
•
Creates compare file and/or compare Report of results
•
For application testing, QA, and to verify database contents
•
Enhances productivity by finding unexpected changes in the data
Optim
Compare
REPORT
Verify Test
Results
Saves QA Validation time
Improves Test Accuracy
© 2009 IBM Corporation
Integrated Data Management
Architecture: Test Data Management/Data Privacy
Server Name
• Server address or name
•DB Alias
• Connectivity via DB Client software
Work Directory
• Server File System
Storage Profile
• Storage
and retention policy
QFED
CUSTOMER
-- ---- ---- ---- ------- ----
Windows
EMPL
-- ---- ---- ---- ------- ----
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Test system 1
Optim
Workstation
HR
-- ---- ---- ---- ------- ----
EMPL
-- ---- ---- ---- ------- ----
Mask on insert
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Extract files
Test system 2
Optim Server
Oracle 9
DB2/i
HP UX
Windows, Unix, Linux, zOs
FINANCE/BUDGET
Load
Files
-- ---- ---- ---- ------- ----
EMPL
-- ---- ---- ---- ------- ----
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Application 2
FINANCE/BUDGET
-- ---- ---- ---- ------- ----
Test System 3
EMPL
-- ---- ---- ---- ------- ----
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
Sql Svr 2K ??
Windows ??
Optim Repository
(OptimDir)
Test System 4
© 2009 IBM Corporation
Integrated Data Management
Optim Data Privacy
© 2009 IBM Corporation
Integrated Data Management
Optim™ Data Privacy Solution
Test
Production
Siebel / DB2
Siebel / DB2
EBS / Oracle
Custom /
Sybase
Contextual,
Application- Aware,
Persistent Data
Masking
Custom /
Sybase
EBS / Oracle
• Substitute confidential information with fictionalized data
• Deploy multiple masking algorithms
• Provide consistency across environments and iterations
• Enable off-shore testing
• Protect private data in non-production environments
© 2009 IBM Corporation
Integrated Data Management
Drivers for Privacy of non production data
• Regulatory & Compliance
• PCI
• HIPPA
• EU Safe Harbour
• ….
• Offshoring test
• Sub subcontracting test & dev.
• Good business practice
• Sensitive data
• Training environnements
© 2009 IBM Corporation
Integrated Data Management
Data Privacy in Application Testing
Only Users authorized to see Private data
Extract a relationally intact subset
from production database(s)
INSERT/
UPDATE
CUSTOMERS
----- ---- ---- ------- ---CUSTOMERS
---- ------- ------- ------- ------------- -------- ---- ---- ---- ------- ----
CUST
-- ---- ---- ---- ------- ----
ORD
ORDERS
ORDERS
-- -- ------ -- --------- ------- ---- ----------- ---- ----------------- ---------- ---- ----------- ---- ----------------- ---------- ---- ----------- ---- ----------------- -------- -- ------ -- --------- ----
TESTDB
-- ---- ---- ---- ------- ----
Transform / mask
sensitive data
DETAILS
DETAILS
Extract
File
-- ---- ---- ---- ------- ------- ------- ------- ------- ------------- ---------- ------- ------- ------- ----------------------- ------- ------- ------- ------------- -------- ---- ---- ---- ------- ----
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
QADB
Load
Files
CUST
-- ---- ---- ---- ------- ----
ORD
-- ---- ---- ---- ------- ----
LOAD
• Most Secure Approach
DETL
-- ---- ---- ---- ------- ----- ---- ---- ---- ------- ----
• Extract data only
• Convert during extract
•Extract file already contains masked data
•Can be shared with testers to reuse
© 2009 IBM Corporation
Integrated Data Management
M
a
s

k
i
n
g
F
u
n
c
t
i
o
n
s
Column Map
 Map unlike column names
 Social Security (US ……)
 Credit Card
 Transform/mask sensitive data
 Email
 Datatype conversions
 Hash Lookup
 Column-level semantic date aging
 Literals
 Lookup
 Random Lookup
 NAME tables (US)
 Registers
 ADDRESS table (US)
 Calculations
 Shuffle
 Default values
 Substring
 Exits
 String manipulation
…
…
…
 Currency conversion
© 2009 IBM Corporation
Integrated Data Management
Consistent Masking and Propagation across the Enterprise
Client Billing Application
DB2
SS#s
SS#s
157342266
157342266
132009824
132009824
Data is masked
SSN#s
134235489
323457245
SSN#s
Masked fields
are consistent
134235489
323457245
© 2009 IBM Corporation
Integrated Data Management
Thank You
© 2009 IBM Corporation