Session id: 40208
Oracle Streams--Simplifying
Information Sharing in
Oracle10g
Patricia McElroy
Product Manager
Oracle Corporation
Patricia McElroy
Product Manager,
Distributed Systems
Oracle Corporation
Oracle Streams-Simplifying
Information Sharing
in Oracle10g
Agenda
 Oracle Streams Overview
 Streams Architecture
 Hints and Tips
Oracle Streams
 Simple solution for information sharing
 Provides
–
–
–
–
uniquely flexible replication
message queuing
data warehouse loading
event management and notification
Streams Basic Elements
Capture
Staging
 Capture
 Staging
 Consumption (apply)
Consumption
Multi-Database Streams
Capture
Staging
Consumption
Capture
Staging
Consumption
 A stream can
contain multiple
elements from
multiple
databases
 Events flow
between
staging areas
Capture
Capture
 Streams captures events
–
–
Implicitly: log-based capture of DML and DDL
Explicitly: Direct enqueue of user messages
 Captured events are published in the staging area
 SQL and messaging APIs in multiple languages
–
JDBC, JMS, PL/SQL, C, SOAP
Log-Based Change Capture
Capture
 Low overhead, low latency change capture
–
–
–
Changes to the database are written to the online
redo log
Oracle9i Streams can extract changes from the
log as it is written (hot mining)
Changes are formatted as a Logical Change
Record (LCR), a representation of the change
Logical Change Record
(LCR)
 Database change = LCR
– DML
 Object name, owner, Type of DML, SCN
 Row change = LCR
 OLD, NEW values
– DDL
 Object name, owner, Type of DDL, SCN
 DDL text
– LOB
 Multiple LCRs per LOB
 Piecewise chunks
Staging
Staging
 Streams publishes captured events into a staging
area
–
–
–
–
Implemented as a queue
Supports for new type “any” datatype allows a single
staging area to hold any type of data
All events, LCRs and user-messages, can be staged in the
same queue
Messages remain in staging area until consumed by all
subscribers
Staging Area Propagation
 Other staging
areas can
subscribe to
events
Staging
–
Propagation
Staging
–
in same
database
in a remote
database
 Events can be
routed through a
series of staging
areas
Transformations
Staging
 Transformations can be performed
–
–
–
as events enter the staging area
as events leave the staging area
as events propagate between staging areas
 Transformation examples
–
change format, data type, column name, table
name
Consumption
Consumption
 Staged events are consumed by subscribers
–
Implicitly: Apply Process
 Default Apply
 User-Defined Apply
–
Explicitly: Application dequeue via open
interfaces
 JMS, C, C++, PLSQL, SOAP (XML/HTTP)
Default Apply
Consumption
 The default apply engine will directly apply the
DML or DDL represented in the LCR
–
–
apply to local Oracle table
apply via DB Link to non-Oracle table
 Automatic conflict detection with optional
resolution
–
unresolved conflicts placed in exception queue
 Parallel apply maximizes concurrency
User-defined Apply
Consumption
 User-written custom apply procedures
 Written in PL/SQL, Java, C, C++
 Uses:
–
–
–
full control over apply
normalizing or denormalizing data
populating related fields or tables
Rule-based Configuration
 Consumers subscribe to published events
 Content-based subscription
 Rule is expressed as SQL WHERE clause
dbms_rule_adm.create_rule(
rule_name=>‘scott.rule1',
condition=>':dml.get_object_owner() = ''SCOTT'' AND
:dml.get_object_name()=''EMP''');
 Rule sets for simplicity
 Rule sets govern capture,staging, and apply
–
Negative, Inclusion
 Dynamic rule maintenance
Replication Rules
 Rule-based
–
–
–
DML, DDL
content
True/False
 Granularity
–
–
–
–
–
Table
Subset of Table
Schema
Database
Tablespace
 Tailored Replication API
DBMS_STREAMS_ADM
–
–
–
–
–
–
ADD_TABLE_RULES
ADD_SCHEMA_RULES
ADD_GLOBAL_RULES
ADD_SUBSET_RULES
MAINTAIN_TABLESPACES
ADD_MESSAGE_RULE
Directed Networks
INSERT …
VALUES (‘EUROPE’,’ ITALY’)
 Propagation
independent of Apply
 Rules-based
subscription determine if
event is locally applied
–
NY
(master)
EUROPE
London
(subset)
London applies UK only
 WAN Friendly
–
–
Send once, fan out
NY-->London,
London-->Milan,
London-->Paris
ITALY
Milan
(subset)
FRANCE
Paris
(subset)
Automatic Conflict Detection
 Automatic conflict detection with userselectable conflict resolution routines
–
–
latest timestamp, earliest timestamp, maximum or
minimum value, overwrite, discard
User-definable resolution routines
 Conflict detection compares current row
values at receiving site with “old” values of
changed row from the originating site
–
–
–
if match, “new” values are applied to row
if not, conflict resolution method is used, if supplied
if still unresolved, place transaction in exception queue
 Ability to disable conflict detection by column or
table
Streams Capture and Apply
empid|job
Update EMP set
job=‘coding’
where
empid=510;
EMP
100 | sales |…
510 | coding| ...
Queue
----LCRs
Capture
Redo Log
|..
Propagation
ACK
Queue
-----LCRs
Apply
EMP
Capture
• Background process, CPnn
• Invokes Logminer
• Staged in buffer queue (SGA)
Queue
----LCRs
Ack
• Restartable
Capture
Redo Log
•Automatic Flow Control
Capture Restart
• Every database change is in redo
• Ordered by SCN
• Last enqueued SCN
• capture restart
• Last acknowledged SCN
• instance restart
Queue
----LCRs
Ack
Capture
Redo Log
Downstream Capture
 Zero impact at source site
 Changes are captured by another database
–
–
Same Platform
Archive logs
 Log transport services/ RFS to move logs
Propagation
• Job Queue processes
IMPLICIT
CAPTUR
E
IMPLICIT
APPLY
Propagation
LCRs
In memory
In memory
Queue/
Queue/
LCRs Acknowledgements LCRs
• Stream LCRs
• Maintain status
information
• 9i:Re-propagate on
instance crash
•10g: Reuse spilled
messages during
recovery
Apply
empid|job
|..
100 | sales |…
510 | coding| ...
Queue
-----LCRs
Reader
EMP
• Background processes
• Assembles transactions
• Exactly once apply
• Dependency computation
• Conflict detection
Coordinator
Ack
Apply
Server
Server
Server
…..
Transformation Example
empid|job |..
Update EMP set
job=‘coding’
where
empid=510;
EMP
100 | sales |…
510 |coding| ...
Queue
----LCRs
Capture
Propagation
ACK
Queue
-----LCRs
Person
Apply
Redo Log
•1-1 Mapping
•LCR based
T
Emp->Person
Customized Apply Example
empid|state|..
Update EMP set
state=‘CA’
where
empid=100;
105 | LA | ...
Queue
----LCRs
Capture
Redo Log
Propagation
ACK
Queue
Emp
-----LCRs Audit
Apply
•Full Control
•Transaction Based
U
100 | CA |…
EMP
95 | GA |…
Bit
Bucket
Do not apply
Changes to
EMP for
state=‘CA’;
Maintain
AUDIT table
of each LCR
Data Subsetting
empid|state|..
100 | CA |…
Update EMP set
job=‘CA’ where
empid=100;
Queue
-----LCRs
ACK
EMP
205 | CA | ...
CA
Queue
----LCRs
Capture
EMP
Apply
empid|state|..
Propagation
100 | LA |…
105 | LA | ...
LA
ACK
Queue
-----LCRs
Redo Log
Apply
EMP
Streams and RAC
 Oracle9i
–
–
–
No hot mining
All Streams activities done on a single instance
(owning instance)
Streams restart after failure not automatic
 Oracle10i
–
–
–
Hot mining
Primary and secondary owning instance
Failover of Streams processes automatic
Messaging Enhancements





Streams encompasses AQ
Simplified messaging APIs
Batch enqueue/dequeue
Criteria-based purge
Gateway to Tibco
Heterogeneous Support
LCR or
user message
 Oracle to non-Oracle
Apply via gateway
–
Apply process on
Oracle node applies
change
 Non-Oracle to Oracle
change capture
supported via explicit
enqueue of LCRs
 Message Gateways
Gateway
MQ Series
Sybase
–
Message
Gateway
MQ Series
Heterogeneous Example
empid|state|..
Update EMP set
state=‘CA’
where
empid=100;
EMP
100 | CA |…
105 | LA | ...
Queue
----LCRs
Capture
Redo Log
Propagation
ACK
Queue
-----LCRs
EMP
Apply
Apply2
TG4
Sybs
EMP
Sybase
Customer Example
UK
 Maintain application tables in
3 regional centers
JP
–
–
–
US
 Provide Special Services
–
DR
Report
autonomous
low overhead
low latency
Gateway
–
Sybase
–
Reporting Database
Disaster Recovery
Legacy Application hosted
on Sybase
Streams Replication Features







Log-based Change Capture
Customizable Apply Engine
Directed Networks
Schema Evolution
Transformations
Heterogeneous Support
Explicit Enqueue/Dequeue
Streams Benefits for
Replication




Flexible Configurations
Reduced Network Traffic
Low Overhead
No Downtime Requirements
Software Updates
 9.2.0.4 patch, minimum 9.2.0.2
 Streams Wizard patch
 Watch OTN Streams website for custom
software
–
http://otn.oracle.com/products/dataint/content.html
9.2.0.4 Configuration Tips
 Use separate queues
–
–
Capture
Apply
 Increase shared_pool_size and
max_sga_size init.ora parameters
 Relocate Streams data dictionary tables from
SYSTEM
–
–
Set tablespace before configuring streams
Including apply-only sites
Rule Tips
 Spelling counts!
 Eliminate duplicate rules in a rule set.
 Make sure that the source_database_name is
correctly specified.
–
At source site: SELECT global_name FROM
GLOBAL_NAME;
 Ensure that rules do not allow objects with invalid
data types
– When using GLOBAL rules, modify existing rule
to eliminate the Streams Administrator schema
and other schemas with queues, ADTs or tables
with invalid data types.
Rule Management Tips
 Having no rule set defined is NOT the same
as an empty rule set.
 Use DBMS_STREAMS_ADM package to
create replication rules
 Use same package to create or remove rules
–
DBMS_STREAMS_ADM
–
DBMS_RULE_ADM
Troubleshooting Tips
 Message Number columns = source DML,DDL SCN
–
–
CAPTURE_MESSAGE_NUMBER, ENQUEUE_MESSAGE_NUMBER
DEQUEUE_MESSAGE_NUMBER, APPLIED_MESSAGE_NUMBER
 ORA-26687 no instantiation scn set
–
–
–
Set Instantiation SCN
Exp/Imp with appropriate clauses
DBMS_APPLY_ADM.SET_TABLE_INSTANTIATION_SCN
 ORA-01031 insufficient privilege
–
Explicitly grant privileges to apply user
 MISSING Streams data dictionary information!
–
DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION
Troubleshooting #2
 Determine the scope of the problem:
–
Apply not working for this site only
 Check DBA_APPLY_ERROR for errors
- Procedure to display error details in doc
 Check trace files
 Check rules at this site
 Still not working? …
–
Apply not working for any site
 Check rules for both propagation and capture
 Check propagation job is working
- Job queue processes
- DB Link
 Make sure COMMIT was performed for transaction
Operational Tips
 Replicating DDL?
–
–
–
Ensure that privileges are explicitly granted to the apply
user.
Modify any manual hotbackup scripts to set an apply tag
before starting the backup.
Configure Instantiation SCN at next higher level
 Removing Archive log files from disk?
–
Ensure that scripts do not automatically remove files that
may be needed for capture to restart.
 Implement “heartbeat” table
–
Update periodically.
Next Steps….
 Recommended sessions36637
–
Data Integration with Oracle Streamsata
Wed 4:30 Room 130 Integration Using Oracle St
 Recommended demos and/or hands-on labs
–
Hands-On Lab: Integrate your Information.
 See Your Business in Our Software
–
Visit the Streams booth in the DEMOgrounds for more
information.
 Relevant web sites to visit for more information
–
http://otn.oracle;com/products/dataint/content.html
QUESTIONS
ANSWERS