Managing Change with Real Application
Testing and Snapshot Standby
Barry Hodges
Senior Solution Architect, Sales Consulting, Oracle NZ
“The art of progress is to preserve
order amid change.”
Alfred North Whitehead
British Mathematician and Philosopher, 1861-1947
IT Challenges
Motivation for change
Application Tier
Application Upgrades,
Object Modifications, etc
Database Tier
Database Upgrades,
Patches, Schema, Parameters,
RAC nodes, Interconnect,
OS Platforms, OS Upgrades,
CPU, Memory, etc
Storage
StorageTier
Tier
SAN Upgrades,
ASM, more disk, etc
Database Upgrade Process: Steps
1.
2.
3.
4.
5.
6.
7.
8.
9.
Gather information about PROD environment
Determine upgrade path & method
Prepare B/R strategy, clone and setup Test DB
Establish performance baseline before upgrade
Develop tests for database & applications
Test upgraded database & applications
Check performance before & after upgrade
Remediate regressions
Go Live!
Majority of effort spent in developing tests, detecting
and remediating regressions
Current Testing Methodology
Requires specialised
application knowledge
Need to
provision the
whole stack
PRODUCTION
Simulated
load 80/20?
BA’s id top n
business
transactions
TEST
Test Scripts need to
be constantly
rewritten as data
changes
Simulate load
by over
throttling?
What is the
correct
throughput?
Can we trust
the simulated
load?
Can we account for
selects?
Database Upgrade Process: Steps
1.
2.
3.
4.
5.
6.
7.
8.
9.
Gather information about PROD environment
Determine upgrade path & method
Prepare B/R strategy, clone and setup Test DB
Establish performance baseline before upgrade
Develop tests for database & applications
Test upgraded database & applications
Check performance before & after upgrade
Remediate regressions
Go Live!
Real
Majority
Application
of effort Testing
spent inprovides
developing
realistic
tests,testing
detecting
for
and remediating
real-world systems
regressions
Database Replay
• Replay actual production database workload in test
environment
• Identify, analyse and fix potential instabilities before making
changes to production
• Capture Workload in Production
–
–
Capture full production workload
Move the captured workload to test system
• Replay Workload in Test
–
–
–
Make the desired changes in test system
Replay workload with with production load, timing & concurrency
characteristics
Honor commit ordering
• Analyze & Report
–
–
–
Errors
Data divergence
Performance divergence
SQL Performance Analyzer
• Identify SQL performance regression before end-users are
impacted
• Identify, analyse and fix potential instabilities before making
changes to production
• Capture SQL in Production
–
–
Capture SQL workload in production including statistics & binds
Move the SQL Tuning Set to test system
• Re-execute SQL queries in Test
–
–
Make the desired changes in test system
Statements can be re-executed as often as is necessary in the
test environment
• Analyze & Report
–
–
–
–
Highlight improvements
Highlight regressions
Highlight highest impact statements
Use SQL Tuning Advisor to tune regressed statements
SQL
Application Tier
Application Upgrades,
Some Object Modifications
Unsupported
Supported Changes
Database Tier
Patches, Schema, Parameters,
RAC nodes, Interconnect,
OS Platforms, OS Upgrades,
CPU, Memory
Storage Tier
SAN Upgrades,
ASM, more disk, etc
Supported
Database Upgrades (9i, 10g  11g),
Current Testing Methodology
(Days)
TimeEffort
Taken
(Days)
80
60
40
20
LoadRunner
DB Replay
0
Install
&Setup
Setup
Install &
Understand
Understand
Application
Application
Usage
Usage
Identify
IdentifyKey
Key
Transactions
Transactions
Generate
Generate
Workload
Workload
Total Testing Time
DB Replay: 2 weeks
LoadRunner: 30 weeks
Run
RunTest
Test
Real Application Testing Demo
Production (wgn1)
Dell OPTIPLEX 755
Intel(R) Dual Core(TM)2 Duo
CPU @ 2.33GHz
4 GB RAM
Oracle Enterprise Linux 5.1
Oracle 11.1.0.6
Standby (wgn1sby)
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
Data Guard
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
DGMGRL> convert database
'sid' to snapshot standby;
Data Guard
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
Data Guard
Capture
SQL
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
Data Guard
Prepare
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
Data Guard
Replay
SQL
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
Data Guard
Report & Analyze
Real Application Testing Demo
Production (wgn1)
Standby (wgn1sby)
DGMGRL> convert database
'sid' to physical standby;
Data Guard
Value Proposition
Oracle
DB Replay & SPA
Capture with low
overhead
Test with
production context
Transfer
resolutions to
production
3rd Party
SQL Testing Tools
Yes
Yes
No
require explicit polling &
external storage
No
not integrated with
optimizer
Yes
Integrated with AWR, ASH,
SQL Tuning Advisor & SQL
Plan Management
Technology that can only be built by Oracle
No
Value Proposition
Weeks
From:
To:
Months of development
Days of development
Artificial workloads
Production workloads
Partial workflows
Complete workflows
Manual intensive
Automated
Complete Stack
Database Tier
High risk
Low risk
Days
Resources
• Oracle.com:
http://www.oracle.com/database/real-application-testing.html
• Oracle Real Application Testing on OTN:
http://www.oracle.com/technology/products/manageability/databas
e/index.html
• Oracle Real Application Testing white papers:
http://www.oracle.com/technology/products/database/oracle11g/pdf
/real-application-testing-whitepaper.pdf
http://www.oracle.com/technology/products/manageability/databas
e/pdf/ow07/db_replay_white_paper_ow07_1.pdf
http://www.oracle.com/technology/products/manageability/databas
e/pdf/ow07/spa_white_paper_ow07.pdf