Integrating Workload Replay into
Database Change Management
Bert Scalzo
Database Domain Expert / Oracle ACE
www.toadworld.com/Bert
www.BertScalzo.com
© 2010 Quest Software, Inc. ALL RIGHTS RESERVED
Author’s Books
Coming Soon
2
Purpose
• Learn how one can rely on Benchmark
Factory and actual captured database
workload replay to more reliably ensure that
normal database changes don’t degrade the
user experience ...
• Might save your reputation, sanity and job 
• Note: Not Oracle specific, but examples here
shown using BMF and Oracle trace file replay
3
Agenda
• Common challenges when dealing with
database change
• Benefits of using database workload replay in
change management methodology
• Performing database workload replay with
Benchmark Factory
The preceding is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality described for
Quest Software’s products remain at the sole discretion of Quest Software.
4
Database Change
•
•
•
•
•
All applications & databases change over time
Often changes are small & thus seemingly safe
Sometimes changes are significant & very risky
All change introduces risk – no matter the size!
In fact studies have shown that the most “buggy”
changes to code occur when less than 5 lines
are modified (i.e. quick fixes are not very
successful)
• Database changes work much the same: when
you try “quick and simple” modifications without
much thought, bad things generally happen ….
5
The Many Sources of Change
•
•
•
•
•
•
•
•
•
•
Hardware upgrades, virtual resource re-allocation, or virtual machine relocation
Software upgrades (e.g. new operating system or update to virtual machine
software)
Operating system and/or virtual machine setup, configuration and/or initialization
parameters
Database patches and/or upgrades (e.g. 10.2.0.3->10.2.0.4 or Oracle 10gR2>11gR2)
Database setup, configuration and/or initialization parameters (e.g. SGA/PGA
memory allocations)
Database structural changes (e.g. add/drop/alter tables, columns, indexes and
views)
Storage nature changes (e.g. start using partitions, sub-partitions, index organized
tables, etc)
New database feature usage (e.g. start using 11G optional OEM “auto plan stability”
features)
Statistics and/or histogram collection frequency, granularity, parameters and/or
method (SQL command vs. PL/SQL package call)
Relative size of the database (e.g. most databases grow over time – some grow
exponentially!)
6
Managing Change
• Measure of success:
– Effective:
– Efficient:
Results AFTER = BEFORE
Performance AFTER ~= BEFORE
• How to measure effectiveness
– Unit testing
– Integration testing
• How to measure efficiency
– Stress Testing (too often a simple hardware stress
scenario to prove that system can handle a “load”)
– Workload replay: only way to assure real success
…
7
What is Workload Replay
• Run the database application to generate
transactions, and capture those transactions
for replay on the same or a different system
• Focus the application transactions captured
around that which has changed (i.e. a new
business rule or process introduces new or
modified code and database objects,
therefore run & capture workload around that
change)
• Also advisable to capture an overall workload
scenario so that bad side effects can be
found
8
Traditional DBA Approach
Usually focused on
just one or a few
SQL statements of
potential interest
Too Much & Too Little
9
Too Much & Too Little Explained
• DBA most likely did not write the application
code that’s now running slowly (i.e. efficiency)
• DBA may not fully understand the underlying
business problem and all of its “requirements“
(i.e. effectiveness)
• #1+#2 = Tune with Uncertainty (“guestimate”)
• Multiple changed dimensions or aspects of
the application and its database often rolled
out all at once – overwhelming the lone DBA
• Never enough time – Needs fixed
Yesterday!
10
Performance Pressure Points
11
Measure Cause & Effect
<<< CHANGE>>>
<<< HOLD CONSTANT >>>
12
Options for Oracle Capture & Replay
• Oracle Real Application Testing (i.e. RAT)
– Expensive (must purchase license for both source & target)
– Generally captures the “entire“ database workload (however
11gR2 offers some new & useful replay filtering options)
– Requires manually performing database backup & recovery
• Manual Trace File Capture via Toad DBA ->
Replayed by a Benchmark Factory Project
– Requires DBA to perform all the capture work manually –
and this is not always easy (i.e. can be many, many steps)
• MOUSE (cute internal name for a new
feature)
– Automates the entire capture & replay process (so BMF will
offer push button simplicity that even a caveman can do )
– Offers extensive filtering on both the capture & replay
13
RAT from Oracle New Features
Training
14
RAT Base Pricing (minus discounts,
•etc)
#CPU * #Cores * CPU Cost Factor *
Processor License Cost + software
Update/Support fee
• Must purchase for both source and target
•
Example (upgrading a Xeon DB server):
– Source: Dual E5335 Quad @ 2.0GHz
2 * 4 * .5 * $11,500 + $0.00 = $46,000
– Target:
Dual E5650 Hexa @ 2.66 GHz
2 * 6 * .5 * $11,500 + $2,530= $71,530
– OEM Diagnostic Pack (Target Only)
2 * 6 *.5 * $5,000 +$1,100 = $61,100
– OEM Tuning Pack (Target Only)
2 * 6 *.5 * $5,000 +$1,100 = $61,100
– TOTAL Cost @ Base Pricing = $239,730 !!!
– Assume Typical Discounting @ 30% = $167,811 !!!
This is in
addition to
Database’s
base cost…
15
Basic BMF Architecture
1
2
Six simple steps:
1. Manually setup to collect the trace files
2. Manually run the DB workload to capture
3. FTP trace files to Windows environment
4. Create BMF project based on trace files
5. BMF import (i.e. parse) the trace files
6. Run that project to replay the workload
3
4
5
6
16
Capture via Toad (Manual Effort)
Need both of
these so
that BMD
can replay
trace files
17
Capture via DB Trigger (Automated)
CREATE OR REPLACE TRIGGER AUTO_SET_TRACE
AFTER LOGON ON DATABASE
BEGIN
IF (USER IN ('MOE','LARRY','CURLY')) /* ONLY TRACE FOR THE 3
STOOGES */
THEN
EXECUTE IMMEDIATE 'ALTER SESSION SET tracefile_identifier =
''XXXXX''';
EXECUTE IMMEDIATE 'ALTER SESSION SET max_dump_file_size =
''10M''';
DBMS_SESSION.SESSION_TRACE_ENABLE (waits=>FALSE,
binds=>TRUE);
END IF;
EXCEPTION You can write sophisticated filters to
WHEN OTHERS
limitTHEN
the number, size and naming (i.e.
RAISE_APPLICATION_ERROR(-10001,'AUTO_SET_TRACE
ERROR');
prefix) of all trace files collected during
END;
the capture
18
Inspect Trace Files via Toad
Much like
Winzip for
Trace files
19
Create BMF Project for Replay
20
Set Some Replay Parameters
Couple of Notes:
- All the trace files need
copied from the database
server to the local BMF
Windows PC(s)
- The more and larger the
trace files the longer it will
take for BMF to parse and
load them
- BMF generally maintains
transaction ordering and
timings (but different
hardware, software, and
configurations may skew the
results as expected)
21
Run the Replay Job
So now we can replay or rerun the captured database
workload at any time by
starting this BMF project
22
I’m Poor & Lazy 
• I don’t have $$$ for RAT (and OEM packs)
• I don’t want to manually do anything (ever)
• I want a very simple, push button, read my
mind, totally automated solution that does not
break the bank ….
• New BMF feature (Q4 – code name MOUSE)
–
–
–
–
–
Simple in concept & totally automates the process
Simple Capture Wizard (creates the BMF project)
Robust filtering options for both capture & replay
Merely replay the BMF project (it handles it all)
Zero percent manual effort (fully automatic!)
23
BMF / MOUSE Architecture
Look Familiar ? 
24
New BMF Option: Capture & Replay
25
Create a Capture
26
Capture Wizard Steps
27
Capture Wizard Steps
28
Capture Wizard Steps
29
Monitoring the Capture
Manually Stop when
Done or by Schedule
30
BMF Job Built for You – Just Run It 
31
More information:
• Visit www.ToadWorld.com
– Expert blogs, how-to-videos and more!
• Paper on topic
– http://www.quest.com/documents/landing.aspx?id=10998&technolo
gy=8&prod=&prodfamily=&loc
• Benchmark Factory Community:
– http://benchmarkfactory.inside.quest.com/index.jspa
– Interact with BMF users and the product developers
32