Doc ID: Note:262946.1
Content Type:
Subject:Performance Issues After Increasing Creation Date:
Workload
Type:
Last Revision
TROUBLESHOOTING
Status: PUBLISHED
Date:
PURPOSE
-------
TEXT/PLAIN
09-FEB2004
18-MAY2004
To help identify possible bottlenecks and points of contention
when workload or number of users increases
SCOPE & APPLICATION
------------------DBA's application developer's
Performance Issues After Increasing Workload
When databases are created they need to be correctly sized by setting the
INIT.ORA parameters appropriately to provide the proper throughput to complete
day to day tasks.
One of the factors that effect performance of the database is workload.
The database initially should be tuned for a specific workload.
(# of users OR data load).
-This article does not address changes to the APPLICATION .
If the number of users or the data load increases then there is a need to
review certain INIT.ORA parameters to see if the database configuration needs
to be changed/increased.
Some potential issues may occur with the following INIT.ORA parameters if the
workload is increased:
shared_pool_size
db_cache_size (db_block_buffers in 8i)
log_buffer
pga_aggregate_target
processes.
SHARED_POOL_SIZE
================
When work load is increased , the shared_pool_size is a common parameter
that may need to be increased. Because each SQL statement resides in the
shared pool more statements in the shared pool will require more space .
Gathering STATSPACK outputs after a increased workload is crucial in
identifying the need to increase the shared pool.
There are 2 ways using STATSPACK output to identify the need to increase
the shared_pool_size:
1)Look for the "Shared Pool Statistics" section and what are the
"Begin" and "End" values.
Shared Pool Statistics
Memory Usage %:
% SQL with executions>1:
% Memory for SQL w/exec>1:
Begin
-----91.95
92.67
90.41
End
-----99.45
98.57
92.78
If the "End" value is in the high 95%-100% range ,this is a indication
that the shared pool needs to be increased (especially if the "Begin" value
is much smaller)
2)Look for "Library Cache Activity for DB" in the STATSPACK output and
look for the number of "RELOADS" and "PCT MISS"
Get
Requests
Namespace
SQL AREA
TABLE/PROCEDURE
Pct
Miss
2,000,000
1,102
20.2
6.1
Pin
Requests
Pct
Miss
Reloads
20,194,000
4,948
5
3.2
70,000
2,459
Invaliddations
7
0
From the above output there are many reloads and misses for "SQL AREA"
that indicate a need for the shared_pool_size to be increased.
***Please see the following NOTES on shared pool issues
[NOTE:146599.1]
[NOTE:62143.1]
[NOTE:105813.1]
DB_CACHE_SIZE
=============
Diagnosing and Resolving Error ORA-04031
Understanding and Tuning the Shared Pool
SCRIPT TO SUGGEST MINIMUM SHARED POOL SIZE
(DB_BLOCK_BUFFERS)
Oracle recommends that you use DB_CACHE_SIZE instead of DB_BLOCK_BUFFERS.
DB_CACHE_SIZE specifies the size of the DEFAULT buffer pool for buffers
with the primary block size (the block size defined by the DB_BLOCK_SIZE
parameter).
A STATSPACK output can give you a indication of the Buffer Cache Hit Ratio.
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00
Redo NoWait %: 100.00
Buffer Hit
%:
99.94
In-memory Sort %:
99.90
You want a ratio >80% . There may be a need for increasing DB_CACHE_SIZE
if the percentage is <90% but it is not the only measurement of performance
See Note:33883.1 STATISTIC "cache hit ratio" - Reference Note
As of Oracle 9i it is possible to use the
"Dynamic Buffer Cache Advisory feature"(DB_CACHE_ADVICE = ON in the INIT.ORA)
which enables and disables statistic gathering for predicting the behavior
of the buffer cache with different cache sizes.
Be careful as this setting will cause more resources to be used!
See the following NOTES concerning the DB_CACHE_SIZE parameter
[NOTE:148511.1] Oracle9i NF: Dynamic Buffer Cache Advisory
[NOTE:223299.1] Top Oracle 9i init.ora Parameters Affecting Performance
[NOTE:62172.1] Understanding and Tuning Buffer Cache and DBWR.
LOG_BUFFER
==========
The redo buffer does as it's name implies, provides a buffer between
LGWR and flushing of blocks from memory
If a process is making a lot of changes, the redo it
generates will be written to the memory buffer. As the buffer fills up,
the LGWR is awakened to empty the buffer.
LGWR will need some lead time, since a sufficiently large transaction
can generate redo faster than LGWR can write it to disk. This is why LGWR
wakes up when the LOG_BUFFER reaches 1/3 full.
A increase of data load / workload may necessitate a increase of the
LOG_BUFFER. In general, larger values for LOG_BUFFER reduce redo log file
I/O, particularly if transactions are long or numerous.
It has been noted previously that values larger than 5M may not make a
difference.
PGA_AGGREGATE_TARGET
====================
If you still have the SORT_AREA_SIZE, SORT_AREA_RETAINED_SIZE,
HASH_AREA_SIZE, CREATE_BITMAP_AREA_SIZE and BITMAP_MERGE_AREA_SIZE
configured in your INIT.ORA then a increase in workload may effect
these parameters.
With PGA_AGGREGATE_TARGET set the database server automatically
distributes this memory among various active queries in an
intelligent manner so as to ensure maximum
performance benefits and the most efficient utilization of memory.
The automatic SQL execution memory management feature is enabled by
setting the parameter WORKAREA_SIZE_POLICY to AUTO and by specifying a size
of PGA_AGGREGATE_TARGET in the initialization file.
Using automatic PGA memory management will help reduce the possibility
of ORA-4030 errors unless a OS limit is hit, because work area sizes
will be controlled and adjusted automatically based on the
PGA_AGGGREGATE_TARGET parameter first and then the current work load.
To determine the appropriate setting for
PGA_AGGREGATE_TARGET parameter
Refer to [NOTE:223730.1] Automatic PGA Memory Management in 9i
[NOTE:223299.1] Top Oracle 9i init.ora Parameters Affecting
Performance
PROCESSES
=========
When the number of users is increased there is a possibility of running
out of SWAP SPACE, "insufficient file descriptor" errors in the ALERT.LOG
and ORA-4030 errors
If the PROCESSES parameter has been increased then you should also check the
corresponding OS Kernel parameters that may also need to be increased.
[NOTE:169706.1] Oracle RDBMS on AIX, HP-UX, Solaris, Tru64, Linux:
Versions, Sizes, Requirements Quick Reference
[NOTE:188149.1] How to Display and Change UNIX Process Resource Limits
Heavier workloads may incur performance issues relating to LATCH or LOCK
and resource contention:
-Database
-Database
-Database
-ORA-4030
Waits for ENQUEUE
Waits on CACHE BUFFERS CHAINS
Waits on LIBRARY CACHE PIN - LIBRARY CACHE LOAD LOCK
and CPU usage
1) Waits for ENQUEUE (LOCK)
For some databases heavier workloads mean a higher rate of
concurrency(multiple sessions issuing DML on the same table)
The side effects of a higher rate of concurrency can mean more contention
for ENQUEUES or locks.
Most common ENQUEUES:
TX
Transaction Lock
See Note:62354.1 TX Transaction locks - Example wait scenarios
TM
DML enqueue
See [NOTE:38373.1] Example TM locks During Referential Integrity
Enforcement
[NOTE:33453.1] TM locks and Foreign Key Constraints
ST
Space management enqueue
See [NOTE:33567.1] ORA-1575 Timeout waiting for Space Management
Enqueue
For diagnosing ENQUEUE issues in general
See [NOTE:102925.1] Tracing sessions: waiting on an enqueue
CACHE BUFFERS CHAINS
====================
As workload increases the CACHE BUFFERS CHAINS latch could be a point
of contention. Because of this queries may slow down leading to slower
response times.
This latch is needed when user processes try to scan the SGA for
database cache buffers. If the same block is needed by other multiple
sessions then they need to obtain the CACHE BUFFERS CHAINS latch to walk
the same chain to access the same block.
We call this a "hot block" as a result because of the
concurrency leading to contention for the CACHE BUFFERS CHAINS latch.
Once the object/table is found you can reduce the number of blocks requested
on the particular object/table by
-Redesigning the application or by spreading the hits in the buffer
cache over different hash chains.
You can achieve this by implementing PARTITIONING and storing segments of
the same table/object in different files by preallocating extents
across multiple datafiles.
-Create a view to select from instead of the base table
-Rebuild tables with a higher PCTFREE to reduce number of rows per block
This could also reduce block chaining
[NOTE:107090.1] Storage Parameters PCTUSED and PCTFREE
[NOTE:42152.1] LATCH:CACHE_BUFFERS_CHAINS
[NOTE:163424.1] How To Identify a Hot Block Within The Database Buffer Cache
LIBRARY CACHE PIN / LIBRARY CACHE LOAD LOCK
===========================================
The LIBRARY CACHE PIN|LOAD|LOCK lock is allocated when pinning database
objects in the library cache. This is held until the object is flushed or
aged out.
Contention for this lock is caused by the following:
-running processes that invalidate objects in the library cache such as
analyze table , compiling pl/sql packages, use of dynamic sql .
-undersized shared_pool_size (tune the shared pool)
-A more generalized issue with this would involve a deeper investigation
utilizing SYSTEMSTATE dumps and HANGANALYZE outputs.
-Sharing SQL statements will help as well
(increase/implement session_cached_cursors)
-Issuing ALTER, GRANT or REVOKING privileges can also invalidate or cause
contention as well
*-BUG 2968709 High "library cache pin" waits possible
FIXED in 9.2.0.3 and 9.2.0.4 with a one-off patch
FIXED in 9.2.0.5 patchset
If a table is heavily used then INVALIDATIONS/RELOADS for this can be
found by issuing the following SQL :
column owner format a20
column name format a20
select owner, name, type, loads
from v$db_object_cache where name='%TABLE_NAME%';
[NOTE:115656.1] WAIT SCENARIOS REGARDING LIBRARY CACHE PIN AND LIBRARY
CACHE LOAD LOCK
[NOTE:34579.1] WAITEVENT: "library cache pin" Reference Note
3) ORA-4030 errors and higher CPU usage
As workload increases so does the memory and space requirements .
Remember that the database memory and I/O depend on the available memory,
cpu, swap space and disk space on the OS, so the OS sizing and
configuration is a major consideration .
-See above under PROCEESES parameter for reference Notes
[NOTE:164768.1] Diagnosing High CPU Utilization
[NOTE:148466.1] Monitor Oracle Resource Consumption in UNIX
FYI:
====
-Make sure that you have historical STATSPACK outputs to compare to
-Make sure you have explain plans and TKPROF outputs to see historical optimizer
paths for critical queries.
-If the application makes use of DELETES then look at TRUNCATING instead
as waits for DB_FILE_SEQUENTIAL_READ may result.
RELATED DOCUMENTS
----------------N/A