Guenadi Nedkov Jilevski
E-mail: gjilevski@gmail.com
BGOUG : Bulgaria April 2010









Top new features in Oracle 11g R1 & 11g R2
Monitoring & Performance Tools
Autotrace, SQL trace, TKPROF, DBMS_STATS,
DBMS_SQLTUNE
Oradebug
Hidden Oracle 11g R2 parameters
Useful V$ and X$ tables for Oracle 11g R2
Powerful system tools: dbx, strace, orakill
ORA-0600 Tips and tricks
KFOD for ASM Monitoring










ADR for monitoring
Data Recovery Advisor with RMAN for Oracle Backup and
Recovery with Oracle 11g
Real Application Testing (RAT) features with 11g
New partitioning features for Oracle 11g Data partitioning such as
(automatic interval, partitioning by reference, composite rangerange, composite list-range, composite list-list, composite listhash, virtual column) and partition advisor
Enhancements for Oracle 11g performance tuning (SQLPA,HM,
SQL plan management, Server/Client result cache, Enhanced
statistics, SQL tuning automation, PL/SQL native compilation
etc...)
Invisible indexes new feature for Oracle 11g
In-Memory Parallel Execution in 11g R2
Auto Degree of Parallelism (DOP) and Queuing in 11g R2
The DBMS_PARALLEL_EXECUTE package in 11g R2
Performance improvement of On-Commit Fast Refresh in 11g R2





ADR is the new Automatic Diagnostic monitor
with 11g release. Many new monitoring and
reporting features.
Completely new paradigm for Oracle DBA
monitoring.
Available in both command line interface (adrci) or
from Oracle 11g Grid/Database control.
Replaces alert.log, trace files text based monitoring
in releases prior to 11g.
Alert.log still available but recommended you
learn ADR.

The new Oracle 11g Automatic Diagnostic
Repository (ADR) has the following new
features:
One stop shop for monitoring and reporting on
Oracle 11g
 New feature for incident reporting and packaging
(IPS)
 Available with Oracle 11g Database/Grid Control
GUI as well as from command line utility (adrci)
 Many useful reporting and monitoring tools.
 Provide access to all of the Oracle 11g trace files for
networking, alert.log etc.



ADR stores the main configuration details in log.xml file
Default ADR directory set Oracle 11g initialization diagnostic_dest
parameter
SQL> show parameter diagnostic_dest
NAME
TYPE
VALUE
------------------------------------ ----------- -----------------------------diagnostic_dest
string /u01/app/oracle
SQL>
SQL> SELECT * FROM V$DIAG_INFO;
INST_ID NAME
VALUE
---------- ---------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------1 Diag Enabled
TRUE
1 ADR Base
/u01/app/oracle
1 ADR Home
/u01/app/oracle/diag/rdbms/racdb/RACDB_1
1 Diag Trace
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace
1 Diag Alert
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/alert
1 Diag Incident
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/incident
1 Diag Cdump
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/cdump
1 Health Monitor
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/hm
1 Default Trace File
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_6723.trc
1 Active Problem Count
0
1 Active Incident Count
0
11 rows selected.


ADR home directory located under default:
<ADR_BASE>/diag/rdbms/<db_name>/<instance_id>
[oracle@raclinux1 RACDB_1]$ pwd
/u01/app/oracle/diag/rdbms/racdb/RACDB_1
[oracle@raclinux1 RACDB_1]$ ls
alert cdump hm incident incpkg ir lck metadata stage sweep trace
[oracle@raclinux1 RACDB_1]$
[oracle@raclinux1 RACDB_1]$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Mon Jan 25 15:51:01 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
ADR base = "/u01/app/oracle"
adrci>


Command line utility (adrci)
Help command with ADR: help and extended help
adrci> help
HELP [topic]
Available Topics:
CREATE REPORT
ECHO
EXIT
HELP
HOST
IPS
PURGE
RUN
SET BASE
SET BROWSER
SET CONTROL
SET ECHO
SET EDITOR
SET HOMES | HOME | HOMEPATH
SET TERMOUT
SHOW ALERT
SHOW BASE
SHOW CONTROL
SHOW HM_RUN
SHOW HOMES | HOME | HOMEPATH
SHOW INCDIR
SHOW INCIDENT
SHOW PROBLEM
SHOW REPORT
SHOW TRACEFILE
SPOOL
There are other commands intended to be used directly by Oracle, type
"HELP EXTENDED" to see the list
adrci>
[oracle@raclinux1 ~]$ adrci
adrci> help extended
HELP [topic]
Available Topics:
BEGIN BACKUP
CD
CREATE STAGING XMLSCHEMA
CREATE VIEW
DDE
DEFINE
DELETE
DESCRIBE
DROP VIEW
END BACKUP
INSERT
LIST DEFINE
MERGE ALERT
MERGE FILE
MIGRATE
QUERY
REPAIR
SELECT
SET COLUMN
SHOW CATALOG
SHOW DUMP
SHOW SECTION
SHOW TRACE
SHOW TRACEMAP
SWEEP
UNDEFINE
UPDATE
VIEW
adrci>

Using ADR to view alert.log
Use the following command with ADR: show alert

[oracle@raclinux1 ~]$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Fri Jan 29 18:10:28 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
ADR base = "/u01/app/oracle"
adrci> show alert
Choose the alert log from the following homes to view:
1: diag/asm/+asm/+ASM1
2: diag/tnslsnr/raclinux1/listener_scan1
3: diag/tnslsnr/raclinux1/listener
4: diag/rdbms/rac11g/RAC11G
5: diag/rdbms/rac11gr1/RAC11gR1
6: diag/rdbms/rac11gr1/RAC11GR1
7: diag/rdbms/racdb/RACDB_1
Q: to quit
Please select option: 7













































2010-01-24 18:18:32.252000 +02:00
Starting ORACLE instance (normal)
2010-01-24 18:18:33.725000 +02:00
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.2.0 configured from GPnP Profile for use as a cluster interconnect
Interface type 1 eth0 192.168.2.0 configured from GPnP Profile for use as a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
2010-01-24 18:19:07.433000 +02:00
NOTE: Volume support enabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining, Oracle Database Vault and Real Application Testing options.
Using parameter settings in client-side pfile /u01/app/oracle/admin/RACDB/pfile/init.ora on machine raclinux1.gj.com
System parameters with non-default values:
processes
= 150
memory_target
= 2336M
control_files
= "+DATA/racdb/control01.ctl"
control_files
= "+DATA/racdb/control02.ctl"
db_block_size
= 8192
compatible
= "11.2.0.0.0"
log_archive_format
= "%t_%s_%r.dbf"
db_recovery_file_dest = "+FLASH"
db_recovery_file_dest_size= 8000M
remote_login_passwordfile= "EXCLUSIVE"
db_domain
= ""
dispatchers
= "(PROTOCOL=TCP) (SERVICE=RACDBXDB)"
remote_listener
= "raclinu-cluster:1521"
audit_file_dest
= "/u01/app/oracle/admin/RACDB/adump"
audit_trail
= "DB"
db_name
= "RACDB"
open_cursors
= 300
diagnostic_dest
= "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
10.10.2.11
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2

Use the show tracefile command for ADR with Oracle 11g
adrci> show tracefile
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4592.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_10352.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_11557.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_5887.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_2678.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_22084.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_16417.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ping_30558.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25283.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4914.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25584.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_qmnc_7024.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_15058.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25409.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_18403.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23833.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23604.trc
diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_30828.trc
adrci>




ADR uses incidents for critical errors with
Oracle 11g R2
These incidents can be packaged by ADR and
sent to Oracle support for analysis
Useful new feature with Oracle 11g for
problem resolution and reporting
Perform many of the tasks used by the RDA
(Remote Diagnostic Assistant) for releases prior
to Oracle 11g.

Critical error appear as incidents. You can use show
incident command for ADR with Oracle 11g
adrci> set homepath diag/rdbms/racdb/RACDB_1
adrci> show incident
ADR Home = /u01/app/oracle/diag/rdbms/racdb/RACDB_1:
*************************************************************************
INCIDENT_ID PROBLEM_KEY CREATE_TIME
---------------------------------------------------------------------------------------------------------------------11009 ORA 600 [kssadpm: null parent] 2010-01-15 19:08:30.464549 -04:00
1 rows fetched

Detail view for incidents with ADR and Oracle 11g
adrci> show incident –mode detail –p “incident_id=11009”
ADR Home = /u01/app/oracle/diag/rdbms/ora11g/ORA11G:
*************************************************************************
**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
INCIDENT_ID 11009
STATUS ready
CREATE_TIME 2010-01-15 19:08:30.464549 -04:00
PROBLEM_ID 1
ADR provide the following commands for
managing incidents:
 IPS CREATE PACKAGE for new incident with
Oracle 11g
 IPS ADD INCIDENT <insident_id> package
<package_id>
 IPS GENERATE PACKAGE <package_id> in
<directory>
This procedure creates a ZIP file with the trace
diagnostics for Oracle support to analyze for
critical errors.
 Use the IPS SHOW CONFIGURATION command
to check configuration for ADR incidents.

First we need to create a new package for the incident:
adrci> ips create package
Created package 1 without any contents, correlation level typical
Then we need to add the incident to the new package for ADR:
adrci> ips add incident 11009 package 1;
Added incident 11009 to package 1
Finally, we need to generate the package with data:
adrci> ips generate package 1 in /u01/app/oracle/adrci/support
Generated package 1 in file
/u01/app/oracle/adrci/support/IPSPKG_20080817214731_COM_1.zip, mode complete
The new file is generated as a zip file in the directory for support
Every DBA should use these tools for tuning
Oracle!
Autotrace
 SQL trace
 TKPROF
 DBMS_STATS
 DBMS_SQLTUNE package
 ORADEBUG



Autotrace useful quick and dirty tool for
tracing with Oracle 11g
Provides many tracing and tuning options:
Explain plan
 Costs associated with SQL queries
 Tuning statistics
Used with SQL*Plus interface
Excellent starting point for tuning SQL queries.

Use the set autotrace on command in SQL*Plus
to enable Autotrace and set autotrace off to
disable autotrace tool.
 Syntax for autotrace:
SQL> set autotrace
Usage: SET AUTOT[RACE] {OFF | ON |
TRACE[ONLY]} [EXP[LAIN]] [STAT[ISTICS]]
SQL>

SQL> set autotrace on
SQL> select prod_name, prod_list_price from
sh.products where prod_list_price > 7.99 and
prod_list_price < 69.99;
52 rows selected.
Execution Plan
---------------------------------------------------------Plan hash value: 1954719464
-----------------------------------------------------------------------------| Id | Operation
| Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------| 0 | SELECT STATEMENT |
| 3 | 93 | 3 (0)| 00:00:01 |
|* 1 | TABLE ACCESS FULL| PRODUCTS | 3 | 93 | 3 (0)| 00:00:01 |
-----------------------------------------------------------------------------Predicate Information (identified by operation id):
--------------------------------------------------1 - filter("PROD_LIST_PRICE"<69.99 AND "PROD_LIST_PRICE">7.99)
Statistics
---------------------------------------------------------7880 recursive calls
0 db block gets
1769 consistent gets
63 physical reads
0 redo size
2555 bytes sent via SQL*Net to client
452 bytes received via SQL*Net from client
5 SQL*Net roundtrips to/from client
88 sorts (memory)
0 sorts (disk)
52 rows processed
SQL>




SQL Trace is powerful tool for the serious
Oracle Performance expert
Event based: 10046 (SQL Trace), 10053 ( Cost
based Optimizer – CBO)
Can be set at instance or session level
Use DBMS_SUPPORT, DBMS_SYSTEM,
DBMS_SESSION, ORADEBUG, or SQL*PLUS
commands to enable tracing activities with
Oracle 11g R2.
SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => TRUE);
SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => FALSE);
SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>TRUE);
SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>FALSE);
SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>8, nm=>' ');
SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>0, nm=>' ');
-- All versions, requires DBMS_SUPPORT package to be loaded.
SQL> EXEC DBMS_SUPPORT.start_trace(waits=>TRUE, binds=>FALSE);
SQL> EXEC DBMS_SUPPORT.stop_trace;
SQL> EXEC DBMS_SUPPORT.start_trace_in_session(sid=>123, serial=>1234, waits=>TRUE,
binds=>FALSE);
SQL> EXEC DBMS_SUPPORT.stop_trace_in_session(sid=>123, serial=>1234);
The DBMS_SUPPORT package is not present by default, but can be loaded as the SYS user by executing the
@$ORACLE_HOME/rdbms/admin/dbmssupp.sql script.
For methods that require tracing levels the following are valid values:

0 - No trace. Like switching sql_trace off.

2 - The equivalent of regular sql_trace.

4 - The same as 2, but with the addition of bind variable values.

8 - The same as 2, but with the addition of wait events.

12 - The same as 2, but with both bind variable values and wait events.


Oracle Metalink Note 199081.1 provides good
overview for SQL trace
Using SQL*PLUS commands for SQL trace at
session/instance levels:
alter system/session set sql_trace=true;
alter system/session set timed_statistcs=true;
Optional for SQL timing tracing details
ALTER SYSTEM/SESSION SET EVENTS’10046 trace name context
forever, level <level>’;

Output SQL trace files to user_dump_dest as .trc
file
SHOW PARAMETER USER_DUMP_DEST




Should use SYS enabled account to run the SQL trace.
Need to have explain plan enabled (i.e.: utlxplan.sql)
Enable parameters for SQL tracing: timed_statistics,
sql_trace
Recommended use ALTER SESSION as ALTER SYSTEM
can place load on Oracle 11g system and impact
performance
SQL> alter session set timed_statistics=true;
Session altered.
SQL> alter session set sql_trace=true;
Session altered.
SQL> show parameter user_dump_dest
NAME
TYPE
VALUE
----------------------------------------------------------------------------user_dump_dest
string
/u01/app/oracle/diag/rdbms/ora
11g/ORA11G/trace

Use the ALTER SESSION SET TRACEFILE_IDENTIFIER command to
name trace file for easy identification
SQL> alter session set tracefile_identifire=‘products_sqltune’;
Session altered
SQL> show parameter tracefile_identifier
content of sample SQL Trace file
=====================
PARSING IN CURSOR #2 len=109 dep=0 uid=88 oct=3 lid=88 tim=1264443780269154 hv=1524029202 ad='8e65be74'
sqlid='gkc6r0xdddpsk'
select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99
END OF STMT
PARSE #2:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269152
EXEC #2:c=0,e=24,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269240
FETCH #2:c=0,e=45,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=1954719464,tim=1264443780269346
FETCH #2:c=0,e=19,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269523
FETCH #2:c=0,e=26,p=0,cr=2,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269889
FETCH #2:c=0,e=16,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780270211
FETCH #2:c=0,e=14,p=0,cr=1,cu=0,mis=0,r=6,dep=0,og=1,plh=1954719464,tim=1264443780270463
*** 2010-01-25 20:23:01.766
CLOSE #2:c=0,e=13,dep=0,type=3,tim=1264443781766794
=====================


TRACEFILE_IDENTIFIER - Another useful parameter
Many a times you need to enable tracing at a session level. And when you do that, you have to make a little bit
of effort in digging out your session's trace file. Why? because the trace file is created in the user_dump_dest
directory and there would be so many other trace files, and all of them would have similar naming convention
"SID_ora_nnnn.trc". However with the help of the parameter "TRACEFILE_IDENTIFIER", you can easily
distinguish your trace file. Very useful specifically when you want to use trace analyzer etc.
Here is how?
1. Set the tracefile identifier as you want, this will be appended to trace file name.
alter session set tracefile_identifier=‘filename_sql_trace';
2. Enable your session level tracing
alter session set sql_trace=true;
alter session set events '10046 trace name context forever, level 12' ;
3. Execute your queries/statements
4. Turn-off tracing
alter session set events '10046 trace name context off';
alter session set sql_trace=false;
5. Check the user_dump_dest folder and you will find a file name "SID_ora_nnnn_filname_sql_trace.trc
See now it's so easy to identify. Having said that you can still find out the trace file name without using
TRACEFILE_IDENTIFIER parameter using the following SQL but when Oracle has provided an easier
method, why not use it?
-- sql to find out trace file name for your session.
select c.value || '/' || d.instance_name ||'_ora_' || a.spid || '.trc' trace
from v$process a, v$session b, v$parameter c, v$instance d
where a.addr = b.paddr
and b.audsid = userenv('sessionid')
and c.name = 'user_dump_dest'
/




TKPROF is a trace kernel profiler – useful for
formatting trace files for tuning purposes
Flexible options for performance report
statistics
Cleans up the raw SQL trace files into human
readable tuning reports.
Oracle Metalink Note: 41634.1 TKPROF
Simplistic overview



TKPROF takes the trace file and output report
file plus performance specific parameters
Many parameter options for using tkprof
Syntax for tkprof:
tkprof trace_file.trc output_file.rpt [waits={yes|no}]
[sort={yes|no}] [print=option] [aggregate={yes|no}]
[insert={filename}] [sys={yes|no}]
[table={schema.table}] [explain={schema/password}]
[record=filename] [width=n]

TKPROF Help :
[oracle@raclinux1 ~]$ tkprof
Usage: tkprof tracefile outputfile [explain= ] [table= ]
[print= ] [insert= ] [sys= ] [sort= ]
table=schema.tablename Use 'schema.tablename' with 'explain=' option.
explain=user/password Connect to ORACLE and issue EXPLAIN PLAN.
print=integer List only the first 'integer' SQL statements.
aggregate=yes|no
insert=filename List SQL statements and data inside INSERT statements.
sys=no
TKPROF does not list SQL statements run as user SYS.
record=filename Record non-recursive statements found in the trace file.
waits=yes|no Record summary for any wait events found in the trace file.
sort=option
Set of zero or more of the following sort options:
prscnt number of times parse was called
prscpu cpu time parsing
prsela elapsed time parsing
prsdsk number of disk reads during parse
prsqry number of buffers for consistent read during parse
prscu number of buffers for current read during parse
prsmis number of misses in library cache during parse
execnt number of execute was called
execpu cpu time spent executing
exeela elapsed time executing
exedsk number of disk reads during execute
exeqry number of buffers for consistent read during execute
execu number of buffers for current read during execute
exerow number of rows processed during execute
exemis number of library cache misses during execute
fchcnt number of times fetch was called
fchcpu cpu time spent fetching
fchela elapsed time fetching
fchdsk number of disk reads during fetch
fchqry number of buffers for consistent read during fetch
fchcu number of buffers for current read during fetch
fchrow number of rows fetched
userid userid of user that parsed the cursor
[oracle@raclinux1 ~]$

Example
[oracle@raclinux1 trace]$ tkprof
RACDB1_ora_9163_products_sqltune.trc tune_sql.rpt
waits=yes aggregate=yes sys=yes explain=sys/oracle
********************************************************************************
SQL ID: gkc6r0xdddpsk
Plan Hash: 1954719464
select prod_name, prod_list_price
from
sh.products where prod_list_price > 7.99 and prod_list_price < 69.99
call count
cpu elapsed
disk
query current
rows
------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse
3
0.00
0.02
0
0
0
0
Execute
4
0.00
0.00
0
0
0
0
Fetch
15
0.00
0.00
0
24
0
156
------- ------ -------- ---------- ---------- ---------- ---------- ---------total
22
0.00
0.03
0
24
0
156
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 88
********************************************************************************

What is the DBMS_SQLTUNE package?


Automates SQL tuning by using profiles to tune
SQL.
DBMS_SQLTUNE package is used as part of the SQL
Tune Advisors (STA) with Oracle 11g

Example
SQL> DECLARE
sql_task_name VARCHAR2(30);
tune_sqltext CLOB;
BEGIN
tune_sqltext :='select * '||'FROM sh.sales '||' where prod_id=20 and '||'
cust_id=200 ';
sql_task_name := DBMS_SQLTUNE.CREATE_TUNING_TASK(
sql_text=>tune_sqltext,
User_name=>'SH',
Scope=>'COMPREHENSIVE',
Time_limit=>60,
Task_name=>'TUNE_SH_TASK',
Description=>'New task for tuning SH query');
END;
/
2 3 4 5 6 7 8 9 10 11 12 13 14
PL/SQL procedure successfully completed.


DBMS_SQLTUNE uses the
CREATE_TUNING_TASK function as part of SQL
Tuning Advisor (STA) with 11g
Once we have created the tuning task, we need to
execute it with the
DBMS_SQLTUNE.EXECUTE_TUNING_TASK:
SQL> begin
DBMS_SQLTUNE.EXECUTE_TUNING_TASK(task_name=>'TUNE_S
H_TASK');
End;
/ 2 3 4
PL/SQL procedure successfully completed.
SQL>

To view SQL Tuning report query the DBMS_SQLTUNE.REPORT_TUNING_TASK;
SQL> select dbms_sqltune.report_tuning_task(‘TUNE_SH_TASK’) drom DUAL;
SQL> /
DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')
-------------------------------------------------------------------------------GENERAL INFORMATION SECTION
------------------------------------------------------------------------------Tuning Task Name : TUNE_SH_TASK
Tuning Task Owner : SYS
Workload Type : Single SQL Statement
Execution Count : 2
Current Execution : EXEC_86
Execution Type : TUNE SQL
Scope
: COMPREHENSIVE
Time Limit(seconds): 60
Completion Status : COMPLETED
DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')
-------------------------------------------------------------------------------Started at
: 01/26/2010 16:34:26
Completed at
: 01/26/2010 16:34:27
------------------------------------------------------------------------------Schema Name: SH
SQL ID : 8z3nbpj9f0nsa
SQL Text : select * FROM sh.sales where prod_id=20 and cust_id=200
------------------------------------------------------------------------------FINDINGS SECTION (1 finding)
------------------------------------------------------------------------------DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')
-------------------------------------------------------------------------------1- Index Finding (see explain plans section below)
-------------------------------------------------The execution plan of this statement can be improved by creating one or more
indices.
Recommendation (estimated benefit: 93.13%)
------------------------------------------ Consider running the Access Advisor to improve the physical schema design
or creating the recommended index. If you choose to create the
recommended index, consider dropping the index "SH"."SALES_CUST_BIX”


What is Oradebug?
Why should you use it?
Performance Tuning
 Hang Analysis
 Dump Analysis for Oracle 11g Memory
 Suspend and Resume Oracle processes





Oradebug introduced in version 7.X release of
Oracle
Poorly documented utility
Recommend caution when using oradebug
Syntax for Oradebug

Oradebug has MANY options!!!
Use the oradebug help command to list all the main options available:

SQL> oradebug help
HELP
[command]
Describe one or all commands
SETMYPID
Debug current process
SETOSPID
<ospid>
Set OS pid of process to debug
SETORAPID <orapid> ['force']
Set Oracle pid of process to debug
SETORAPNAME <orapname>
Set Oracle process name to debug
SHORT_STACK
Get abridged OS stack
CURRENT_SQL
Get current SQL
DUMP
<dump_name> <lvl> [addr] Invoke named dump
DUMPSGA
[bytes]
Dump fixed SGA
DUMPLIST
Print a list of available dumps
EVENT
<text>
Set trace event in process
SESSION_EVENT <text>
Set trace event in session
DUMPVAR
<p|s|uga> <name> [level] Print/dump a fixed PGA/SGA/UGA variable
DUMPTYPE
<address> <type> <count> Print/dump an address with type info
SETVAR
<p|s|uga> <name> <value> Modify a fixed PGA/SGA/UGA variable
PEEK
<addr> <len> [level] Print/Dump memory
POKE
<addr> <len> <value> Modify memory
WAKEUP
<orapid>
Wake up Oracle process
SUSPEND
Suspend execution
RESUME
Resume execution
FLUSH
Flush pending writes to trace file
CLOSE_TRACE
Close trace file
TRACEFILE_NAME
Get name of trace file
LKDEBUG
Invoke global enqueue service debugger
NSDBX
Invoke CGS name-service debugger
-G
<Inst-List | def | all> Parallel oradebug command prefix
-R
<Inst-List | def | all> Parallel oradebug prefix (return output
SETINST
<instance# .. | all> Set instance list in double quotes
SGATOFILE
<SGA dump dir>
Dump SGA to file; dirname in double quotes
DMPCOWSGA <SGA dump dir> Dump & map SGA as COW; dirname in double quotes
MAPCOWSGA <SGA dump dir>
Map SGA as COW; dirname in double quotes
HANGANALYZE [level] [syslevel]
Analyze system hang
FFBEGIN
Flash Freeze the Instance
FFDEREGISTER
FF deregister instance from cluster
FFTERMINST
Call exit and terminate instance
FFRESUMEINST
Resume the flash frozen instance
FFSTATUS
Flash freeze status of instance
SKDSTTPCS <ifname> <ofname>
Helps translate PCs to names
WATCH
<address> <len> <self|exist|all|target> Watch a region of memory
DELETE
<local|global|target> watchpoint <id> Delete a watchpoint
SHOW
<local|global|target> watchpoints
Show watchpoints
DIRECT_ACCESS <set/enable/disable command | select query> Fixed table access
CORE
Dump core without crashing process
IPC
Dump ipc information
UNLIMIT
Unlimit the size of the trace file
PROCSTAT
Dump process statistics
CALL
[-t count] <func> [arg1]...[argn] Invoke function with arguments

First we need to obtain SPID from v$SESSION and
V$PROCESS
SQL> select username, sid, serial#, paddr from v$session where username='SH';
USERNAME
SID SERIAL# PADDR
------------------------------ ---------- ---------- -------SH
153
3370 994694F4
SQL> select addr, pid , spid from v$process where addr='994694F4';
ADDR
PID SPID
-------- ---------- -----------------------994694F4
35 20933
SQL>

Then we use the SPID in oradebug to trace
session:
SQL> oradebug setospid 20933
Oracle pid: 35, Unix process pid: 20933, image:
oracle@raclinux1.gj.com (TNS V1-V3)
SQL>

We then set the unlimit parameter for the trace
file with Oradebug:
SQL> oradebug unlimit
Statement processed.
SQL>

Now we are ready to start the trace for our
session with Oradebug
SQL> oradebug event 10046 trace name context forever, level 4
Statement processed.
SQL>





Oradebug uses Events for tracing options
Some common trace events:
10053 – Cost Based Optimizer (CBO) trace
10046 – SQL Trace
To start tracing with Oradebug
SQL> oradebug event 10046 trace name context forever, level 4
Statement processed.
SQL>

To stop and turn off tracing for Oradebug:
SQL> oradebug event 10046 trace name context off
Statement processed.
Trace file output based on user_dump_dest parameter
 Naming convention for the trace file
sid_ora_pid_traceid.trc
 Easy way find trace file is to set trace_identifier parameter:
alter session set tracefile_identifier=‘mytrace’;
 Useful query to find trace file

Select
u_dump.value||’/’||db_name.value||’_ora_’||v$PROCESS.spid||n
vl2(v$process.traceid,’_’||v$process.traceid,
Null )||’.trc’ “Trace File” from v$parameter u_dump cross join
v$parameter db_name cross join v$process join v$session on
v$process.addr = ‘usr_dump_dest’ and db_name.name = ‘db_name’ and
v$session.audsid=sys_context(‘userenv’,’sessionid’);


Locate the trace file for oradebug
Format it with tkprof
cp RACDB_1_ora_20933_test_sh_query.trc trc.trc
[oracle@raclinux1 trace]$ tkprof trc.trc rpt.rpt aggregate=yes waits=yes sys=no explain=sh/sh
TKPROF: Release 11.2.0.1.0 - Development on Tue Jan 26 18:29:25 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
[oracle@raclinux1 trace]$

Open the trace file with text editor
$vi rpt.rpt
SQL ID: 3j4064f628jmk
Plan Hash: 1954719464
select prod_name, prod_list_price
from
sh.products where prod_list_price > 7.99 and prod_list_price < 69.99
call count
cpu elapsed
disk query current
rows
------- ------ -------- ---------- ---------- ---------- ---------- ---------Parse
6 0.00
0.00
0
0
0
0
Execute 6 0.00
0.00
0
0
0
0
Fetch
30 0.00
0.00
0
48
0
312
------- ------ -------- ---------- ---------- ---------- ---------- ---------total
42 0.00
0.00
0
48
0
312
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 88 (SH)
Rows Row Source Operation
------- --------------------------------------------------52 TABLE ACCESS FULL PRODUCTS (cr=8 pr=0 pw=0 time=0 us cost=3 size=1820 c
ard=52)
Rows Execution Plan
------- --------------------------------------------------0 SELECT STATEMENT MODE: ALL_ROWS
52 TABLE ACCESS MODE: ANALYZED (FULL) OF 'PRODUCTS' (TABLE)



Oradebug is excellent tool for solving database
hang issue.
Use the HANGANALYZE option with oradebug
Syntax:
oradebug hanganalyze <level>

For Oracle RAC environment the syntax is:
oradebug –g all hanganalyze <level>


Level indicates amount of details to collect for a
hang.
Refer to Metalink Note: 175006.1 and
Note:215858.1

The debug levels defined for HANGANALYZE
with oradebug are as follows Note 215858.1:
Level 10: Dump all process ( IGN state)
 Level5: Level 4 + Dump all processes involved with
chain (NLEAF)
 Level 4: Level 3 + Dump leaf nodes (blocker) in wait
chains (LEAF, LEAF_NW, ING_DMP state)
 Level 3: Level 2 + Dump only processes considered
in hang condition (IN_HANG state)
 Levels 1-2: Generate ONLY HANGANALYZE
output with no process dump


Set environment for hang analysis:
SQL> oradebug setmypid
Statement process.
SQL> oradebug unlimit
Statement processed.

Perform Hang Analysis:
SQL> oradebug -g all hanganalyze 10
Hang Analysis Dumps in
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB
_1_diag_6112.trc
SQL>exit

Review output hang analysis from trace file:
Trace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_diag_6112.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining, Oracle Database Vault and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name: Linux
Node name:
raclinux1.gj.com
Release:
2.6.18-164.el5PAE
Version:
#1 SMP Thu Sep 3 02:28:20 EDT 2009
Machine:
i686
Instance name: RACDB_1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 5
Unix process pid: 6112, image: oracle@raclinux1.gj.com (DIAG)
*** 2010-01-26 13:02:55.529
*** SESSION ID:(127.1) 2010-01-26 13:02:55.529
*** CLIENT ID:() 2010-01-26 13:02:55.529
*** SERVICE NAME:() 2010-01-26 13:02:55.529
*** MODULE NAME:() 2010-01-26 13:02:55.529
*** ACTION NAME:() 2010-01-26 13:02:55.529
Node id: 0
List of nodes: 0,
*** 2010-01-26 13:02:55.565
Reconfiguration starts [incarn=0]
*** 2010-01-26 13:02:55.566
I'm the master node
*** 2010-01-26 13:02:55.566
Reconfiguration completes [incarn=1]
Group reconfiguration cleanup
Use oradebug to dump contents of UGA and PGA memory
Useful for debugging memory leak issues

Dump Analysis for Oracle 11g Memory

Describe events
To start diagnosing a problem with the process size, such as a suspected leak,
a heapdump of the offending process is required (Also helps to check with pmap –x <pid> for memory leaks):


$sqlplus
SQL> connect '/ as sysdba'
SQL> oradebug setospid <pid>
SQL> oradebug unlimit
SQL> oradebug dump heapdump 5 --this dumps PGA and UGA heaps
The above information should be provided to Oracle Support Services.
Oracle 8i / 9i / 10g
SQL> oradebug dump events <level>
Level 1 - Dump session group's event settings
Level 2 - Dump process group's event settings
Level 4 - Dump system group's event settings (Ie the instance wide events)
Oracle 10g / 11g and above
SQL> oradebug eventdump <level>
SESSION - Dump session group's event settings
PROCESS - Dump process group's event settings
SYSTEM - Dump system group's event settings (Ie the instance wide events)
SQL> oradebug setmypid
Statement processed.
SQL> oradebug eventdump session
Statement processed.
SQL> oradebug eventdump system
Statement processed.
SQL> alter system set events '942 trace name errorstack level 3';
System altered.
SQL> oradebug eventdump system
942 trace name errorstack level 3
SQL>
SQL> oradebug eventdump session
942 trace name errorstack level 3

Use the oradebug dumpsga command:
SQL> oradebug setmypid
Statement processed.
SQL> oradebug unlimit
Statement processed.
SQL> oradebug dumpsga
Statement processed.
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_2
5351.trc
SQL> exit

Contents of SGA dump in oradebug trace file
Trace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25351.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management,
Oracle Label Security,
OLAP, Data Mining, Oracle Database Vault and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name: Linux
Node name: raclinux1.gj.com
Release:
2.6.18-164.el5PAE
*** SESSION ID:(154.6347) 2010-01-26 19:14:21.438
*** CLIENT ID:() 2010-01-26 19:14:21.438
*** SERVICE NAME:(SYS$USERS) 2010-01-26 19:14:21.438
*** MODULE NAME:(sqlplus@raclinux1.gj.com (TNS V1-V3)) 2010-01-26 19:14:21.438
*** ACTION NAME:() 2010-01-26 19:14:21.438
Processing Oradebug command 'setmypid'
*** 2010-01-26 19:14:21.438
Oradebug command 'setmypid' console output: <none>
*** 2010-01-26 19:14:30.092
Processing Oradebug command 'unlimit'
*** 2010-01-26 19:14:30.092
Oradebug command 'unlimit' console output: <none>
*** 2010-01-26 19:15:11.671
----- Fixed Areas Dump (level=2) --------- Dump of the Fixed SGA ----ksmsgft ksmsgf_ [20000000, 20001000) = 00000000 00000000 00000000 00000000 ...
Dump of memory from 0x20000010 to 0x20001000
20000010 00000000 00000000 00000000 00000000 [................]
Repeat 254 times
kywmr * kywmrsga_ [20001000, 20001004) = 990CD020



Oradebug useful for stopping and restarting
Oracle sessions
Be careful when using oradebug in production
environments!
Syntax for suspending process with oradebug
oradebug suspend

To resume a process that has been suspended
oradebug resume

Useful when need to delay batch process for
more critical job.

Find the session information to suspend:
SQL> select s.username, s.pid, p.spid
2
From
3
V$session s, v$process p
4
Where s.paddr=p.addr
5
And
6
S.username <> ‘SYS’;
USERNAME
SID
----------------------------SH
140
SPID
---------18335

Oradebug suspend
SQL> oradebug setospid 18335
SQL> oradebug suspend
Statement processed.

Oradebug resume
SQL> oradebug resume
Statement processed.


oradebug is used to trace issues related to IPC
in interconnects and RAC
Syntax for IPC memory dump with Oracle
RAC:
SQL> oradebug ipc

Example
SQL> oradebug setmypid
Statement processed.
SQL>oradebug unlimit
Statement processed.
SQL>oradebug ipc
Information written to trace file.
SQL> oradebug trace_file
/u01/app/oracle/admin/RACDB/udump/racdb1_ora_6391.tr
c
SQL>exit

Review trace file for interconnect trace with Oradebug
for RAC:
SSKGXPT 0xcc75e9c flags SSKGXPT_READPENDING info for network 0
socket no 7 IP 10.10.10.11 UDP 2247
sflagsSSKGXPT_UP
info for network 1
socket no 0 IP 0.0.0.0 UDP 0
sflagsSSKGXPT_DOWN
active 0 actcnt1
context timestamp 0
no ports
sconnoacconoerttstate seq# sent asyncsync rtransacks


Oracle 11g has many new additional hidden
parameters!
There are more than 2050 hidden parametrs for
Oracle 11g R2!
SQL> select count(*) from x$ksppi where substr(ksppinm,1,1)='_';
COUNT(*)
---------2057
SQL>

Query to find out these hidden Oracle 11g R2
parameters:
Select ksppinm from x$ksppi where substr(ksppinm,1,1)=‘_’;


Some hidden parameters for 11g new features
Hang manager hidden parameters for Oracle 11g
R2:
_hm_analysis_ouput_disk
_hm_analysis_oradebug_node_dump_level
_hm_analysis_oradebug_sys_dump_level
_hm_verification_interval
_hm_log_incidents
Be careful using these
 First change in test and sandbox
 Best to work with Oracle support since those are
undocumented parameters!




Each release of Oracle adds many new V$ and X$
performance view tables.
More than 945 different X$ tables in Oracle 11g R2.
There are over 525 different v$ performance views
in Oracle 11g R2 .
Query to find the V$ performance views for Oracle
11g R2:
Select name, type from v$fixed_table where name like ‘V$%’;

Query to find the list of X$ tables for Oracle 11g
R2:
Select name, type from v$fixed_table where name like ‘X$%’;

Some interesting new V$ Dynamic Performance Views
V$MEMORY_DYNAMIC_COMPONENTS - displays information about the dynamic
SGA components. This view summarizes information based on all completed SGA
resize operations since instance startup. All sizes are expressed in bytes.
V$MEMORY_TARGET_ADVICE - information about how the MEMORY_TARGET
parameter should be sized based on current sizing and satisfaction metrics.
V$ALERT_TYPES – information for server alert types
V$IR_FAILURE – provides information on the failure. Note that records in this view
can have parent records within this view. (DRA)
V$IR_REPAIR – show the results of repair failure commands. (DRA)
V$IR_MANUAL_CHECKLIST – detailed information messages related to the failure.
Messages provide information on how to manually correct the problem. (DRA)
V$IR_FAILURE_SET – list of advice records associated with the failure. (DRA)
V$HM_CHECK - information about all the checks registered with Health Monitor.
Each check is uniquely identified by a name or an ID.
V$HM_CHECK_PARAM - information about the input parameters of all Health
Monitor checks.
V$HM_RUN - information about all Health Monitor checks and their status. All checks
ran.
V$HM_FINDING - information about all the findings of various Health Monitor runs.
V$HM_RECOMMENDATION - information about any recommendation from Health
check runs.
V$HM_INFO - Information about runs, finding and recommendations.
Perform a describe on each of these V$ dynamic performance views
for detail information.

These are some useful new X$ tables for Oracle
11g R2:
X$DIAG_INFO
X$IR_WORKING_FAILURE_SET – incident reporting
X$IR_WORKING_REPAIR_SET – incident reporting
X$IR_REPAIR_OPTION – incident reporting
X$IR_MANUAL_OPTION – incident reporting
X$IR_REPAIR_STEP – incident reporting
When Oracle tools are not enough, these OS tools are essential:
 dbx (Solaris and Linux)
 gdb (Linux) similar to dbx
 truss (Solaris)
 strace (Linux)
 orakill (Windows)
 STACKX (UNIX) Metalink Note: 362791.1
Useful References on Metalink for core Debug Analysis:
Oracle Metalink Note:16534.1 “debug Core Dumps / core files
on Unix”
Oracle Metalink Note: 1007808.6 “How to handle core dumps
on UNIX”




DBX is trace debugger for Solaris and also
available for free download from Sun.com as part
of Solaris Developer Studio.
Provides comprehensive debugger level
commands for Solaris and Linux OS systems calls
and Oracle 11g instance and network level internal
processes.
DBX is an advanced complex tool with complex
syntax.
Complete reference guide from SUN:
Sun Studio 12: Debugging a Program with dbx:
http:/docs.sun.com/app/docs


DBX can be used to trace Oracle 11g processes
Getting help with dbx
[root@raclinux1 bin]# ./dbx
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc
(dbx) help
dbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8'
dbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8'
command summary
Use `commands' to see a command summary consisting of one-line
descriptions of each dbx command.
Execution and Tracing
cancel
catch
clear
cont
fix
fixed
handler ignore
next
pop
replay
rerun
run
runargs
save
status
stop
trace
unintercept when
delete
intercept
restore
step
whocatches
Displaying and Naming Data
assign
call
dis
display
down
dump
examine exists
fortran_modules
frame
hide
print
undisplay unhide
up
watch
whatis
where
whereami
whereis
which
Accessing Source Files
bsearch
cd
edit
file
func
funcs
line
list
loadobjects module
modules
search
files
loadobject
pathmap pwd
Debugging Multiple Threads
lwp
lwps
sync
syncs
thread
threads

Find the PID for Oracle process
[root@raclinux1 bin]# ps -ef | grep oracleRACDB
oracle 9546 9545 0 18:26 ?
00:00:00 oracleRACDB_1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
root 15103 12453 0 20:02 pts/3 00:00:00 grep oracleRACDB
[root@raclinux1 bin]#

Now we need to get a trace for Oracle
[root@raclinux1 bin]# ./dbx /u01/app/oracle/product/11.2.0/db_1/bin/oracle 9546
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc
Reading oracle
Reading ld-linux.so.2
……………………………
Reading libnque11.so
Attached to process 9546
t@3671744 (l@9546) stopped in (unknown) at 0xdcd410
0x00dcd410: popl %ebp
(dbx)

Enable Collection Process and continue trace - The collector command
collects performance data for analysis by the Performance Analyzer. It is
valid only in native mode.
(dbx) collector enable
Back to our DBX trace session:
(dbx) cont

Use the dbx thread command to look at Oracle 11g R2 memory process stack. Use the dbx thread command. Displays
the current thread.
(dbx) thread t@3671744
t@3671744 (l@9546) stopped in (unknown) at 0xdcd410
0x00dcd410: popl %ebp
(dbx) where
current thread: t@3671744
[1] 0xdcd410(0x2010, 0x108ba596, 0x7cb763, 0x105d4874, 0xf4351a0, 0x9), at 0xdcd410
[2] __read_nocancel(0x108b1108, 0x108b1770, 0x108ae8f8, 0x108ba596, 0xbfaa3d10, 0x0), at 0x7cb763
[3] ntpfprd(0x108ae790, 0x108ba596, 0x2010, 0xbfaa3e24, 0x8), at 0xf4350d9
[4] nsbasic_brc(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41b0b7
[5] nsbrecv(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41dba6
[6] nioqrc(0x106dad04, 0x0, 0xbfaa5f0c, 0x1, 0x0, 0x1), at 0xf422429
[7] __PGOSF25_opikndf2(0x106dad04, 0x2, 0xbfaa5f0c, 0x1, 0x0), at 0xf1c88e6
[8] opitsk(0x0, 0x0), at 0x8f3b9fb
[9] opiino(0x3c, 0x4, 0xbfaa6e14), at 0x8f3f89b
[10] opiodr(0x3c, 0x4, 0xbfaa6e14, 0x0), at 0xf1cb036
[11] opidrv(0x3c, 0x4, 0xbfaa6e14), at 0x8f382f3
[12] sou2o(0xbfaa6df8, 0x3c, 0x4, 0xbfaa6e14), at 0x92ed66c
[13] opimai_real(0x2, 0xbfaa6f48, 0x108721a8, 0x0, 0x0, 0x651f14), at 0x8542cdb
[14] ssthrdmain(0x2, 0xbfaa6f48), at 0x92f1f2d
[15] main(0x2, 0xbfaa7044, 0xbfaa7050, 0x640810, 0x0, 0x1), at 0x8542c57
(dbx)
(dbx) thread -info
Thread t@3671744 (0x0) at priority 0
state: active on l@9546
base function: 0x0: 0x00000000() stack: 0x0[0]
flags: (none)
masked signals: (none)
Currently active at 0x00dcd410
(dbx)

The debug function with dbx is useful for
Oracle 11g R2 (debug lists the program being
debugged) :
(dbx) debug –f
Debugging: oracle ( process id 24976)


Truss provides ability to walk through trace at
step by step level for Oracle 11g internal
process
Syntax:
truss [-fcaeil ] [-[tvx] [!]syscall..] [-s [!] signal..] [-m [!]fault..] [[rw] [!]fd..] [-o outfile] command | -p pid
Typical options for truss:
-o output to file
-f trace child processes
-c count system calls
-p trace calls based on Unix pid

Example
solaris02$truss –cp 3316
^C
Syscall
seconds
calls
Read
Write
Times
Yield
2
2
33
226
.000
.000
.000
.000
errors


Strace is similar to truss for Linux O/S tracing
with Oracle 11g
Powerful tool with command line syntax:
usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file]
[-p pid] ... [-s strsize] [-u username] [-E var=val] ...
[command [arg ...]]
or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val]
...
[command [arg ...]]

Options available for strace
[root@raclinux1 ~]# strace
usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file]
[-p pid] ... [-s strsize] [-u username] [-E var=val] ...
[command [arg ...]]
or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ...
[command [arg ...]]
-c -- count time, calls, and errors for each syscall and report summary
-f -- follow forks, -ff -- with output into separate files
-F -- attempt to follow vforks, -h -- print help message
-i -- print instruction pointer at time of syscall
-q -- suppress messages about attaching, detaching, etc.
-r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs
-T -- print time spent in each syscall, -V -- print version
-v -- verbose mode: print unabbreviated argv, stat, termio[s], etc. args
-x -- print non-ascii strings in hex, -xx -- print all strings in hex
-a column -- alignment COLUMN for printing syscall results (default 40)
-e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]...
options: trace, abbrev, verbose, raw, signal, read, or write
-o file -- send trace output to FILE instead of stderr
-O overhead -- set overhead for tracing syscalls to OVERHEAD usecs
-p pid -- trace process with process id PID, may be repeated
-s strsize -- limit length of print strings to STRSIZE chars (default 32)
-S sortby -- sort syscall counts by: time, calls, name, nothing (default time)
-u username -- run command as username handling setuid and/or setgid
-E var=val -- put var=val in the environment for command
-E var -- remove var from the environment for command
[root@raclinux1 ~]#

Perform system call trace for Oracle 11g:
[oracle@raclinux1 ~]$ strace oracle
execve("/u01/app/oracle/product/11.2.0/db_1/bin/oracle", ["oracle"], [/* 36 vars */]) = 0
brk(0)
= 0x10792000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x8b9000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/u01/app/oracle/product/11.2.0/db_1/lib/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)
open("/u01/app/oracle/product/11.2.0/db_1/lib/libodm11.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\254\6\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=8857, ...}) = 0
mmap2(NULL, 6768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40d000
mmap2(0x40e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x40e000
close(3)
=0
open("/u01/app/oracle/product/11.2.0/db_1/lib/libcell11.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340|\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=383547, ...}) = 0
Similar to truss:
Check for system call errors for Oracle 11g with strace

[oracle@raclinux1 ~]$ strace -c oracle
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ---------------40.53 0.000308
308
1
execve
15.79 0.000120
0
338
49 open
15.13 0.000115
7
16
16 mkdir
12.37 0.000094
1
75
read
8.29 0.000063
0
132
munmap
7.89 0.000060
0
293
close
0.00 0.000000
0
20
write
0.00 0.000000
0
1
chmod
0.00 0.000000
0
3
lseek
0.00 0.000000
0
3
3 access
0.00 0.000000
0
1
dup
0.00 0.000000
0
4
times
0.00 0.000000
0
4
brk
0.00 0.000000
0
2
setrlimit
0.00 0.000000
0
5
gettimeofday
0.00 0.000000
0
5
uname
0.00 0.000000
0
9
mprotect
0.00 0.000000
0
3
_llseek
0.00 0.000000
0
6
rt_sigaction
0.00 0.000000
0
5
rt_sigprocmask
0.00 0.000000
0
5
getrlimit
0.00 0.000000
0
307
mmap2
0.00 0.000000
0
48
19 stat64
0.00 0.000000
0
7
2 lstat64
0.00 0.000000
0
26
fstat64
0.00 0.000000
0
2
getuid32
0.00 0.000000
0
1
geteuid32
0.00 0.000000
0
6
fcntl64
0.00 0.000000
0
1
gettid
0.00 0.000000
0
2
futex
0.00 0.000000
0
1
set_thread_area
0.00 0.000000
0
1
set_tid_address
0.00 0.000000
0
1
set_robust_list
0.00 0.000000
0
3
socket
0.00 0.000000
0
1
bind
0.00 0.000000
0
2
2 connect
0.00 0.000000
0
5
shmat
0.00 0.000000
0
5
shmdt
0.00 0.000000
0
12
9 shmget
------ ----------- ----------- --------- --------- ---------------100.00 0.000760
1362
100 total
[oracle@raclinux1 ~]$

Example : using strace to find system calls for Oracle DBWR process
oracle 6123 1 0 12:16 ?
00:00:01 ora_dbw0_RACDB_1
oracle 31050 8191 0 20:24 pts/2 00:00:00 grep dbw
[oracle@raclinux1 admin]$ strace -c -p 6123
Process 6123 attached - interrupt to quit
Process 6123 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ---------------57.30 0.049795
732
68
io_submit
35.52 0.030866
123
250
pread64
3.14 0.002729
0 20064
getrusage
1.55 0.001350
11
123
io_getevents
1.15 0.000999
4
235
125 semtimedop
0.34 0.000297
2
151
semctl
0.32 0.000279
0 17913
gettimeofday
0.25 0.000219
4
58
mmap2
0.25 0.000214
4
50
sendmsg
0.10 0.000087
2
57
open
0.08 0.000068
1
57
read
0.00 0.000000
0
57
close
0.00 0.000000
0
766
times
------ ----------- ----------- --------- --------- ---------------100.00 0.086903
39849
125 total
[oracle@raclinux1 admin]$




Useful tool available for Oracle 11g on
Windows called orakill
Oracle Metalink Note: 69882.1
Useful for killing runaway processes on
Oracle 11g on Windows as well as for zombie
and hung sessions
Syntax:
orakill <sid> ,<thread>
Can use QSLICE.exe to find threads ( available
on Windows NT/200X Resource kit from
Microsoft)

Query if not using QSLICE.EXE for details:
C:\Users\User>orakill
Usage: orakill sid thread
where sid = the Oracle instance to target
thread = the thread id of the thread to kill
The thread id should be retrieved from the spid column of a query such as:
select spid, osuser, s.program from
v$process p, v$session s where p.addr=s.paddr;
C:\Users\User>
QSLISE.EXE OUTPUT

Use Qslice to find
Oracle processes
Than use orakill to end
process that is hung
C:> orakill ora11g 985
Kill of thread id 985 in instance
ora11g successfully signaled.




ORA-0600: Internal errors
Call Oracle support first
Leverage Metalink ORA-0600 support matrix
Collect dumps with oradebug if possible




Internal Oracle errors generate kernel memory
dump.
Dump core file and trace files.
Useful Metalink Notes: 146580.1, 146581.1
Use the Oracle Metalink ORA-0600 lookup tool
for addressing these internal errors.
Note 153788.1 – Troubleshoot an ORA-0600 Error
using the ORA-600 Argument lookup tool
From Oracle support see Note 153788.1
Troubleshoot an ORA-600 Error using the ORA-600 argument lookup tool
Can also be used to review and analyze ORA-7445 errors




Work closely with Oracle Support
Usually caused by bug such as memory leak
Test recent changes between development, test,
QA and production to ensure continuity of
issue.
Ideal to have same release of Oracle to be able
to reproduce the error for all environments
and platforms.

Use ADR to review incident and associated
trace file
Dump continued from file:
/u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora
_12562.trc
ORA-00600: internal error code, arguments: [kssadpm: null parent],
[], [], [], [], [], [], []


Plug in the arguments from trace file to ORA0600 Metalink troubleshooting lookup tool.
The related solution will then be displayed:
Known Issues: Bug# 5605370 * See Note 5605370.8* This bug is
alerted in Note 454464.1Various dumps / instance crash possible
Fixed: 10.2.0.4, 11.1.0.6


KFOD for ASM monitoring
KFOD help=y
[oracle@raclinux1 ~]$ kfod help=y
_asm_a/llow_only_raw_disks
KFOD allow only raw devices [_asm_allow_only_raw_disks=TRUE/(FALSE)]
_asm_l/ibraries
ASM Libraries[_asm_libraries=lib1,lib2,...]
_asms/id
ASM Instance[_asmsid=sid]
a/sm_diskstring
ASM Diskstring [asm_diskstring=discoverystring, discoverystring ...]
c/luster
KFOD cluster [cluster=TRUE/(FALSE)]
db/_unique_name
db_unique_name for ASM instance[db_unique_name=dbname]
di/sks
Disks to discover [disks=raw,asm,all]
ds/cvgroup
Include group name [dscvgroup=TRUE/(FALSE)]
g/roup
Disks in diskgroup [group=diskgroup]
h/ostlist
hostlist[hostlist=host1,host2,...]
metadata_a/usize
AU Size for Metadata Size Calculation
metadata_c/lients
Client Count for Metadata Size Calculation
metadata_d/isks
Disk Count for Metadata Size Calculation
metadata_n/odes
Node Count for Metadata Size Calculation
metadata_r/edundancy
Redundancy for Metadata Size Calculation
n/ohdr
KFOD header suppression [nohdr=TRUE/(FALSE)]
o/p
KFOD options type
[OP=DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/CLIENTS/RM/RMVERS/DFLTDSTR
/GPNPDSTR/METADATA/ALL]
p/file
ASM parameter file [pfile=parameterfile]
s/tatus
Include disk header status [status=TRUE/(FALSE)]
v/erbose
KFOD verbose errors [verbose=TRUE/(FALSE)]
KFOD-01000: USAGE: kfod op=<op> asm_diskstring=... | pfile=...
[oracle@raclinux1 ~]$ kfod disk=all
-------------------------------------------------------------------------------Disk
Size Path
User Group
================================================================================
1: 15358 Mb /dev/sdb1
oracle oinstall
2: 15358 Mb /dev/sdc1
oracle oinstall
3: 15358 Mb /dev/sdd1
oracle oinstall
4: 40954 Mb /dev/sde1
oracle oinstall
-------------------------------------------------------------------------------ORACLE_SID ORACLE_HOME
================================================================================
+ASM1 /u01/app/11.2.0/grid
[oracle@raclinux1 ~]$ kfod op=groups
-------------------------------------------------------------------------------Group
Size
Free Redundancy Name
================================================================================
1: 40954 Mb 30440 Mb EXTERN FLASH
2: 46074 Mb 42080 Mb NORMAL DATA
[oracle@raclinux1 ~]$
[oracle@raclinux1 ~]$ kfod op=insts
-------------------------------------------------------------------------------ORACLE_SID ORACLE_HOME
================================================================================
+ASM1 /u01/app/11.2.0/grid
[oracle@raclinux1 ~]$
[oracle@raclinux1 ~]$ kfod op=version
-------------------------------------------------------------------------------ORACLE_SID RAC VERSION
================================================================================
+ASM1 YES 11.2.0.1.0
[oracle@raclinux1 ~]$ kfod op=clients
-------------------------------------------------------------------------------ORACLE_SID VERSION
================================================================================
RACDB_1 11.2.0.1.0
+ASM1 11.2.0.1.0
+ASM1 11.2.0.1.0
RACDB_1 11.2.0.1.0
[oracle@raclinux1 ~]$
[oracle@raclinux1 ~]$ kfod op=rm
-------------------------------------------------------------------------------Rolling Migration State
================================================================================
Inactive
[oracle@raclinux1 ~]$ kfod op=rmvers
-------------------------------------------------------------------------------Rolling Migration Compatible Versions
================================================================================
11.1.0.6.0
11.1.0.7.0
[oracle@raclinux1 ~]$



Thank you for attending
Fill out your evaluation
Questions & Answers