<Insert Picture Here>
AWR 이용한 Detailed Analysis
Jun Kim
System Analyst
Solution Support Center
Agenda
•
•
•
•
•
•
•
•
•
Do You Know What is normal at your site
AWR 이란 ?
Statistics Stored In AWR
Viewing AWR Report
Thinking About Queue
AWR 이용한 분석 및 모니터링
Oracle ACS 진단 Service
Q&A
Appendix
<Insert Picture Here>
Do U Know What is normal at your site?

Get an accurate description of the problem

Check the OS for exhausted resources

Gather database statistics

Check for the most common pitfalls

Analyze the data gathered, focus on wait events, theorize the cause of
the problem

Propose a series of remedial actions, then implement them

Measure the effect of the changes

Repeat any steps, as required, until performance goals met
Do U Know What is normal at your site?
 Get an accurate description of the problem
 The scope of problem
 Instance wide or local to a user / group of users or a program.
 The expected response time or throughput
 The response time is getting slower or throughput decreased.
 The Time frame
 When does the problem occur?
 Any recent changes
 Data volume increases, New optimizer statistics or application
modification, SW/HW upgrades
Do U Know What is normal at your site?
 Check the OS for exhausted resources
 CPU Utilization
 CPU utilization / Load average ( Run queue )
 Memory Utilization
 Page out ( Swap out ) / SR ( scan ratio )
 Network Socket overflow
 Socket overflow / Drop Packets, Fragments
 Top Processes
 Top Processes have consumed lots of CPU
 CPU Utilization ( User 100% ) , it need more CPU or sub-optimal SQL ….?
Do U Know What is normal at your site?
 Gather database statistics
 Indicate the primary bottleneck
 Find primary bottleneck in database.
 STATSPACK / AWR
 STATSPACK ( 9i ) / AWR ( 10g ~~~)
 Third Party tool
 Third Party productions
 scripts
 Using Oracle dictionary view as you needed (dbmon.sql)
Do U Know What is normal at your site?
 Check for most common pitfalls
 Bad connection management
 Bad use of cursors and shared pool
 Use bind variable ( avoid hard parsing )
 Redo log setup problems
 Checkpoint ( IO related ) / Archive problem
 Serialization of data blocks in data buffers
 Application design / Heavy insert / block parameter…
 Long full table scan
 Missing Index / sub-optimal SQL Plan
 In disk sorting
 High amount of recursive SQL
 Space management
 Schema errors / Optimizer problems
 Use of nonstandard initialization parameters
Do U Know What is normal at your site?
 Analyze data gathered
 Find pattern
 Building Conceptual model
 Identify bottleneck
 Top 5 wait events
 Load profile
 Instance Efficiency
 Check OS resources as you need
 Top processes have used lots of CPU
 Top CPU SQL
 Top IO SQL
Do U Know What is normal at your site?
 Propose a series of remedial actions, implement them
 Change one thing one time
 Do not overlap when multiple changes are applied
 Measure the effect of the changes
 Re-examine the statistics
 Implement changes affect the performance ( good or bad )
 Iteration
 Until performance goal met
AWR(Automatic Workload Repository) 이란 ?
 Automatically collects & persists database instance statistics
 How to enable / disable & How to persist the data.
 STATSPACK++
 Cumulative value
 Default in 10G
 Database Load + Resources ( internal + external )
 Stored in SYSAUX
 SYSAUX TABLESPACE 크기 산정
 Provides in DBA_HIST_* View
 View sample
Statistics Stored In AWR
 Counter Statistics
 Ex “session logical reads”
 Value Statistics
 Ex “logons currents”
 Time Statistics
 Ex “DB time”
 Metric
 Ex “DB Block gets Per Txn”
 Sampled
 Ex “ASH ( Active Session History )”
Viewing AWR Report
 AWR Report displays all statistics captured over a snapshot
range
 Roughly equivalent to the STATSPACK report
 Available EM
 Manual Creation through SQL*Plus




awrrpt.sql
awrddrpt.sql  comparison two snapshots periods
ashrpt.sql  Detailed analysis of ASH data over small periods of time
awrsqrpt.sql  Detailed SQL Report
Think About Queue
•
Queuing 이론이란…..?



•
롯데리아에서 햄버거를 하나 주문 함에 있어 주문을 하기 전 기다리는 시간과 주문을
하여 햄버거를 받는 시간을 생각해 보자.
컴퓨터 시스템도 cpu를 할당 받아 일하는 시간과 cpu를 할당 받기 위해 기다리는 시간이
있다.
롯데리아와 다른점은 기다리는 줄이 하나라는 점이다.
Rt(Response Time) = Qt(Queuing Time) + St ( Service Time)


햄버가 하나 주문하는데 걸린 시간은 줄서 있는 시간 + 주문하여 받은 시간이다.
시스템의 응답시간도 위와 마찬가지 모델로 이루어져 있다.
Server 1
Server 2
Queue
Server3
Job1 Job 2 Job 3
 workload = job = load profile
 CPU = Server
Job N
Server4
AWR Report Header
DB Name
PROD
DB Id
Instance Inst num
3347383166 PROD1
1
Release
RAC
Host
10.2.0.3.0
YES
Hostname
Snap Id
Snap Time
Sessions
Cursors/Session
Begin Snap:
26885
06-Apr-09 07:30:10
1721
172.2
End Snap:
26907
06-Apr-09 18:30:01
6416
166.9
Elapsed:
659.86 (mins)
DB Time:
43,790.57 (mins)
 해당 SNAP구간에서 DB Server 에 대한 건강 검진을 받아 보자
Operating System Statistics
 Operating System Statistics
 시스템 CPU 사용에 대해 해당 snap 구간 및 추이를 확인 하여 특이사항 여부 를 판단해야 함.
24 18:00
24 17:00
24 16:00
24 15:00
24 14:00
24 13:00
24 12:00
24 11:00
24 10:00
24 09:00
24 08:00
Statistic
Total
AVG_BUSY_TIME
0  CPU는 혼자 움직이는 것이 아님.
AVG_IDLE_TIME
0 손님이 없는 식당의 종업은 항상 idle상태 ,혹은 serving이 아닌
AVG_IOWAIT_TIME
907,133 다른 일을 함 (ex:주방일… )
AVG_SYS_TIME
3,049,915  손님이 들어와 serving을 시작 하는 것과 같이 CPU User time을
증가시키는 workload가 있음. 해당 workload는 무엇인가 ?
AVG_USER_TIME
581,501
BUSY_TIME
249,064  CPU 추이를 일단위 , 일주일 더는 월단위로 파악하여 추세를
알 수 있도록 함.
IDLE_TIME
656,704
IOWAIT_TIME
114,461,748 ( 식당 종업원의 효율 및 고용 증가 및 감소 자료 활용)
SYS_TIME
384,451,747
USER_TIME
73,432,519
CPUCPU
Us age (%)
- LGEGERPIF1Q
Usage
(%)
LOAD
31,548,446 100%
%idle
90%
OS_CPU_WAIT_TIME
82,913,302
80%
%w io
RSRC_MGR_CPU_WAIT_
70%
%sys
0
60%
TIME
50%
%usr
40%
VM_IN_BYTES
104,440,000
30%
20%
VM_OUT_BYTES
0
10%
PHYSICAL_MEMORY_BY
0%
###############
TES
NUM_CPUS
126
Date & Time
NUM_CPU_SOCKETS
63
Load Profile
 CPU에 일을 시키는 여러 가지 작업임.
 Load Profile
Redo size:
Logical reads:
Block changes:
Physical reads:
Physical writes:
User calls:
Parses:
Hard parses:
Sorts:
Logons:
Executes:
Transactions:
Per Second
2,979,473.47
20,190.94
954,506.71
6,468.39
17,533.29
118.82
17,519.53
118.72
999.93
6.78
4,604.07
31.2
Per Transaction
2,979,473.47
20,190.94
954,506.71
6,468.39
17,533.29
118.82
17,519.53
118.72
999.93
6.78
4,604.07
31.2
 해당 load 파일을 통해 주어진 시간의
workload를 파악 할 수 있음
 어떤 workload를 통해 서버가 busy하게
되는지를 파악할 수 있음.
 예를 들어 커피숍에서 점심시간 동안
바쁘게 일을 했다면 , 그 시간 동안에
커피를 몇잔, 주스를 몇잔 혹은 홍차를
몇잔 아니면 음식재료를 준비하면서
시간을 소비했는지를 알 수 있음.
이를 통해 다른 날 보다 어떤 품목이 많이
팔렸는지를 알 수 있음.(특이 사항 검증 )
 Let’s Talk
위 load profile을 보고 해당 DB Server의 load를 말 할 수 있는가 ?
현재 Load profile을 확인 하고 load에 대한 서버의 영향 도를 말 할 수 있는가 ?
 DB의 STAT 은 DB Server에서 일어나는 여러가지 job으로 해석할 수 있으며, 특히 load profile의
stat은 DB Server내의 주된 workload로 이에 대한 trend 파악이 필요함. dba_hist_sysstat view참조.
Global Cache Load Profile
 앞에서 보았던 일반 load지표와 별도로
RAC에서의 load지표임.
 Global Cache Load Profile
Per Second
Per Transaction
Global Cache blocks received:
5,883.06
33.39
Global Cache blocks served:
7,278.89
41.31
GCS/GES messages received:
69,624.98
395.12
GCS/GES messages sent:
49,963.55
283.54
101.46
0.58
DBWR Fusion writes:
Estd Interconnect traffic (KB)
 예를 들어 커피숍에서 일하는 직원이
혼자가 아닌 둘 이상이라면 커피 주문에
대해 주문을 받고 , 커피를 만들고 하면서
두 직원 사이에 주고 받는 메시지 며,
커피잔 등…두사람 사이에 일어나는
동작에 대한 지표라 생각 할 수 있다.
 오라클에서 LMS 프로세스가 이런
operation을 하는 대표적이 프로세스라
할 수 있다.
128,652.73
 위 지표들은 노드간의 주고 받은 일의 양을 나타내는 지표로 이들 Job에 따른 성능의 영향에 대해
확인이 필요함.  workload Characteristics 참조
 interconnect traffic 양이 증가 한다며 , 우선적으로 cluster wait이 높은 SQL을 참조 할 필요가 있음.
 CR FLOW DIAGRAM
Global Cache Efficiency Percentages
 Global Cache Efficiency Percentages
Buffer access - local cache %:
97.8
Buffer access - remote cache %:
0.64
Buffer access - disk %:
1.56
 Global Cache and Enqueue Services - Workload Characteristics
Avg global enqueue get time (ms):
0.6
Avg global cache cr block receive time (ms):
0.8
Avg global cache current block receive time (ms):
0.7
Avg global cache cr block build time (ms):
0
Avg global cache cr block send time (ms):
0
Global cache log flushes for cr blocks served %:
3.9
Avg global cache cr block flush time (ms):
2.9
Avg global cache current block pin time (ms):
0
Avg global cache current block send time (ms):
0
Global cache log flushes for current blocks served %:
0.3
구분
CR Block Received
Time
CURR Block Received
Time
권고치
12 이하
30 이하
Load Profile Logical Reads & OS Stat Example
 Logical Reads
Logic al Re
ads (pe /
r s
ec)
Logical
Reads
Sec
1,400,000
1,200,000
1,000,000
800,000
600,000
400,000
24 18:30
24 18:00
24 17:30
24 17:00
24 16:30
24 16:00
24 15:30
24 15:00
24 14:30
24 14:00
24 13:30
24 13:00
24 12:30
24 12:00
24 11:30
24 11:00
24 10:30
24 10:00
24 09:30
24 09:00
24 08:30
0
24 08:00
200,000
Date & Time
 OS CPU 사항
CPU Usage (%)
CP U Us age (%) - LG EG ERP IF1Q
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
%idle
%w io
%sys
Date & Time
24 18:00
24 17:00
24 16:00
24 15:00
24 14:00
24 13:00
24 12:00
24 11:00
24 10:00
24 09:00
24 08:00
%usr
Wait Class
Wait Class
Waits
%Time - Total Wait
outs
Time (s)
Avg wait
(ms)
Waits /txn
User I/O
221,611,868
0
1,478,813
7
37.93
Cluster
493,638,715
0.03
406,741
1
84.49
System I/O
24,842,756
0
36,841
1
4.25
Other
24,548,596
51.48
35,896
1
4.2
1,445,321
3.64
26,966
19
0.25
Concurrency
147,553,001
0.05
23,523
0
25.26
Network
166,524,214
0
13,272
0
28.5
Commit
3,623,870
0
9,074
3
0.62
805,549
76.55
1,065
1
0.14
Application
RT = ST + WT
Configuration
Service Time은
어떻게 ?
 커피 점원이 커피 주문을 받아 본인이 하는 일과 기계가 하는 일로 나누어 볼 수 있다. 점원이 커피를 기계에서 기다리는
시간 동안이 wait 상태라 말 할 수 있음.
 오라클은 RT = ST + WT 으로 정의를 하고 있으며,여러 가지 wait event를 9개의 category로 분류 하고 있음.
 위 테이블을 통해 해당 snap 구간 중 어떤 wait class가 가장 많은 부분을 차지 하고 있음을 알 수 있음.
Top 5 Timed Events
Event
Waits
Time(s)
Avg Wait(ms)
% Total Call
Time
Wait Class
CPU time
910,471
45.2
db file sequential read 55,208,969
690,127
13
34.2
User I/O
log file sync
7,241,625
100,931
14
5
Commit
gc current block 3-way 31,287,718
75,500
2
3.7
Cluster
gc current block 2-way 41,850,395
70,941
2
3.5
Cluster
 Wait Class별이 아닌 개별 wait event로 snap 구간 중 가장 wait time이 높은 top 5 event임.
 평사 시 top 5 event는 업무 형태가 변경이 되거나 추가 되지 않는 다면 크게 변동이 없을 확률이 높음.
 top 5 event에 대해 평균 wait time에 대한 모니터링 및 평상시에 발생 하지 않은 event가 top 5 event로 올라 온다면
해당 event에 대해 검증이 필요함.
 예를 들어 과도하게 IO wait이 증가 하였다면 , AWR의 SQL Part의 high buffer gets / High disk read등을 통해 해당
시간의 SQL을 검증을 통해 특이 사항 여부를 확인 해야 함. 또는 과도하게 librarycache 및 shared pool관련 wait이
증가 하였다면 SGA의 Memory Part에서 해당 메모리의 증감 여부, invalidation여부 등을 확인 해야 함.
 오라클은 많은 수의 wait event를 가지고 있으며 , 각 event에 대해서는 oracle manual을 참조.
db file sequential read
 IO 성능 여부 검토
 AWR 리포트의 Avg Wait(ms), “IO Stats” section에서 Av Rd (ms) 항목의
수치가 다소 높게 나온다면 원인을 찾기 위해 O/S 및 Storage Level 에서의 IO
통계치를 살펴본다.
 IO 부하를 많이 발생시키는 SQL문들을 튜닝한다.
 AWR의 ‘SQL ordered by physical read‟를 검사하고, 튜닝을 실시한다.
 충분한 튜닝을 하였음에도 불구하고 여전히 IO 시스템에 대한
부하가 큰 경우는 주로 디스크 성능 자체의 bottleneck이거나
파일이나 디스크에 대한 작업배분이 잘못 설정되었을 가능성
검토
 이와 같은 IO bandwidth 문제에 대한 정확한 증거로는 ‘db file parallel write‟,
„direct read‟, „direct write‟, 그리고 ‘log file parallel write‟ 등의 event가 Top-5
waits events에 나타나는 경우가 많다.
log file sync
Redo log
1 commit
shadow
LGWR
4
redo
write down
2
3
commit 완료
 log file sync란 ?
Shadow process가 commit을 요청 시 LGWR가
해당 redo 를 online redo log file에 write한 후
해당 메시지를 shadow process에게 전달하는
event로 1 ~ 4번을 거쳐 완료 됨.
대표적인 log file sync 성능 저하 원인은…
 IO Bottleneck
On-line redo log
 IPC 통신 ( ex : semaphore )
 RAC 환경 하에서 SCN 동기화
등으로 해당 항목에 대해서 drill down해야 함.
Global Cache Wait Events
gc current block 2-way
Wait:
gc current request
LMS
1
Direct send
SGA2
2
LGWR
SGA1
Block transfer
Wait complete:
gc current block 2-way
LMS
3
LGWR:
Log sync
Global Cache Wait Events
gc current block 3-way
Wait:
gc current request
1
LMS
Direct
message
LMS
Resource
Master
2
SGA2
3
LGWR
SGA1
Block transfer
Wait complete:
gc current block 3-way
4
LMS
SGA Breakdown
Pool
Name
Begin MB
End MB
% Diff
large
free memory
221.68
221.55
-0.06
large
PX msg pool
2.32
2.45
5.39
shared
library cache
495.53
1,534.90
209.75
shared
sql area
4,131.00
6,888.16
66.74
shared
free memory
6,783.22
2,152.72
-68.26
 SGA의 large pool 및 shared pool의 free memory & sql area 등 주요 항목에 대해 증감 여부를 확인 할 수 있음.
 특정 메모리가 과도하게 증가하거나 감소 하였다면 해당 시점의 어떤 변화 가 있었는지를 load profile 및 wait event를
통해 먼저 확인 후 drill down 해야 함.
 메모리의 변동 사항 또한 CPU추이와 같이 하루 , 혹은 일주일 pattern을 모니터링 하는 것이 필요함.
Memory Statistics – library cache statistics
Namespace
BODY
CLUSTER
Get
Requests
4,383,881
Pct Miss
Pin Requests
0.01 137,380,417
Pct Miss
Invalidations
Reloads
0
443
0
995
0.4
1,438
0.56
4
0
1,433,787
0.02
1,553,075
0.16
2,047
0
2,468
0
40
7.5
3
0
189
0
567
0.35
2
0
PIPE
894,065
0.1
1,342,415
0.06
0
0
SQL AREA
299,644
97.391,458,251,177
0.12
233,980
12,122
4,479,060
0.2 189,209,372
0.05
33,573
0
0
129
0
INDEX
JAVA DATA
JAVA RESOURCE
TABLE/PROCEDURE
TRIGGER
105,057
0.08
5,933,235
 SGA의 shared pool area중 library cache영역에 대한 데이터로 reloads 및 invalidation에 대해 모니터링이 필요함.
SQL Statistics
 SQL Ordered by Elapsed Time
 SQL Ordered by CPU Time
 SQL Ordered by Gets
 SQL Ordered by Reads
 SQL Ordered by Executions
 SQL Ordered by Parse Calls
 SQL Ordered by Sharable Memory
 SQL Ordered by Version Count
 SQL Ordered by Cluster Wait Time
실제 OUPTPUT
TIME MODEL Statistics Hierarchy
DB TIME
•
•
•
•
DB CPU
Connection Management Elapsed Time
Sequence Load Elapsed Time
SQL Execute Elapsed Time
Repeated Binding Elapsed Time
• Parse Time Elapsed Time
Hard Parse Elapsed Time
Hard Parse (Sharing Criteria) Elapsed Time
Hard Parse Bind Mismatch Elapsed Time
Failed Parse Elapsed Time
Failed Parse (Out of Shared Memory)
•
•
•
•
PL/SQL Execution Elapsed Time
Inbound PL/SQL RPC Elapsed Time
PL/SQL Compilation Elapsed Time
Java Execution Elapsed Time
Time Model Statistics
 Time Model Statistics
Statistic Name
sql execute elapsed time
Time (s)
% of DB Time
2,206,102.92
97.5
DB CPU
672,296.64
29.71
PL/SQL execution elapsed time
248,800.73
11
parse time elapsed
64,204.13
2.84
connection management call elapsed time
60,816.54
2.69
repeated bind elapsed time
50,368.84
2.23
hard parse elapsed time
4,922.33
0.22
hard parse (sharing criteria) elapsed time
2,343.76
0.1
sequence load elapsed time
1,688.86
0.07
DB time
848.57
0.04
background elapsed time
616.18
0.03
343.6
0.02
background cpu time
Oracle ACS 진단 Service
 DCR ( Database Configuration Review )
 DPI ( Database Performance Inspection )
 PDI ( Periodic Database Inspection)
 ORA ( Operation Readiness Assessment )
 OSR ( Oracle System Review )
 BPR (Backup & Recovery Review )
 KNT ( Knowledge Transfer )
QUESTIONS
ANSWERS
Appendix
•
Create & Drop & Modify Snapshot
•
AUTOMATIC WORKLOAD REPOSITORY
•
CR Flow Diagram
•
DBA_HIST_SYSTAT
•
STATISTICS
•
Wait Events
<Insert Picture Here>
Create & Drop & Modify snapshot
 Create Snapshot
BEGIN
DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT ();
END;
/
 Drop Snapshot
BEGIN
DBMS_WORKLOAD_REPOSITORY.DROP_SNAPSHOT_RANGE (low_snap_id => 22,
high_snap_id => 32, dbid => 3310949047);
END;
/
 Modify Snapshot
BEGIN
DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS( retention => 43200,
interval => 30, topnsql => 100, dbid => 3310949047);
END;
/
Automatic Workload Repository
Report items
Main Report
Main Report
More RAC Statistics
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
• RAC Report Summary
(RAC Statistics)
• Global Enqueue Statistics
• Global CR Served Stats
• Global CURRENT Served Stats
• Global Cache Transfer Stats
Report Summary
Wait Events Statistics
SQL Statistics
Instance Activity Statistics
IO Stats
Buffer Pool Statistics
Advisory Statistics
Wait Statistics
Undo Statistics
Latch Statistics
Segment Statistics
Dictionary Cache Statistics
Library Cache Statistics
Memory Statistics
Streams Statistics
Resource Limit Statistics
init.ora Parameters
Automatic Workload Repository
Main Report
Report Summary
RAC Report Summary
•
•
•
•
•
• Global Cache Load Profile
• Global Cache Efficiency Percentages (Target
local+remote 100%)
• Global Cache and Enqueue Services - Workload
Characteristics
• Global Cache and Enqueue Services - Messaging
Statistics
Cache Sizes
Load Profile
Instance Efficiency Percentages (Target 100%)
Shared Pool Statistics
Top 5 Timed Events
Automatic Workload Repository
Main Report
Wait Events Statistics
SQL Statistics
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Time Model Statistics
Wait Class
Wait Events
Background Wait Events
Operating System Statistics
Service Statistics
Service Wait Class Stats
SQL ordered by Elapsed Time
SQL ordered by CPU Time
SQL ordered by Gets
SQL ordered by Reads
SQL ordered by Executions
SQL ordered by Parse Calls
SQL ordered by Sharable Memory
SQL ordered by Version Count
SQL ordered by Cluster Wait Time
Complete List of SQL Text
Automatic Workload Repository
Main Report
Instance Activity Statistics
Advisory Statistics
• Instance Activity Stats
• Instance Activity Stats - Absolute Values
• Instance Activity Stats - Thread Activity
•
•
•
•
•
•
•
•
•
•
IO Stats
• Tablespace IO Stats
• File IO Stats
Buffer Pool Statistics
Instance Recovery Stats
Buffer Pool Advisory
PGA Aggr Summary
PGA Aggr Target Stats
PGA Aggr Target Histogram
PGA Memory Advisory
Shared Pool Advisory
SGA Target Advisory
Streams Pool Advisory
Java Pool Advisory
Automatic Workload Repository
Main Report
Wait Statistics
Advisory Statistics
• Buffer Wait Statistics
• Enqueue Activity
•
•
•
•
•
•
•
•
Undo Statistics
• Undo Segment Summary
• Undo Segment Stats
Latch Statistics
•
•
•
•
•
Latch Activity
Latch Sleep Breakdown
Latch Miss Sources
Parent Latch Statistics
Child Latch Statistics
Segments by Logical Reads
Segments by Physical Reads
Segments by Row Lock Waits
Segments by ITL Waits
Segments by Buffer Busy Waits
Segments by Global Cache Buffer Busy
Segments by CR Blocks Received
Segments by Current Blocks Received
Automatic Workload Repository
Main Report
Dictionary Cache Statistics
Streams Statistics
• Dictionary Cache Stats
• Dictionary Cache Stats (RAC)
•
•
•
•
•
•
Library Cache Statistics
• Library Cache Activity
• Library Cache Activity (RAC)
Memory Statistics
• Process Memory Summary
• SGA Memory Summary
• SGA breakdown difference
Streams CPU/IO Usage
Streams Capture
Streams Apply
Buffered Queues
Buffered Subscribers
Rule Set
Resource Limit Statistics
Init.ora Parameters
Overall CR Flow Diagram
Context Switch
Interconnect
Message
Master
Node
(1)
ask for CR
and LOCK
in SHARE
mode
LMS
(2)
no conflict mode
so grant LOCK
Requestor
Node
LMS
(3 ’)
AST
for conversion
I/O
Holder
Node
(10) Send CR
buffer
LMS
(5)
ask
LGWR
to flush
REDO
(4 ’)
read since
LOCK is
granted
(4) Build CR block
and stop when
completed or IO
required
(8)
(6)
FG
(5 ’)
(3) Send
information
to LMS
including
(port,IP)
address
for answer
Database
LGWR
(7)
Log
Read With No Transfer
1
2
3
4
Read to Read Transfer
1
2
3
4
Read to Writer Transfer
1
2
3
4
Write to Writer Transfer
1
2
3
4
Write to Read Transfer
1
2
3
4
Writing Dirty Blocks
1
2
3
4
DBA_HIST_SYSSTAT & OSSTAT
 User calls
select snap_id,stat_name,
nvl(value,0) - nvl(lag(value) over (partition by stat_name order by snap_id),0) delta
from dba_hist_sysstat
where dbid=1134139816
and instance_number=4
and snap_id between 22199 and 22203
and stat_name='user calls‘ ;
 CPU
select snap_id ,
decode ( max(decode(stat_name, 'USER_TIME', delta, 0)) + max(decode(stat_name, 'SYS_TIME', delta,
0))
+ max(decode(stat_name, 'IDLE_TIME', delta, 0)),0,0, max(decode(stat_name, 'USER_TIME', delta, 0)) /
( max(decode(stat_name, 'USER_TIME', delta, 0)) + max(decode(stat_name, 'SYS_TIME', delta, 0)) +
max(decode(stat_name, 'IDLE_TIME', delta, 0)) ))
from
( select snap_id, stat_name,nvl(value,0) - nvl(lag(value) over (partition by stat_name order by snap_id),0)
delta ,
value
from dba_hist_osstat
where dbid=1134139816 and instance_number=4 and snap_id between 22199 and 22203
and stat_name in ( 'USER_TIME','SYS_TIME','IOWAIT_TIME','IDLE_TIME')
)
where a.snap_id > 22199 group by snap_id ;
Statistic : V$SYSSTAT
General Check Statistics
Statistic Name
consistent gets from cache
CPU used by this session
db block changes
db block gets from cache
execute count
Statistics Descriptions
Number of times a consistent read was requested for a block from
buffer cache.
Amount of CPU time (in 10s of milliseconds) used by a session from the
time a user call starts until it ends. If a user call completes within 10
milliseconds, the start and end user-call time are the same for purposes
of this statistics, and 0 milliseconds are added.
Closely related to "consistent changes", this statistic counts the total
number of changes that were part of an update or delete operation that
were made to all blocks in the SGA. Such changes generate redo log
entries and hence become permanent changes to the database if the
transaction is committed.
This approximates total database work. It statistic indicates the rate at
which buffers are being dirtied (on a per-transaction or per-second basis,
for example).
Number of times a CURRENT block was requested from the buffer
cache. This is a subset of "db block gets" statistics value
Total number of calls (user and recursive) that executed SQL
statements
Statistic : V$SYSSTAT
General Check Statistics
Statistic Name
logons cumulative
parse count ( hard )
parse count ( total )
parse time cpu
parse time elapsed
Statistics Descriptions
Total number of logons since the instance started. Useful only in
V$SYSSTAT. It gives an instance overview of all processes that logged
on.
Total number of parse calls (real parses). A hard parse is a very
expensive operation in terms of memory use, because it requires Oracle
to allocate a workheap and other memory structures and then build a
parse tree
Total number of parse calls (hard and soft). A soft parse is a
check on an object already in the shared pool, to verify that the
permissions on the underlying object have not changed
Total CPU time used for parsing (hard and soft) in 10s of
milliseconds
Total elapsed time for parsing, in 10s of milliseconds. Subtract
"parse time cpu" from the this statistic to determine the total
waiting time for parse resources
Statistic : V$SYSSTAT
General Check Statistics
Statistic Name
physical reads
physical reads cache
physical reads direct
physical writes
physical writes direct
redo entries
redo log space requests
Statistics Descriptions
Total number of data blocks read from disk. This value can be greater
than the value of "physical reads direct" plus "physical reads cache" as
reads into process private buffers also included in this statistic.
Total number of data blocks read from disk into the buffer cache. This is
a subset of "physical reads" statistic
Number of reads directly from disk, bypassing the buffer cache. For
example, in high bandwidth, data-intensive operations such as parallel
query, reads of disk blocks bypass the buffer cache to maximize transfer
rates and to prevent the premature aging of shared data blocks resident
in the buffer cache
Total number of data blocks written to disk. This statistics value equals
the sum of "physical writes direct" and "physical writes from cache"
values
Number of writes directly to disk, bypassing the buffer cache (as in a
direct load operation)
Number of times a redo entry is copied into the redo log buffer
Number of times the active log file is full and Oracle must wait for disk
space to be allocated for the redo log entries. Such space is created by
performing a log switch
Statistic : V$SYSSTAT
General Check Statistics
Statistic Name
redo size
session logical reads
Statistics Descriptions
Total amount of redo generated in bytes
The sum of "db block gets" plus "consistent gets". This
includes logical reads of database blocks from either the buffer
cache or process private memory
sorts ( disk )
Number of sort operations that required at least one disk write
sorts ( memory )
Number of sort operations that were performed completely in memory
and did not require any disk writes
user calls
user commits
user rollbacks
Number of user calls such as login, parse, fetch, or execute
Number of user commits. When a user commits a transaction, the redo
generated that reflects the changes made to database blocks must be
written to disk. Commits often represent the closest thing to a user
transaction rate
Number of times users manually issue the ROLLBACK statement or an
error occurs during a user's transactions
Statistic : V$SYSSTAT
RAC Related Statistics
Statistic Name
Statistics Descriptions
DBWR fusion writes
1. Fusion writes occur when a block previously changed by another
instance needs to be written to disk in response to a checkpoint or
cache aging
2. Oracle sends a message to notify the other instance that a fusion
write will be performed to move the data block to disk
3. Fusion writes do not require an additional write to disk
4. Fusion writes are a subset of all physical writes incurred by an
instance
gcs messages sent
lms및 lck에 의해 수행되는 gcs service에 대한 메시지임.
ges messages sent
lmd및 lmon에 의해 수행되는 ges / cgs service에 대한 메시지임.
Total number of consistent read blocks successfully received from
another instance.
gc cr blocks received
gc cr block receive time
gc current blocks received
gc current block receive
time
Accumulated round-trip time for all requests for consistent read blocks.
Number of current blocks received from the holding instance over the
interconnect
Accumulated round trip time for all requests for current blocks
Statistic : V$SYSSTAT
RAC Related Statistics
Statistic Name
gc cr blocks served
gc cr block build time
gc cr block flush time
gc cr block send time
gc current blocks served
gc current block pin time
gc current block flush time
gc current block send time
Statistics Descriptions
Number of requests for a CR block served by LMS
The time that the LMS process requires to create a CR block on the
holding instance.
Time waited for a log flush when a CR request is served (part of the
serve time)
Time required by LMS to initiate a send of the CR block.
The number of current blocks shipped to the requesting instance over
the interconnect
The time it takes to pin the current block before shipping it to the
requesting instance. Pinning is necessary to disallow changes to the
block while it is prepared to be shipped to another instance
The time it takes to flush the changes to a block to disk (forced log
flush), before the block is shipped to the requesting instance
The time it takes to send the current block to the requesting instance
over the interconnect
Statistic : V$SYSSTAT
RAC Related Statistics
Statistic Name
gc blocks lost
gc blocks corrupt
Statistics Descriptions
Block losses during transfers. May indicate network problems.
Blocks that were corrupted during transfer. High values indicate an IPC,
network, or hardware problem.
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
Description
CLASS
enq: TX - row lock when a session is waiting for a row level lock that is already held
contention
by another session
APPLICATION
This event indicates that the block was pinned or held up by a
gc buffer busy
session on a remote instance.
CLUSTER
gc cr block 2-way CR block received from master node ( master node=holder node) CLUSTER
CR block received from holder node ( master node != holder
gc cr block 3-way
node)
CLUSTER
the remote instance received the block after a remote instance
gc cr block busy
processing delay. In most cases, this is due to a log flush.
CLUSTER
gc cr block lost
The cr block request lost
CLUSTER
gc cr block
request
Placeholder event which is active while waiting for a block
CLUSTER
When a user session commits (or rolls back), the session's redo
information must be flushed to
the redo logfile by LGWR.
The server process performing the COMMIT or ROLLBACK waits under
this event for the write to
the redo log to complete.
COMMIT
log file sync
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
buffer busy waits
latch: cache
buffers chains
CLASS
This wait indicates that there are some buffers in the buffer
cache that multiple processes are attempting to access
concurrently. Query V$WAITSTAT for the wait statistics for each
class of buffer. Common buffer classes that have buffer busy
waits include data block, segment header, undo header, and
undo block.
Concurrency
The cache buffers chains latches are used to protect a buffer list
in the buffer cache. These latches are used when searching for,
adding, or removing a buffer from the buffer cache. Contention
on this latch usually means that there is a block that is greatly
contended for (known as a hot block).
Concurrency
latch: shared pool Shared pool latch contention
row cache lock
library cache pin
Concurrency
The session is trying to get a data dictionary lock.
Concurrency
This event manages library cache concurrency. Pinning an object
causes the heaps to be loaded into memory. If a client wants to
modify or examine the object, the client must acquire a pin after
the lock.
Concurrency
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
library cache lock
enq: HW contention
enq: SQ contention
enq: TT contention
CLASS
This event controls the concurrency between clients of the library
cache. It acquires a lock on the object handle so that either:
•One client can prevent other clients from accessing the same
object
•The client can maintain a dependency for a long time which
does not allow another client to change the object
This lock is also obtained to locate an object in the library cache. Concurrency
The HW enqueue is used to serialize the allocation of space beyond
the high water mark of a segment.
Configuration
Sequence number enqueue
Configuration
Temporary table enqueue
Configuration
The number of ITL slots in any block in an object is controlled by
the INITRANS and MAXTRANS attributes. INITRANS is the
enq: TX - allocate number of slots initially created in a block when it is first used,
while MAXTRANS places an upper bound on the number of
ITL entry
entries allowed. Each transaction which wants to modify a block
requires a slot in this 'ITL' list in the block.
Configuration
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
free buffer waits
latch: redo copy
log buffer space
CLASS
This wait event indicates that a server process was unable to find
a free buffer and has posted the database writer to make free
buffers by writing out dirty buffers. A dirty buffer is a buffer whose
contents have been modified. Dirty buffers are freed for reuse
when DBWR has written the blocks to disk.
Configuration
When a sessions redo buffer is larger than Parameter:
log_small_entry_max_size the kernel first allocates a redo copy
buffer, protected by a redo copy latch.
The buffer will not be used until space is allocated on the log
buffer and some header has been set. However, the redo copy
latch is acquired to reduce the code inside the allocation latch
holding and to prevent further contention.
Configuration
The system is waiting for space in the log buffer because data is
being written into the log buffer faster than LGWR can write it out. Configuration
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
CLASS
Waiting for a log switch because the log that the LGWR will be
switching into has not been archived yet. Check the alert file to
log file switch
make sure that archiving has not stopped due to a failed archive
(archiving needed)
write. To speed archiving, consider adding more archive
processes or putting the archive files on striped disks.
Configuration
Waiting for a log switch because the session cannot wrap into the
log file switch
next log. Wrapping cannot be performed because the checkpoint
(checkpoint
for that log has not completed
incomplete)
Configuration
log file switch
Waiting for a log switch to complete.
completion
Configuration
enq: US Undo segment DDL
contention
Other
The process waits for a latch that is currently busy (held by
latch free
another process).
Other
DFS lock handle
The session waits for the lock handle of a global lock request. The lock
handle identifies a global lock. With this lock handle, other operations
can be performed on this global lock (to identify the global lock in
future operations such as conversions or release). The global lock is
maintained by the DLM.
Other
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
db file parallel
write
CLASS
This event occurs in the DBWR. It indicates that the DBWR is
performing a parallel write to files and blocks. When the last I/O has
gone to disk, the wait ends.
System I/O
log file parallel
write
log file sequential
read
This event involves writing redo records to the redo log files from
the log buffer.
System I/O
Waiting for the read from this logfile to return. This is used to read
redo records from the log file.
System I/O
Waiting for the write to this logfile to complete. This event is used
while updating the header of the logfile. It is signaled when
log file single write
adding a log file member and when incrementing sequence
numbers.
System I/O
This happens during recovery. It can also happen during buffer
prefetching, as an optimization (rather than performing multiple
db file parallel read
single-block reads). Database blocks that need to be changed as
part of recovery are read in parallel from the database.
User I/O
This event signifies that the user process is reading a buffer into
db file sequential
the SGA buffer cache and is waiting for a physical I/O call to
read
return. A sequential read is a single-block read.
User I/O
db file single write This event is used to wait for the writing of the file headers.
User I/O
Wait Event : V$SESSION_WAIT
General Check Wait Event
NAME
CLASS
A db file scattered read issues a scattered read to read the data
into multiple discontinuous memory locations. A scattered read is
db file scattered
usually a multiblock read. It can occur for a fast full scan (of an
read
index) in addition to a full table scan.
User I/O
During Direct Path operations, the data is asynchronously written
to the database files. At some stage the session needs to make
direct path write /
sure that all outstanding asynchronous I/O have been completed
temp
to disk. This can also happen if, during a direct write, no more
slots are available to store outstanding load requests
User I/O
During Direct Path operations the data is asynchronously read
from the database files. At some stage the session needs to
direct path read /
make sure that all outstanding asynchronous I/O have been
temp
completed to disk. This can also happen if during a direct read no
more slots are available to store outstanding load requests
User I/O