Waits Events

Waits Events
and the
Geeks who love them
Kyle Hailey
http://perfvision.com
#.2
WaitEvents
Wait Events
Copyright 2006 Kyle Hailey

#.3
And the Geeks Who Love Them

#.4
In this Presentation:
Tuning Methodology
Plan of Action
Statspacks for Collection Data
Based on Waits
Waits Cheat Sheet

top 36 Waits
Details
4 hardest Waits

#.5
Word of Wisdom
Half of the game is knowing when to act and
how much effort to put in

#.6
Database is Hung!
Everybody blames the database
Poor Database
Yet 9 out of 10 dbas agree its not the
database
How do you prove it to management?
On the off chance its the database, now
we are in some serious trouble!

Database: #.7
Guilty until proven innocent
*$%@!!

#.8
Oracles Defense
After years of false accusations
Oracle took action and created a defense
system:
WAIT EVENTS
To the rescue

#.9
Oracle Instrumentation
CPU
Locks
Redo Lib Buffer

Cache Cache
Network
IO

#.10
Waits
Instrumented code to indicate bottlenecks
Number of times waited
Amount of time waited
Examples
IO
Locks
SQL*Net
Wouldnt it be great if Applications took some

responsibility and instrumented their code?

#.11
Use Statspack to Find Waits
Statspack Cheat Sheet:

Install
Connect as SYSDBA
@?/rdbms/admin/spcreate.sql
Run
Exec statspack.snap;
Generate Reports
@?/rdbms/admin/spreport.sql

#.12
Statspack
Look at Top 5 Timed Events (~50 lines down)
Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Call Time
-------------------------------------------- ------------ ----------- ---------
buffer busy waits 2,748 250 78.72
CPU time 32 10.16
free buffer waits 1,588 15 4.63
write complete waits 10 8 2.51
log buffer space 306 5 1.51

#.13
Method
Relax if
CPU + WAIT << Available CPU
Get to work if
CPU + WAIT >= Available CPU

#.14
Available CPU
Available CPU = # CPUs * Elapsed time
# of CPUs
SQLPLUS> show parameters cpu_count
NAME TYPE VALUE

------------------ -------- ----------
cpu_count integer 2
#.15
Elapsed Time
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host

------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess

--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8
Elapsed: 1.00 (mins)

#.16
Available CPU
# CPUs * Elapsed Time
Example:
2 CPU * ( 1.0 min * (60 sec/min) )
= 120 secs CPU time available

#.17
Used CPU Time and Wait Time

Look at Top 5 Timed Events
Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
-------------------------------------------- ------------ ----------- ---------
CPU time 32 10.16

#.18
Example
CPU + WAITS
= 250 + 32 + 15 + 8 + 5 = 310 secs
Available CPU was 120 secs
310 >> 120
Get to work tuning ! Event Time (s)
--------------------------- -----------
buffer busy waits 250
CPU time 32
free buffer waits 15
write complete waits 8
log buffer space 5
#.19
Dependable Tuning Strategy

Run Statspack
Top 5 Timed Events
~50 lines down from top
Need Available CPU
Elapsed Time
CPU_COUNT
OEM 10g
Performance Page does everything

#.20
Data Sources
Statspack
Top 5 Timed Events
10g ASH
OEM
ASH Report : ashrpt.sql
Custom
V$session,v$session_wait
v$active_session_history

Statspack #.21
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host

------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess

--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8

#.22
Statspack
Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
-------------------------------------------- ------------ ----------- ---------
CPU time 32 10.16

#.23
OEM 10g
9i &10g
Licensed but worth it
Does all the work
Aggregates Wait time
Sums with CPU time
Displays available CPU
Groups Waits
Filters out unusable waits

#.24
OEM 10g Performance Tab

#.25
OEM 10g Zoom-In
CPU + WAIT
Available CPU
#.26
OEM 10g Zoom-In
Relax Get to Work!

#.27
OEM 10g Zoom-In

#.28
Top Activity Page

#.29
Tuning Methodology
Machine
Run queue (CPU)
reduce CPU usage or add CPUs
Paging
Reduce memory usage or add memory
Oracle
Waits + CPU > Available CPU
Tune waits
CPU 100% We are going to
Tune SQL concentrate here
on WAITS
#.30
Waits
360 waits in 9i
36 waits represent 99% of all Bottlenecks
From Anjo Kolks site www.oraperf.com
Based on total wait times
12 Wait events represent 80%

#.31
Top 12 Waits
NAME Count % Total
1. db file sequential read 23,850.00 11.67%
2. log file sync 20,594.00 10.08%
3. db file scattered read 15,505.00 7.59%
4. latch free 11,078.00 5.42%
5. enqueue 7,732.00 3.78%
6. SQL*Net more data from client 7,510.00 3.67%
7. direct path read 5,840.00 2.86%
8. direct path write 4,868.00 2.38%
9. buffer busy waits 4,589.00 2.25%
10. SQL*Net more data to client 3,805.00 1.86%
11. log buffer space 2,990.00 1.46%
12. log file switch completion 2,878.00 1.41%
Above is over 80% of wait times reported
Anjo Kolk www.oraperf.com

#.32
Top 36 Waits
1. db file sequential read 19. write complete waits
2. log file sync 20. library cache lock
3. db file scattered read 21. SQL*Net more data from dblink
4. latch free 22. log file switch (checkpoint incomplete)
5. enqueue 23. library cache load lock
6. SQL*Net more data from client 24. row cache lock
7. direct path read 25. local write wait
8. direct path write 26. sort segment request
9. buffer busy waits 27. process startup
10. SQL*Net more data to client 28. unread message
11. log buffer space 29. file identify
12. log file switch completion 30. pipe put
13. library cache pin 31. switch logfile command
14. SQL*Net break/reset to client 32. SQL*Net break/reset to dblink
15. io done 33. log file switch (archiving needed)
16. file open 34. Wait for a undo record
17. free buffer waits 35. direct path write (lob)
18. db file parallel read 36. undo segment extension
minus Background, OPS, RAC, PQ, Resource Mgr http://www.oraperf.com
#.33
Wait Areas
Buffer Cache
I/O
Locks
Waits Library Cache
Redo
SQL*Net

#.34
Write IO
Wait Tree Read IO Rollback
Buffer Busy Free lists
IO
Cache Latches IO Read
Buffer Cache
Library Cache
Library Cache
Shared Pool
Waits
Lock TX Row Lock
Redo TX ITL Lock
SQL Net HW Lock
Log File
Log Buffer
Log File Sync
#.35
Redo
REDO

#.36
Redo Wait Solutions
2 log file sync
Commit less, put redo logs on faster disks
11 log buffer space
Increase log buffer
12 log file switch completion
Increase log file sizes
22 log file switch (checkpoint incomplete)
Add log files (or increase log file size)
30 switch logfile command
Avoid switching log files
32 log file switch (archiving needed) ***
Archive log running out of space
#.37
Library Cache
REDO Lib
Cache

#.38
Library Cache
4 latch free need p1,p2,p3
library cache
shared pool
13 library cache pin
Reduce parsing the same cursor concurrently
20 library cache lock
Avoid hard parsing same cursor at same time
23 library cache load lock
24 row cache lock
need p1
#.39
Buffer Cache
REDO Lib Buffer

Cache Cache

#.40
Buffer Cache Details
4 latch free need p1,p2,p3
cache buffers chains
cache buffers chains LRU
9 buffer busy waits need p1,p2,p3 and SQL
11 log buffer space
Increase log buffer
17 free buffer waits
Increase buffer cache size
19 write complete waits
Increase buffer cache (or tune DBWR)
25 local write wait
truncates / reduce cache size

#.41
IO
REDO Lib Buffer

Cache Cache
IO

IO #.42
1 db file sequential read

Tune SQL, speed up disks (5-15ms), increase buffer cache
3 db file scattered read
FTS - Tune SQL, add indexes, speed up disks (5-15ms)
sorts or PQO - tune IO, sort less
7 direct path read
8 direct path write
direct path load or temp io, improve disk speed
15 io done
Io done by IO slaves, oracleIxxx could also be an idle wait sort of worthless
16 file open
Reduce logons/logoffs or keep log files open
18 db file parallel read
Tune sql, tune io subsystem, increase buffer cache
26 sort segment request
SMON busy, process rollback problem
28 file identify
Keep log files open, reduce checkpoints
34 direct path write (lob)
Improve IO, reduce lob write size
#.43
Locks
Locks
REDO Lib Buffer

Cache Cache
IO

#.44
Waits 1-9
5 enqueue need p1,p2,p3 and SQL
HW
TM
TX 4
TX 6
ST
SQ
TS

#.45
Network
Locks
Redo Lib Buffer

Cache Cache
Network
IO

#.46
SQL*Net
6 SQL*Net more data from client
Usually OK, reduce data transferred, possible Network problems
10 SQL*Net more data to client
Usually OK, reduce amount of data transferred, possible Network
tuning needed
14 SQL*Net break/reset to client
Check for errors in sql statement
21 SQL*Net more data from dblink
Reduce data transfer, check net response
31 SQL*Net break/reset to dblink
Check for errors in sql statement sent

#.47
CPU
100% CPU
tune highest SQL
Locks
Redo Lib Buffer

Cache Cache
Network
IO

#.48
CPU
Different from Waits
Still a Timed Event
High CPU & Low Waits
Tune SQL

Application #.49
100% CPU
tune highest SQL
Locks
Redo Lib Buffer ?

Cache Cache
Network
IO

#.50
Its the Application

If all these areas show no contention , its the
application

#.51
Waits we will Ignore

Background
Idle
RAC
Resource Manager

#.52
Background & Foreground

Background Processes
DBWR
LGWR
PMON
SMON
Etc
Foreground Processes
SQL*Plus
Pro*C
SQL*Forms
Oracle applications
Only interested in Foreground waits

#.53
Background Processes
SGA
PMON Log Library Buffer
Buffer Cache Cache
Buffer Cache
SMON
Log Buffer
DBWR
LGWR
User1
User2
User3
REDO Log Files Data Files

#.54
Background Waits
Filter Out Background Waits
Statspack
ASH : SESSION_TYPE='FOREGROUND
V$session_wait : type='USER'

#.55
Background Waits
ASH 10g
Avoid Background waits in ASH with
Select from v$active_session_history

where SESSION_TYPE='FOREGROUND'
V$session_wait joined to v$session
select
from v$session s,
v$session_wait w
where w.sid=s.sid
and s.type='USER'
#.56
Idle Waits
Filter Out
10g
where wait_class != Idle
Create a list
Select name from v$event_name where
wait_class=Idle;
9i
Create a list with
Documentation
List created from 10g

#.57
Parallel Query Waits

Filter Out
Parallel Query Wait events are unusable
Save waits are both idle and waits
Parallel Query Waits start with PX or KX
PX Deq: Par Recov Reply
PX Deq: Parse Reply

#.58
RAC Waits
You are on your own
Check documentation
If you are not using RAC then no worries
10g
Select event from v$event_name where

wait_class=Cluster;
9i
RAC and OPS waits usually contain the word global

#.59
Resource Manager Waits

Resource manager throttles user
Createswait
Obfuscates problems
10g
select name from v$event_name where

wait_class='Scheduler';

#.60
Easy Waits vs Difficult Waits

Easy Waits
Can take action without further investigation
Difficult Waits, need
Real time analysis or historical collections
V$session_wait
V$session
V$sql

#.61
Easy Waits
1. log buffer space
Increase log buffer
2. log file switch completion
3. log file switch (checkpoint incomplete)
4. log file switch (archiving needed)
5. switch logfile command
6. write complete waits
7. free buffer waits

#.62
Intermediate
Library Cache locks & Pins
Run queries to find lockers
IO
Check average read times per file
Should be between 5-20 ms
Data in Statspack under File IO Stats
Check Cache buffer Hit ratio

Check db_cache_advice 9i and higher
Data in Statspack under Buffer Pool Advisory
Tune High IO SQL
Need to follow Advanced Wait steps to find SQL

#.63
Intermediate : Lib Cache Handles

1. library cache lock
2. library cache pin
Reduce parsing the same cursor concurrently
3. library cache load lock

#.64
Intermediate - IO
1. db file sequential read
2. db file parallel read -
3. db file scattered read
FTS , Tune SQL, add indexes, speed up disks (5-15ms)
4. direct path write (lob)
5. direct path read
6. direct path write

#.65
Library cache pin

select
w.sid,
kglob.KGLNAOBJ
from
x$kglob kglob,
v$session_wait w
where
kglob.KGLHDADR= w.P1RAW and
event like '%library%';

#.66
Library cache lock
Find the waiters and who blocks them
column wevent format a20
column bevent format a20
select
waiter.sid waiter,
waiter.p1raw wlockp1,
waiter.event wevent,
blocker_event.sid blocker,
blocker_event.event bevent
from
x$kglpn p,
gv$session blocker_session,
gv$session_wait waiter,
gv$session_wait blocker_event
where
p.kglpnuse=blocker_session.saddr and
p.kglpnhdl=waiter.p1raw and
(waiter.event like 'library cache lock' ) and
blocker_event.sid=blocker_session.sid
order by
waiter.p1raw,waiter.sid
/ Copyright 2006 Kyle Hailey
#.67
IO
2. db file parallel read
FTS , Tune SQL, add indexes, speed up disks (5-15ms)
5. direct path read
direct path load or temp io, improve disk spee

#.68
IO Solutions
If
Db scattered Read
Db file sequential Read
Db file parallel Read
Then
Check average read times per file
Should be between 5-15 ms
Data in Statspack under File IO Stats
Check Cache buffer Hit ratio

Check db_cache_advice 9i and higher
Data in Statspack under Buffer Pool Advisory
Tune High IO SQL
#.69
IO Solutions : Ave Read Time

File IO Stats DB/Inst:labsf03 Snaps: 1-2
->Mx Rd Bkt: Max bucket time for single block read
->ordered by Tablespace, File
Tablespace Filename
------------------------ ----------------------------------------------------
Av Mx
Av
Av Rd Rd Av Av Buffer BufWt
Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)
---------- ---- ------- ----- --- ------- ------------ -------- ---------- --
SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf
445 15 0.4 16 1.0 1,157 39 2,744 93.3
USERS /u01/app/oracle/oradata/labsf03/users01.dbf
223 7 0.5 ### 1.0 9,725 324 4 100.0

IO Solutions : Buffer Pool #.70
Advisory
Buffer Pool Advisory
Size for Size Buffers for Read Estimated
P Est (M) Factor Estimate Factor Physical Reads
--- -------- ------ ------------ ------ --------------
D 56 .1 6,986 2.3 58,928
D 112 .2 13,972 1.6 42,043
D 224 .4 27,944 1.0 25,772
D 336 .6 41,916 1.0 25,715
D 448 .8 55,888 1.0 25,715
D 596 1.0 74,351 1.0 25,715
D 728 1.2 90,818 1.0 25,715
D 840 1.4 104,790 1.0 25,715
D 952 1.6 118,762 1.0 25,715
D 1,064 1.8 132,734 1.0 25,715

#.71
IO Solutions
After Checking
FileIO response times
Buffer Cache Hit Ratio
Then the only choice is to

Tune the SQL
See Advanced Wait on how to find SQL

#.72
Advanced
1. buffer busy waits
Buffer wait Statistics
v$waitstats
P1 file#, p2 block#, p3 class
2. row cache lock
Dictionary Cache Stats
v$rowcache
P1 rowcache #
3. latch free
Latch Sleep breakdown
V$latch
P1 latch#
4. Enqueue
Statspack doesnt help
V$lock
P1 lock type and mode

#.73
Advanced Waits Need

Wait Arguments
P1
P2
P3
SQL Statements
Session IDS

#.74
Gathering More Data - Basic
select
s.sid, /*SESSION */
w.event , /* WAIT */
s.sql_hash_value, /* SQL */
w.p1, w.p2, w.p3 /* P1, P2 , P3 */
from
v$session s,
v$session_wait w
where
w.sid=s.sid
/
#.75
Gathering More Data SQL Text

select
w.sid, w.event,
w.p1, w.p2, w.p3 ,
sql.sql_text
from
v$session_wait w,
v$session s,
v$sql sql
where
s.sid = sw.sid and
sql.address(+) = s.sql_address and
s.type='USER
/ Copyright 2006 Kyle Hailey
#.76
SQL Text
col status for a35
select s.sid,
s.sql_hash_value,
decode(w.wait_time, 0, w.event , 'CPU') as status,
w.p1, w.p2, w.p3
sql_text
from v$session s,
v$session_wait w,
v$sql sql
where w.sid=s.sid
and s.status='ACTIVE'
and s.type='USER'
and w.event not in ('jobq slave wait',
'rdbms ipc reply')
and sql.address(+) = s.sql_address ;
#.77
Gathering More Data - Filtering I
and lower(w.event) not in (
'queue monitor wait',
'null event',
'pl/sql lock timer',
'px %',
'sql*net message from client',
'sql*net message from dblink',
'dispatcher timer',
'lock manager wait for remote message',
'pipe get',
% timer',
'queue messages',
'rdbms ipc message',
'slave wait',
'virtual circuit status',
'wakeup time manager',
'i/o slave wait',
'jobq slave wait',
'queue monitor wait
) Copyright 2006 Kyle Hailey
#.78
Gathering More Data Filtering II
col status for a35
select s.sid,
s.sql_hash_value,
w.p1, w.p2, w.p3
from v$session s,
v$session_wait w
where w.sid=s.sid
and s.type='USER'
'rdbms ipc reply');
#.79
Detailed Data Results
SID SQL_HASH STATUS P1 P2 P3

---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
235 3336613934 enq: US - contention 1431502854 44 0
248 3333220954 CPU 1650815232 1 0
260 1272059733 CPU 1431502854 44 0

#.80
Wait Arguments: P1,P2,P3
Each Wait has a 3 parameters P1,P2,P3

Give detailed information
Meaning different for each wait
Meaning defined in V$event_name

#.81
Wait Arguments Example

NAME PARAMETER1 PARAMETER2 PARAMETER3
----------------------------------- --------------- --------------- ---------------
latch: cache buffers chains address number tries

free buffer waits file# block# set-id#
buffer busy waits file# block# class#
latch: redo copy address number tries
log buffer space
switch logfile command
log file sync buffer#
db file sequential read file# block# blocks
enq: TM - contention name|mode object # table/partition
undo segment extension segment#
enq: TX - row lock contention name|mode usn<<16 | slot sequence
Select
row cache lock cache id mode request
library cache pin handle address pin address name,
100*mode+namesp
library cache load lock
pipe put
object address
handle address
lock address
1,
record length
parameter
100*mask+namesp
timeout
2, parameter
3 parameter
Copyright 2006 Kyle Hailey from
#.82
Difficult Waits
Multiple causes and solutions
Latches
Locks
Buffer Busy
Row Cache Lock

#.83
Latches
Protect memory for concurrent use
protect lines of code
Light weight locks
Bitin memory
Atomic processor call
Fast and cheap
Gone if memory is lost
Exclusive Generally
Sharing reading has been introduced for some latches
#.84
Finding Latches
latch free
Covers many latches, find the problem latch by
1. select name from v$latchname where latch# = p1;
OR
2. Find highest sleeps in Statspack latch section
In 10g, important latches have a wait event
latch: cache buffers chains
latch: shared pool
latch: library cache

#.85
Statspack : Latch free

Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------- ----------- --------
CPU time 76 62.62
latch free 168,185 41 33.45
wait list latch free 251 5 3.71
library cache pin 2 0 .08
log file parallel write 52 0 .06

#.86
Latches : Find Highest Sleeps

Use Statspack (or AWR report)
Latch Sleep breakdown for DB: ORA9
-> ordered by misses des
Latch Name Get Requests Misses Sleeps
------------------- -------------- ----------- -----------
shared pool 3,044,462 184,517 19,098
library cache 2,242,805 79,554 15,198
kks stats 263,464 2,842 1,774
row cache object 434,844 955 147
library cache lock 666,668 734 54

#.87
Easy Latches
Cache Buffers LRU Chain
Too much activity , uses multiple buffer caches and/or increase LRU
latches
Shared Pool
Too much hard parsing, too small a shared pool
Library Cache Latch

Hard Parsing
Use bind variables
Uses cursor_sharing=force
Increase shared pool size if reloads are high
Avoid invalidations
Soft Parsing
Hold cursors open
Use session_cached_cursors
Use cursor_space_for_time
#.88
Advanced Latches
Cache Buffers Chains
Hot blocks
Need to find the SQL to solve
Root of Index
Partition
X$dual
Select for update

#.89
Buffer Busy Waits

First need to see p3
Read (100,110,120,130 )
IO
In10g becomes
read by other session
Write (200,210,220,230, 231 )
Depends on Block type

#.90
Buffer Busy Cases

Undo Header - Not enough rollback segments
Undo Block hot spot in RBS
Data Block
Depends on Object type
Segment Header add free Lists (with Datablock)
Free List add free lists groups

#.91
Buffer Busy Wait - Object

Table Block w/ DML : Table lacks Free lists
Index : has hot block, partition or change SQL
select segment_name,
segment_type
from dba_extents
where file_id = P1
and P2 between
block_id and block_id + blocks 1;

#.92
Buffer Busy Waits

Statspack
Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3
-> ordered by wait time desc, waits desc

Tot Wait Avg
Class Waits Time (s) Time (ms)
------------------ ----------- ---------- ---------
data block 1,490 2 1
undo header 2 0 0

#.93
Buffer Busy Waits

Statspack
Buffer wait Statistics DB/Inst: linux3 Snaps: 39-42
-> ordered by wait time desc, waits desc
Class Waits Wait Time(s) Avg Time (ms)

------------------ ------ ------------- -------------
undo header 1,159 18 16
data block 132 2 12
file header block 45 0 2
segment header 4 0 10
undo block 1 0 0

#.94
Buffer Busy Solutions

IF Buffer Busy Waits high
First look at v$waitstat
Class
Time/Count
SQL> select * from v$waitstat;
CLASS COUNT TIME

------------------ ---------- ----------
data block 1 1
segment header 0 0
free list 0 0
undo header 0 0
undo block 0 0
#.95
Row Cache Lock

Statspack
^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2

->"Pct Misses" should be very low (< 2% in most cases)
->"Cache Usage" is the number of cache entries being used
->"Pct SGA" is the ratio of usage to allocated size for that cache
Get Pct Scan Pct Mod Final

Cache Requests Miss Reqs Miss Reqs Usage
----------------- --------- ------ ------- ----- -------- ----------
dc_object_ids 45 0.0 0 0 958
dc_objects 89 0.0 0 0 1,129
dc_segments 69 0.0 0 0 807
dc_tablespaces 12 0.0 0 0 13
dc_usernames 22 0.0 0 0 19
dc_sequences 120,003 0.0 0 120,003 5

#.96
Row Cache Lock
Need p1 to see the cache type
SQL> select cache#, parameter from v$rowcache;
CACHE# PARAMETER
---------- --------------------------------
1 dc_free_extents
4 dc_used_extents
2 dc_segments
0 dc_tablespaces
5 dc_tablespace_quotas
6 dc_files
7 dc_users
3 dc_rollback_segments
8 dc_objects
17 dc_global_oids
12 dc_constraints
#.97
Row Cache Lock

Find Row Cache
Find SQL
Example
row cache : sequence
sql : select seq.next_val
problem : sequence had cache of 1
solution: increase sequence cache to 20
#.98
Enqueues aka Locks

Enqueue wait covers all locks pre 10
Protect data against concurrent changes
Lock info written into data structures
Block headers
Data blocks
Written in cache structures
Shareable in compatible modes

#.99
Finding Locks
Statspack no help
V$session_wait needs lots of decoding
P1 tells Lock Type and Mode
P2,P3 give more data
Usually Need SQL to solve

#.100
Finding Locks
select sid,
event,
chr(bitand(P1,-16777216)/16777215)||
chr(bitand(P1,16711680)/65535) as "Type",
mod(p1,16) as "mode"
from v$session_wait
where event = 'enqueue;
SID EVENT Ty mode P2 P3

--- ------- -- ---- ------- ----
240 enqueue TX 6 2686995 433

#.101
Locks 10g
10g breaks Enqueues out
enq: HW - contention Configuration
enq: TM - contention Application
enq: TX - allocate ITL entry Configuration
enq: TX - index contention Concurrency
enq: TX - row lock contention Application
enq: UL - contention Application

#.102
Locks : TM & TX
select * from v$lock where type in ('TX', 'TM');
SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK

--- -- ---------- ---------- ----- ------- ----- -----
151 TX 589855 1592 0 6 4049 0
135 TM 53737 0 3 0 4058 0
151 TM 53737 0 3 0 4049 0
135 TX 589855 1592 6 0 4058 1

#.103
Utllockt.sql
@?/rdbms/admin/catlock.sql
@?/rdbms/admin/utllockt.sql
WAIT_SES LOCK_TYPE MODE_REQ MODE_HELD LOCK_ID1 LOCK_ID2

--------- ----------- --------- --------- -------- --------
13 None
12 Transaction Exclusive Exclusive 65537 14960

#.104
Summary
Waits make Tuning Easy
Check Machine Health
Tune Waits
Tune CPU
Tune SQL
Use Statspack or OEM 10g
Ignore Background, Idle, PQO (Rac, Resmgr)
Cheat Sheet on Top 36 Waits
Details on the 4 hardest Waits
See http://perfvision.com for more info

#.105
10g v$active_session_history
Best source
10g only
Data exists since v7
Can simulate v$active_session_history

#.106
Querying ASH
Top 5 Timed Events Count
------------------------------------ ---------
log file parallel write 8
select CPU 7
event as "TOP 5 Timed Events", control file parallel write 3
cnt as "Count" from ( null event 3
select decode(ash.session_state, log file sync 2
'ON CPU','CPU',en.name) event ,
count(decode(ash.session_state,
'ON CPU','CPU',en.name)) cnt
from
v$active_session_history ash,
v$event_name en
where en.event# (+) = ash.event#
and SAMPLE_TIME > sysdate - (5/(24*60))
group by decode(ash.session_state,'ON CPU','CPU',en.name)
order by count(decode(ash.session_state,
'ON CPU','CPU',en.name)) desc
) where rownum < 6; Copyright 2006 Kyle Hailey
#.107
Querying ASH for P1,P2,P3

select
session_id,
event,
p1, p2, p3
SID EVENT P1 P2 P3 from v$ash
--- ----------------------- --- --- ---
236 db file scattered read 3 35291 16
248 read by other session 1 13646 1
245 db file sequential read Copyright
1 13591 1
2006 Kyle Hailey
#.108
ASH Report
ASH report
@?/rdbms/admin/ashrpt.sql
Pick interval over last 7 days !
1) General info 9) Top SQL using literals
2) Top User Events *** 10) Top Sessions ***
3) Top Background Events 11) Top Blocking Sessions
4) Top Event P1/P2/P3 Values 12) Top Sessions running PQs
5) Top Service/Module 13) Top DB Objects
6) Top Client IDs 14) Top DB Files
7) Top SQL Command Types 15) Top Latches
8) Top SQL Statements *** 16) Activity Over Time ***
#.109
V$session_wait
Moment in Time data
select
decode(w.wait_time, 0, w.event , 'CPU') as "TOP 5 Timed Events,
count(*)
from v$session s,
v$session_wait w
where w.sid=s.sid
and s.type='USER'
'rdbms ipc reply')
group by
decode(w.wait_time, 0, w.event , 'CPU')
order by count(*) desc; Copyright 2006 Kyle Hailey
#.110
V$session_wait
Top 5 Timed Events COUNT

----------------------------------- ----------
enq: US contention 4
CPU 3
buffer busy waits 1
latch: row cache objects 1

#.111
V$session_wait
col status for a35
select s.sid,
s.sql_hash_value,
w.p1, w.p2, w.p3
from v$session s,
v$session_wait w
where w.sid=s.sid
and s.type='USER'
'rdbms ipc reply');
#.112
V$session_wait
Moment in Time data
SID SQL_HASH STATUS P1 P2 P3
---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
248 3333220954 CPU 1650815232 1 0
260 1272059733 CPU 1431502854 44 0

#.113
Waits 1-9
2. log file sync
Commit less, put redo logs on faster disks
FTS - Tune SQL, add indexes, speed up disks (5-15ms)
4. latch free need p1,p2,p3 and sometimes SQL
5. enqueue need p1,p2,p3 and SQL
6. SQL*Net more data from client
Usually OK, reduce data transferred, possible Network problems
7. direct path read
9. buffer busy waits need p1,p2,p3
#.114
Waits 10-18
10. SQL*Net more data to client
Usually OK, reduce amount of data transferred, possible Network tuning needed
11. log buffer space
Increase log buffer
12. log file switch completion
13. library cache pin
Reduce parsing the same cursor concurently
14. SQL*Net break/reset to client
Check for errors in sql statement
15. io done
Io done by IO slaves, oracleIxxx could also be an idle wait sort of worthless
16. file open
Reduce logons/logoffs or keep log files open
17. free buffer waits
18. db file parallel read

#.115
Waits 19-27
19. write complete waits
20. library cache lock
21. SQL*Net more data from dblink
Reduce data transfer, check net response
22. log file switch (checkpoint incomplete)
23. library cache load lock
24. row cache lock
need p1 & SQL
25. local write wait
truncates / reduce cache size
26. sort segment request
SMON busy, process rollback problem
27. process startup
If MTS, increase min servers and dispatchers in init.ora
#.116
Waits 28-36
28. file identify
Keep log files open, reduce checkpoints
29. pipe put
Speed up pipe readers
30. switch logfile command
31. SQL*Net break/reset to dblink
Check for errors in sql statement sent
32. log file switch (archiving needed)
33. Wait for a undo record
??
35. undo segment extension
Use UNDO or with RBS, increase RBS size, avoid OPTIMAL
36. undo segment tx slot
Use UNDO, increae # of RBS segs
#.117
Custom Collecting
collecting:
ash.collect(sleep,loops)
save data in "ash_data" table
sleep = wait time between loops
loops = # of loops
Debug or testing:
ash.print(sleep,loops)
prints with dbms_output
to see output, run
set serveroutput on
execute dbms_output.enable(1000000)

#.118
How much data to keep

internally
max_rows number := 10000;
sets max rows saved in each of 3 tables
in view
ash_data_v create view ash_data_v as
select * from ash_data_1
union all
select * from ash_data_2
union all
select * from ash_data_3;
#.119
Sampling ASH Data

Foreground and Background
Waiting and CPU
Mimics main Fields of ASH

#.120
Select
Sampling
to_char(sysdate,'SSSSS')+
trunc(sysdate-to_date('JAN-01-1970 00:00:00','MON-DD-YYYY HH24:MI:SS'))*86400 ,
sysdate,
s.indx "SID",
decode(w.ksusstim,
0,decode(n.kslednam,
'db file sequential read', 'I/O',
'db file scattered read','I/O',
'WAITING'),
'CPU') "STATE",
s.ksuseser "SERIAL#",
s.ksuudlui "USER#",
s.ksusesql "SQL_ADDRESS",
s.ksusesqh "SQL_HASH_VALUE" ,
s.ksuudoct "COMMAND" /* aka SQL_OPCODE */,
s.ksuseflg "SESSION_TYPE" ,
w.ksussopc "EVENT# ",
w.ksussseq "SEQ#" /* xksuse.ksuseseq */,
w.ksussp1 "P1" /* xksuse.ksusep1 */,
w.ksusstim "WAIT_TIME" /* xksuse.ksusetim */,
s.ksuseobj "ROW_WAIT_OBJ#",
s.ksusefil "ROW_WAIT_FILE#",
s.ksuseblk "ROW_WAIT_BLOCK#",
s.ksusepnm "PROGRAM",
s.ksuseaph "MODULE_HASH", /* ASH collects string */
#.121
Sampling
from
x$ksuse s ,
x$ksusecst w,
x$ksled n
where
s.indx != ( select distinct sid from v$mystat ) and
bitand(s.ksspaflg,1)!=0 and
bitand(s.ksuseflg,1)!=0 and
n.indx=w.ksussopc and
s.indx = w.indx and
( (
/* status Active - seems inactive & "on cpu"=> not on CPU */
w.ksusstim != 0 and /* on CPU */
bitand(s.ksuseidl,11)=1 /* ACTIVE */
)
or
w.ksussopc not in /* waiting and the wait event is not idle */

#.122
Sampling
Select event#
from
v$event_name
where
lower(name) in (
'null event',
'pl/sql lock timer',
'px deq: execution msg',
'px deq: table q normal',
'px idle wait',
'sql*net message from client',
'sql*net message from dblink',
'dispatcher timer',
'lock manager wait for remote message',
'pipe get',
'pmon timer',
'queue messages',
'rdbms ipc message',
'slave wait',
'smon timer',
'virtual circuit status',
'wakeup time manager',
'i/o slave wait',
'jobq slave wait',
'SQL*Net message from client'
)
) Copyright 2006 Kyle Hailey
#.123

Waits Events

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Waits Events

Diunggah oleh

Hak Cipta:

Format Tersedia

Waits Events

Copyright 2006 Kyle Hailey

And the Geeks Who Love Them

Copyright 2006 Kyle Hailey

Waits Cheat Sheet

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Guilty until proven innocent

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Redo Lib Buffer

Copyright 2006 Kyle Hailey

Wouldnt it be great if Applications took some

Copyright 2006 Kyle Hailey

Statspack Cheat Sheet:

Copyright 2006 Kyle Hailey

Top 5 Timed Events

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

NAME TYPE VALUE

DB Name DB Id Instance Inst Num Release RAC Host

Snap Id Snap Time Sessions Curs/Sess

Copyright 2006 Kyle Hailey

2 CPU * ( 1.0 min * (60 sec/min) )

= 120 secs CPU time available

Elapsed: 1.00 (mins)

Copyright 2006 Kyle Hailey

Used CPU Time and Wait Time

Top 5 Timed Events

Copyright 2006 Kyle Hailey

Dependable Tuning Strategy

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host

Snap Id Snap Time Sessions Curs/Sess

Copyright 2006 Kyle Hailey

Top 5 Timed Events

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

OEM 10g Zoom-In

OEM 10g Zoom-In

Relax Get to Work!

Copyright 2006 Kyle Hailey

OEM 10g Zoom-In

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

12 Wait events represent 80%

Copyright 2006 Kyle Hailey

Above is over 80% of wait times reported

Anjo Kolk www.oraperf.com

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

REDO Lib Buffer

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

REDO Lib Buffer

Copyright 2006 Kyle Hailey

1 db file sequential read

REDO Lib Buffer

Copyright 2006 Kyle Hailey