Ian Jones
Database Specialists,
Inc.
www.dbspecialists.com
1
Session Topics
Statspack introduction and features
Mechanics
– installing
– generating snapshots
– producing reports
Discussion of the generic report
Examples
2
Session Topics
Statspack introduction and features
Mechanics
– installing
– generating snapshots
– producing reports
Discussion of the generic report
Examples
3
What is Statspack?
An Oracle provided set of SQL*Plus scripts
and a PL/SQL package that allows the
convenient collection, automation, storage
and reporting of performance and diagnostic
data
A PERFSTAT schema containing 42 ‘stats$’
tables and a PL/SQL package ‘statspack’
Replacement for utlbstat/utlestat
4
Overview of How Statspack
Works
Oracle instances constantly update lots of internal
statistics, most visible through the v$ views e.g.
system statistics, wait events and SQL activity, etc
(timed_statistics, resource_limit, 9i statistics_level)
Using ‘statspack.snap’ we save away these values
from 34 v$ views into stats$ tables when desired
Then we run the statspack report script
‘spreport.sql’ which calculates and displays the
differences between any two sets of statistics
Straightforward and effective
5
What Questions Can
Statspack Answer?
What work load is the database under now?
What activities/events are we waiting for?
Which SQL is consuming most resources?
Which segments are most problematic?
Where is the I/O, and are we CPU bound?
How does all this compare with earlier data?
Statspack provides diagnostic data
to solve problems. 6
Why Use Statspack?
Simple and quick to install and use
Provided with all editions version 8.1.6+
Written by Oracle - in sync with RDBMS
Small system overhead (varies with level)
Source code is available for review
Snapshot data held in tables and available
for historical or custom analysis
7
Replacement For
(utl)bstat/estat
Statspack has an improved design over bstat/estat
– Flexible reporting because data held in tables
– Different levels of data collection
– User defined thresholds
Wider range of data
– SQL statements
– Wait events
– Segment statistics (9.2)
Bstat/estat not updated with new features
8
Statspack Main Files
Set of 19 files named sp* (stat* in 8.1.6)
located in $ORACLE_HOME/rdbms/admin
spdoc.txt – Good description of mechanics
spcreate.sql – Sqlplus installation script
spreport.sql – Generic reporting script
sprepsql.sql – Explain plan report script
spauto.sql – Creates dbms_job to automate
data collection (job_queue_processes>0)
9
Session Topics
Statspack introduction and features
Mechanics
– installing
– generating snapshots
– producing reports
Discussion of the generic report
Examples
10
Installation
Run the ‘spcreate.sql’ script using SQL*Plus as
user SYS. User PERFSTAT is created by this
script, owning all objects needed by the statspack
package.
E.g. On Unix:
cd $ORACLE_HOME/rdbms/admin
sqlplus “/ as sysdba” @spcreate.sql
To set up automatic collection of data every hour:
cd $ORACLE_HOME/rdbms/admin
sqlplus perfstat/<pwd> @spauto.sql
11
Snapshots
A single set of performance data captured
using the statspack PL/SQL package:
Begin
perfstat.statspack.snap(i_snap_level=>6);
End;
Different snapshot levels determine data captured:
Level = 0 General performance statistics (8i,9i)
Level = 5 SQL Statements (default) (8i,9i)
Level = 6 SQL Plans (9i)
Level = 7 Segment statistics (9.2)
Level = 10 Parent and Child latches (8i,9i)
12
Generic Report
(spreport.sql)
Generates a report between any two
snapshots as long as the instance was not
restarted between the snapshots
sqlplus perfstat/<pwd> @spreport.sql
Enter the start and end snapshot id’s and
optionally enter the output file name (or
accept the default sp_<b>_<e>.lst)
13
Session Topics
Statspack introduction and features
Mechanics
– installing
– generating snapshots
– producing reports
Discussion of the generic report
Examples
14
Sections of the Generic
Context Report 0
Cache Sizes
0
Load Profile 0
Instance Efficiency 0
Timed/Wait Events (renamed now includes CPU time) 0
SQL (Buffer Gets/Disk Reads/Executions/Parses) 5
Instance Statistics 0
Tablespace and Datafile IO 0
Buffer Pool Statistics 0
Rollback Activity 0
Latch Statistics 0,10
Segment Statistics (introduced in 9.2) 7
Library Cache Statistics 0
SGA Pool Breakdown 0 15
Instance Parameters 0
Context/Cache Sizes
DB Name DB Id Instance Inst Num Release Cluster Host
------- -------- -------- -------- --------- ------- -------
HAW1 39997887 haw1 1 9.2.0.1.0 NO HAWKING
23
Wait Events - Comments
A very important diagnostic provided by Oracle.
The major ‘jumping off point’ if the elapsed times
are a significant proportion of the interval time (i.e.
if most of the time is not spent in idle waits)
See Reference Guide for details of each wait
Common I/O related waits:-
Db file sequential read – Index reads or scans
Db file scattered read – Full table scans
Direct path read/write – Temp IO
Log related waits - IO, switches, buffer
24
Wait Events – Where to
Jump?
Db file * read ->SQL by buffer gets/disk reads,
File IO stats
CPU Time -> Parse rates, Sorts, SQL executions,
SQL buffer gets/disk reads, SMP processes(bugs)
Direct path reads/writes -> Sorts, Hash joins,
hash/sort_area_size, File IO Stats
Buffer busy waits -> Buffer pool, Buffer waits,
File IO stats, Segment statistics
Other important wait events (e.g. latches,
enqueues) have corresponding statspack sections
to themselves 25
SQL Section
Four sections of “worst SQL” ranked by
buffer gets, disk reads, executions, parse
counts.
SQL ordered by Gets for DB: HAW1 Instance: haw1 Snaps: 117 -118
CPU Elapsd
Buffer Gets Execs Gets per Exec %Total Time(s) Time(s) Hash Value
----------- ----- ------------- ------ ------- ------- ----------
13,192 1 13,192 74.2 1.83 8.76 3097336866
Module: SQL*Plus
SELECT * FROM policies WHERE policy_type = :b1
26
SQL Section - Comments
Sub optimal SQL is the most common source of
database problems. “Can we get the same results
by consuming fewer resources?”
SQL ranked by total numbers, often the ‘number
per execution’ is more useful
What is our current execution plan and has it
changed recently? Second statspack report
available (9i, level >= 6)
sqlplus perfstat/<pwd> @sprepsql.sql
This report provides breakdown across snapshots
based on SQL hash value. Reveals changing
execution plans (see later example) 27
Segment Statistics
Historically difficult to isolate segment specific
data, new 9.2 view v$segstat greatly simplifies this
Tablespace Filename
---------- -------------------------------------------------
Av Av Av Av Buffer Av Buf
Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
------- ------- ------ ------- ------ -------- ------ ------
PAY_6 /u01/oradata/payroll/PAY_6_1.dbf
438,860 638 4.8 7.4 10 0 5,750 9.7
30
Buffer Pool and Buffer
Waits
Buffer Pool Statistics for DB: NETMON Instance: netmon
-> Pools D: default pool, K: keep pool, R: recycle pool
Free Write Buffer
Buffer Consistent Physical Physical Buffer Complete Busy
P Gets Gets Reads Writes Waits Waits Waits
- --------- ---------- --------- -------- ------ -------- --------
D 4,859,734 4,765,667 4,755,716 1,740 0 4 8,333
------------------------------------------------------------------
32
Latches
Latch Activity for DB: Pct Avg
Pct
Get Get Slps NoWait
NoWait
Latch Name Requests Miss /Miss Requests
Miss
----------------------- --------- ---- ----- ---------
------
cache buffers lru chain 4,925,313 4.3 0.2 4,749,919
4.4
------------------------------------------------------------
-
Latch Sleep breakdown for DB
-> ordered by misses desc
35
Examples
Monitoring Madness
Out of Sorts
Distributed SQL
Changing Plans
Freelists and 9i Auto Managed Segment
36
Example #1: Monitoring
Madness
A previously stable system, a third party
monitoring package, is suddenly consuming large
amounts of CPU time. The Unix administrators
want to know if they should kill these ‘out of
control’ Oracle processes
Snap Id Snap Time Sessions
------- ------------------ --------
Begin Snap: 2503 07-Aug-02 12:20:24 33
End Snap: 2512 07-Aug-02 12:31:52 33
Elapsed: 11.47 (mins)
37
Example #1: Load Profile
Per Second Per Transaction
Redo size: 7,734.59 8,737.93
Logical reads: 7,168.02 8,097.87
Block changes: 31.11 35.15
Physical reads: 6,916.97 7,814.25
Physical writes: 3.04 3.43
User calls: 21.54 24.34
Parses: 2.72 3.07
Hard parses: 1.08 1.22
Sorts: 1.75 1.98
Logons: 0.04 0.04
Executes: 16.15 18.24
Transactions: 0.89
% Blocks changed per Read: 0.43 Recursive Call %: 60.07
Rollback per trans %: 0.49 Rows per Sort: 13.08
38
Example #1: Wait Events
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.83 Redo NoWait %: 99.99
Buffer Hit %: 3.50 In-memory Sort %: 99.33
Library Hit %: 91.25 Soft Parse %: 60.11
Execute to Parse %: 83.17 Latch Hit %: 98.74
Parse CPU/Parse Elapsd %: 73.37 % Non-Parse CPU: 100.00
40
Example #1: Comments
Execution plan of offending statement
SELECT STATEMENT Hint=CHOOSE
SORT UNIQUE
UNION-ALL
INDEX UNIQUE SCAN SYS_C001289
TABLE ACCESS FULL NWT_ACT_MESSAGES
INDEX UNIQUE SCAN SYS_C001322
TABLE ACCESS FULL NWT _HIST_MESSAGES
New networking equipment and network problems
introduced over the weekend caused major flood
of messages
41
Example #2: Out of Sorts
Load Profile
Per Second Per Transaction
---------- ---------------
Logical reads: 101.25 14,884.00
Physical reads: 51.7 7,610.00
Physical writes: 51.7 7,610.00
Parses: 0.7 113.00
Tablespace IO Stats
Tablespace Reads Reads/s Writes Writes/s
---------- ------ ------- ------ --------
TEMP 3,155 21 7,610 52 43
Example #2: Conclusions
sort_area_size parameter was set to 8i
default value of 64k
Virtually all the I/O to TEMP tablespace
due to disk sorting, even though in
memory sorts were 96.65%
Increasing sort_area_size produced over
90% improvement in benchmark
performance
44
Example #3: Distributed
SQL
Users complaining of poor performance
Nothing strange in report (e.g. no bad SQL) except
47
Example #3: Conclusions
Search of (v$sql) based on previous fragment
identified the following statement on the primary
SELECT DISTINCT b.auth_role_code
FROM person@paw a, person_auth_roles@paw b,
access_roles@paw c, resources@paw d
WHERE upper(a.user_login) = upper(‘G243311')
AND a.person_id = b.person_id
AND b.auth_role_code = c.auth_role_code
AND c.resource_id = d.resource_id
AND upper(d.resource_id) IN
(SELECT upper(ga_resource_id)
FROM apps_mapping)
48
Example #3: Conclusions
Third party package (in remote database) is not
analyzed. This results in a poor distributed
execution plan
Rows Execution Plan
---------- ---------------------------------------------------------
0 SELECT STATEMENT GOAL: CHOOSE
110703 REMOTE [PAW.WORLD]
SELECT "RESOURCE_ID"
FROM "RESOURCES" "D"
WHERE :1= "RESOURCE_ID"
49
Example #4: Changing
Plans
A batch job that had previously performed
well was now taking much longer to run. A
conventional statspack report showed that a
particular statement was dominating the
resource usage. What has changed?
Begin
statspack.snap(I_snap_level=>6);
End;
sqlplus perfstat/<pwd> @sprepsql.sql
50
Example #4: Changing
Plans
Plans in shared pool between Begin and End Snap Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Shows the Execution Plans found in the shared pool between the begin
and end snapshots specified.
-------------------------------------------------------------------
Operation | PHV/Object Name |Rows|Bytes|Cost
-------------------------------------------------------------------
SELECT STATEMENT |----- 3101143917 ----| | | 1417
SORT ORDER BY | | 8K| 2M| 1417
TABLE ACCESS BY INDEX ROWID|MOVIE_REVIEWS | 8K| 2M| 481
INDEX RANGE SCAN |MOVIE_REVIEWS_I1 | 8K| | 23
SELECT STATEMENT |----- 3302467752 ----| | |19468
SORT ORDER BY | |211K| 12M|19468
TABLE ACCESS FULL |MOVIE_REVIEWS |211K| 12M| 4639
-------------------------------------------------------------------
51
Example #5: Hot Blocks
High rates of concurrent inserts cause busy
buffer waits. Lets analyze this using
statspack to illustrate enqueues & buffers
This example uses 20 processes running
concurrently each inserting 10,000 rows into
the same log table (9.0.1.3 rdbms)
9i Introduces new feature known as
‘Segment Management Auto’ to compare
our conventional results against
52
Example #5: Initial Results
Elapsed: 1.42 (mins) Buffer Nowait %: 74.34
Enqueue activity
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
-- -------- --------- ----------- ----- --------- --------
SQ 2,674 2,674 0 2,469 16.28 40
HW 3,123 3,123 0 1,209 163.19 197
55
Example #5: Seg Manage
Elapsed: 0.87 (mins) Auto Buffer Nowait %:
98.35
Total Avg
Class Waits Time(s) Time (ms)
------------------ ----- ------- ---------
data block 8,582 109 13
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
-- -------- --------- ----------- ----- --------- --------
56
SQ 2,669 2,669 0 2,466 16.41 40
References
Two Oracle whitepapers ‘Performance
Tuning With Statspack, Part I & II’
Ch 10, ‘Expert one-on-one Oracle’ Tom
Kyte
Statspack readme spdoc.txt
http://www.oraperf.com provides free
automated analysis of Statspack reports
57
Contact Information
Ian Jones
Database Specialists, Inc.
388 Market Street, Suite 400
San Francisco, CA 94111
Tel: 415/344-0500
Email: ijones@dbspecialists.com
Web: www.dbspecialists.com
58