Performance on LINUX
OOUG 2009
Columbus, OH
July 16th
© 2008 Quest Software, Inc. ALL RIGHTS RESERVED.
Agenda
• Apply “low hanging fruit” fixes to speed up Oracle
performance on Linux
Oracle Background:
• Worked with Oracle databases for over two decades (starting with version 4)
• Work history includes time at both “Oracle Education” and “Oracle Consulting”
Academic Background:
• Several Oracle Masters certifications
• BS, MS and PhD in Computer Science
• MBA (general business)
• Several insurance industry designations
Key Interests:
• Data Modeling
• Database Benchmarking
• Database Tuning & Optimization
• "Star Schema" Data Warehouses
• Oracle on Linux – and specifically: RAC on Linux
Coming in 2009 …
4
Couple of Questions (for both our benefit)
#1 OS !!!
www.netcraft.com/survey/index-200007.html
LINUX Web Popularity
http://survey.netcraft.com/index-200106.html
LINUX Web Popularity
http://news.netcraft.com/archives/2003/07/index.html
LINUX Gaining Momentum
LINUX Gaining Momentum
http://www.gartner.com/DisplayDocument?ref=g_search&id=414985
Popularity != Performance
What is Tunable:
• Hardware
Linux Tuning
• Operating System
• Database
• Network
SQL Tuning
• Application
Network
Hardware
OS
DBMS
Application
Our Main Goal
Simulate
200
Users
Carpenter Needs Tools
Between the hardware, LINUX and Oracle – there
are far too many dependent variables for tuning to
rely merely on human intuition and experience.
Free is nice – but you often get what you pay for!
Benchmark Factory
• Create, populate and index the test database (200 megs)
• Simulate 200 concurrent users via a single or many PC’s
LINUX Freebies
• Command line utilities: sar, mpstat, iostat, vmstat, linmon, ipcs,
top, free, hdparm, linuxconf, slmon, …
• X-Windows utilities: gtop, ktop, xload, xosview, kperfmeter,
gkrellmm, procmeter, gpowertweak, …
Benchmark Factory - GUI
Benchmark Factory - Agent
LINUX cmd tools
sar –r
01:00:00 PM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached kbswpfree kbswpused %swpused
01:10:00 PM 465132 48128 9.37 0 2708 26136 525288 0 0.00
01:20:00 PM 463352 49908 9.72 0 2784 26732 525288 0 0.00
01:30:00 PM 463356 49904 9.72 0 2784 26736 525288 0 0.00
01:40:03 PM 1652 511608 99.67 0 532 18216 447360 77928 14.83
01:50:01 PM 1604 511656 99.68 0 768 17228 369024 156264 29.74
02:00:00 PM 376852 136408 26.57 0 1120 25692 503344 21944 4.17
vmstat
SGA Buffer Cache = 64M Prior to 9i, the tool’s default settings were
ridiculously lower than these. A novice DBA
or system admin user might use those lower
SGA Shared Pool = 64M default values and get much worse results!
Tablespaces = Dictionary
DB2 – Increase Buffer Cache & Shared Pool
Tablespaces = Dictionary
DB3 – Increase Redo Cache & Log Files
Tablespaces = Dictionary
DB4 – 4K Block Size
Tablespaces = Dictionary
DB5 – Local Tablespaces
Tablespaces = Local
DB6 – 8K Block Size
Tablespaces = Local
DB7 – IO Slaves + Increase Redo Log Files
Tablespaces = Local
Oracle Modification Results
Kernel = 2.2.14-5smp
Monolithic Kernel - If you compile
Linuxconf = monolithic everything into the kernel to exactly
match your hardware and thus
make minimal use of modules.
Shared memory
/usr/src/linux-2.2.16/include/asm/shmparam.h
#define SHMMAX 0x13000000
Semaphors
/usr/src/linux-2.2.16/include/linux/sem.h
#define SEMMNI 100
#define SEMMSL 512
#define SEMMNS (SEMMNI*SEMMSL)
#define SEMOPM 100
#define SEMVMX 32767
OS2 – Newer Minor Kernel Update
Kernel = 2.2.16-3smp
Linuxconf = monolithic
Shared memory
/usr/src/linux-2.2.16/include/asm/shmparam.h
#define SHMMAX 0x13000000
Semaphors
/usr/src/linux-2.2.16/include/linux/sem.h
#define SEMMNI 100
#define SEMMSL 512
#define SEMMNS (SEMMNI*SEMMSL)
#define SEMOPM 100
#define SEMVMX 32767
OS3 – Newer Major Kernel Update
kernel.shmmax = 2147483647
Linuxconf = monolithic kernel.sem = 250 32000 100 128
kernel.shmmax = 2147483647
Linuxconf = monolithic kernel.sem = 250 32000 100 128
The ext2 file system normally records when a file Actually, this can be done on
was last modified and last accessed. We don’t need Windows server as well!
to know access time for Oracle files as background
programs open and access the files until shutdown.
HKEY_LOCAL_MACHINE\Sy
chattr +A file_name stem\CurrentControlSet\Control\
FileSystem
chattr –R +A directory_name
The bdflush file is closely related to the operation of 1st parm is max # of dirty buffers in
the virtual memory (VM) subsystem of the Linux cache. Higher = delayed disk writes.
kernel and also has a little influence on disk usage. 2nd parm is max # of dirty buffers per
write. Higher = delayed, bursty I/O.
Default = "40 500 64 256 500 3000 500 1884 2" 3rd parm is # of buffers added to
list of free buffers by refill_freelist.
Redhat 6.1
4th parm refill_freelist comes
echo “100 1200 128 512 15 500 1884 2” > /proc/sys/vm/bdflush across more than nref_dirt dirty
buffers, it will wake up bdflush.
Redhat 6.2 (edit /etc/sysctl.conf)
5th parm is max time Linux waits
vm.bdflush = 100 1200 128 512 15 5000 500 1884 2 before writing dirty buffers to disk
for data blocks.
Restart daemon 6th parm is max time Linux waits
/etc/rc.d/init.d/network restart
before writing dirty buffers to disk
for file system metadata.
OS7 – Ext3 File System
Ext3 is a journaling file system based upon ext2 – benefits include:
DB1 DB Final OS1 OS2 OS3 OS4 OS5 OS6 OS7 OS Final Total
load time 49.41 10.48 9.54 9.40 8.32 8.20 5.58 4.43 3.80 3.80 3.80
% Improved -371.47% -9.85% -1.49% -12.98% -1.46% -46.95% -25.96% -16.58% -151.05% -1200.26%
trans/sec 8.15 10.72 11.51 11.52 12.82 12.90 13.88 14.99 20.51 20.51 20.51
% Improved 23.93% 6.90% 0.10% 10.09% 0.66% 7.09% 7.37% 26.92% 43.88% 60.25%
Some Final Thoughts
IDE Drives
• Default IDE settings stink – they must be changed and generally yield 500% improvement!!!
• Test speed via hdparm –Tt /dev/hda
• Tune spped via hdparm –X66 –d1 –u1 –m16 –c3 /dev/hda
MTS
• Problematic under LINUX, more than a few TAR’s registered with Oracle technical support
Hardware RAID
• LINUX driver support generally tier 2 and often provided by someone other than the vendor
4K File System
• mke2fs –b 4096 mount_point
• LINUX default file system block size = 1K, but max is 4K
• Typical improvement of 5% for each increase (i.e. 5% for 1k–>2k and 5% for 2k–>4k)
PGCC Compiler
• www.goof.com/pcg
• PGCC is a version of the GNU C compiler (gcc) with special optimizations for Pentium CPU
• The 30% improvement achieved by the Intel engineers is rare in the real world (don’t bank on it)
• Typical improvement of 5% using intermediate optimization level like -O3 and no fine-tuning
• Recompile kernel – only small improvement since Oracle makes few calls to LINUX kernel
• Recompile glibc – bigger improvement possible, but headache unless already know process
Questions and Answers …
Thank You
Presenters:
Bert Scalzo: Bert.Scalzo@Quest.com
www.toadworld.com/Experts/BertScalzosToadFanaticism/tabid/318/Default.aspx