In RAC We Trust

ORACLE - From Dream To Production

BGOUG Gabrovo 22.04.2005

,, Let someone k n o w

Plamen Zyumbyulev

Presentation Goals
Describe the major steps in RAC implementation
Show and explain the main problems and obstacles
Show and explain solution and workarounds
Give some practical ideas and variations
Try to look at the problems from different angles
Non-Goals
Explain RAC concepts and fundamentals
Show ALL aspects of RAC

Agenda
Introduction
RAC Installation
High Availability Configuring
Testing and Tuning RAC
Implementing RAC in production

Introduction

RAC Installation
Problems
Solutions

Problems
Need to evaluate and purchase cluster hardware
Need of knowledgeable OS, High Availability,
Network and Storage professionals
All this takes time

Solutions
RAC on Single Node
RAC on Single VMware Node
RAC on Multiple VMware Nodes
RAC and Network Block Device
Other Solutions
All of the solutions presented here are for testing
purposes ONLY. These configurations are not
certified or supported by Oracle Support Services

RAC on Single Node

Why not?
Metalink Note:241114.1 - Step-By-Step Installation of
RAC on Linux - Single Node (Oracle9i 9.2.0 with OCFS)

Key Points

No need of fencing configuration

No need of using clustered file system or raw devices
No need of multiple oracle homes (ORACLE_HOME)

RAC on Single Node (contd)

Key Points (contd)

One oracle user with 2 or more profiles one for every

instance. (e.g. .rac1, .rac2, )
zyumix:/# su - oracle
oracle@zyumix:~$ . rac1
oracle@zyumix:~$ echo $ORACLE_SID
rac1
zyumix:/# su - oracle
oracle@zyumix:~$ . rac2
oracle@zyumix:~$ echo $ORACLE_SID
rac2

RAC on Single Node (contd)

Key Points (contd)

Oracle Universal Installer needs Clusterware software in

order to install RAC option.

Disadvantages

RAC on Single Node (contd)

Client side
load balancing
Listener
A

Listener
B

Server side
load balancing

Server A
Instance A

Server B
Instance B

Database

RAC on Single VMware Node

Even easier !!!
The Oracle-on-Linux VMware Cookbook -

http://www.oracle.com/technology/tech/linux/vmware/cookbook/index.html

An easy, hands-on, step-by-step guide describing how-to

install VMware, Linux (RHEL/SLES) and Oracle RAC
(again on single node)
VMware Workstation (90-day free eval; registration required)
RHEL3 (not free) , SLES8 (not free)

Disadvantages

RAC on Multiple VMware Nodes

VMware GSX/ESX Server permits the sharing of plain disks
with multiple virtual machines running on the same host,
provided the disk in question is a SCSI disk.
This approach is very powerful but complex. You can create
very complex environments multiple NICs, switches, disks
etc.
Now there are a number of nodes although virtual
Disadvantages

RAC and Network Block Device

This solution allows you to build scalable and high available
database system only with common Intel PCs connected into
Ethernet network.
In this solution, a standard shared disk subsystem is replaced
by a native Linux technology - Network Block Device
(NBD) that maps remote files to local block devices (e.g.
/dev/nb0) via TCP/IP network. One computer (not
necessarily Linux machine) serves as data storage for all
cluster nodes (Linux machines) instead of expensive disk
array.

RAC and Network Block Device

(contd)

With this thing compiled into the kernel, Linux can use a
remote server as one of its block devices. Every time the
client computer wants to read /dev/nd0, it will send a request
to the server via TCP, which will reply with the data
requested.
The remote resource doesn't need to be a whole disk or even
a partition. It can be a file.

RAC and Network Block Device

(contd)

Typical configuration

Simple NBD configuration

RAC and Network Block Device

(contd)

Installation
Both client and server machines are with RHEL3

Download source from http://nbd.sourceforge.net/

As root do

bunzip2 nbd-2.7.3.tar.bz2
tar -xvf nbd-2.7.3.tar
cd nbd-2.7.3
./configure
make
make install

RAC and Network Block Device

(contd)

Creating new empty files at NBD server

[root@rac2 root]# dd if=/dev/zero
of=/u01/oradata/rac/system.01 count=300 bs=1M
300+0 records in
300+0 records out
314572800 bytes transferred in 1.683993 seconds (186801738
bytes/sec)
[root@rac2 root]#

Running NBD server

Syntax: nbd-server <port> <filename>
[root@rac2 root]# nbd-server 4101 /u01/oradata/rac/system.01
[root@rac2 root]#

RAC and Network Block Device

(contd)

NBD client
NBD client must be run as root (because of kernel parts of
NBD). Before starting NBD client you would have to install
Linux kernel NBD module

Installing ndb module RHEL3

[root@rac3 root]# rpm -Uvh kernel-unsupported-2.4.214.EL.i686.rpm
warning: kernel-unsupported-2.4.21-4.EL.i686.rpm: V3 DSA
signature: NOKEY, key ID db42a60e
Preparing...
########################################### [100%]
1:kernel-unsupported
########################################### [100%]
[root@rac3 root]#

RAC and Network Block Device

(contd)

NBD client (contd)

Loading nbd module

[root@rac3
[root@rac3
[root@rac3
nbd
[root@rac3

dev]# lsmod | grep nbd

dev]# modprobe nbd
dev]# lsmod | grep nbd
16388
0 (unused)
dev]#

running nbd client

Syntax: nbd-client <data server> <port> /dev/nb<n>

[root@rac3 dev]# nbd-client rac2 4101 /dev/nb0
Negotiation: ..size = 307200KB
bs=1024, sz=307200
[root@rac3 dev]#

RAC and Network Block Device

(contd)

Now block devices are configured and it is possible to access

remote data. Oracle Real application clusters need raw access
to shared disk subsystem so mapping raw devices to block
devices is needed. This could by done with standard raw
command.
Syntax: raw /dev/raw/raw<N> /dev/<blockdev>
[root@rac3 root]# raw /dev/raw/raw1 /dev/nb0
/dev/raw/raw1: bound to major 43, minor 0
[root@rac3 root]#

Other Solutions

RAC and FireWire

Build

Your Own Oracle RAC 10g Cluster on Linux and

FireWire

http://www.oracle.com/technology/pub/articles/hunter_rac10g.html

RAC and NFS

Locking
Caching
Write

through cache

HA Configuration

Simplified RAC schema

NET

Database

HA System

Maximum Availability
Architecture
Application Server

Application Server

Dedicated Network

Data Guard

Primary Site

Secondary Site

Extended Distance Clusters

Application Server

Application Server

RAC
Dedicated Network

Virtualization storage layer

What about the Quorum Server???
Primary Site

Secondary Site

Extended Distance Clusters (contd)

Resolving Distance Problems

Application partitioning

gc_files_to_locks

Wavelength Division Multiplexing

Dense Wavelength Division Multiplexing DWDM
Coarse Wavelength Division Multiplexing CWDM

ACTIVE_INSTANCE_COUNT
*.active_instance_count = 1
*.cluster_database_instances = 2

Testing and Tuning RAC

Introduction
RAC testing steps
Functional Application Tests
RAC High Availability tests
Scalability tests

Digging into RAC performance problems

Introduction
Testing isnt trivial !!!
Classical testing/tuning methods.

Always tune single instance first!

Specific RAC issues

RAC aware tools

RAC testing steps

Functional Application Tests
RAC High Availability tests

Be aware about the timeouts!!!

Scalability tests

RAC testing steps (contd)

Scalability tests (contd)
Patterns of application scalability
Performance (TPS, 1/response time)
of one user

nearly static
linear
exponential

constrained

Load ( # users, size of tables)

RAC testing steps (contd)

Scalability tests (contd)
Performance -TPS (for all users)

Good scalability

2 node RAC
Single node

# of concurrent users

RAC testing steps (contd)

Scalability tests (contd)
Possible disk bottleneck

Performance -TPS (for all users)

Problem!!!

2 node RAC
Single node

# of concurrent users

RAC testing steps (contd)

Scalability tests (contd)
Possible interconnect bottleneck

Performance -TPS (for all users)

Problem!!!

Single node
2 node RAC

# of concurrent users

Digging into RAC performance

problems
Interconnect and shared storage are the two most
possible performance problem areas in RAC
Interconnect speed

Throughput
Latency

average latency of a consistent block request. AVG CR BLOCK

RECEIVE TIME should typically be about 15 milliseconds
depending -- on your system configuration and volume

Digging into RAC performance

problems (contd)
Interconnect types
Measureme
nt

SMP
BUS

Memory
Channel

Myrinet Sun SCI Gigabit Hyper

Ethernet Fabric

Infiniband

Latency
(s)

0.5

7 to 9

10

100

20

< 10

CPU
overhead

low

low

low

N/A

high

low

low

Bandwidth
MB/sec

> 500 > 100

~250

~ 70

100

400

1000

Digging into RAC performance

problems (contd)
cluster_interconnects parameter
It provides Oracle with information about additional cluster
interconnects available for use and can be used to load balance
the interconnect traffic to different physical interconnects thus
increasing interconnect bandwith.
When you set CLUSTER_INTERCONNECTS in cluster
configurations, the interconnect high availability features are not
available. In other words, an interconnect failure that is normally
unnoticeable would instead cause an Oracle cluster failure as
Oracle still attempts to access the network interface which has
gone down.

Digging into RAC performance

problems (contd)
STATSPACK reports

The STATSPACK report show statistics ONLY for the

node or instance on which it was run
Top 5 Timed Events
Global Cache Service and Global Enqueue Service

Note:135714.1 Script to Collect RAC Diagnostic

Information (racdiag.sql)

Digging into RAC performance

problems (contd)
BAD PERFORMANCE
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Ela Time
--------------------------------------- ------------ ----------- -------global cache cr request
34,568
958
31.44
buffer busy global CR
6,513
620
20.35
db file sequential read
64,214
455
14.92
latch free
13,542
453
14.88
buffer busy waits
10,971
295
9.69

GOOD PERFORMANCE
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Ela Time
--------------------------------------- ------------ ----------- -------latch free
10,969
666
51.28
buffer busy waits
15,379
159
12.23
CPU time
149
11.47
PL/SQL lock timer
51
105
8.05
db file sequential read
25,163
96
7.36

Digging into RAC performance

problems (contd)
Global Cache Service - Workload Characteristics
----------------------------------------------Ave global cache get time (ms):
Ave global cache convert time (ms):
Ave build time for CR block (ms):
Ave flush time for CR block (ms):
Ave send time for CR block (ms):
Ave time to process CR block request (ms):
Ave receive time for CR block (ms):
Ave pin time for current block (ms):
Ave flush time for current block (ms):
Ave send time for current block (ms):
Ave time to process current block request (ms):
Ave receive time for current block (ms):
Global cache hit ratio:
Ratio of current block defers:
% of messages sent for buffer gets:
% of remote buffer gets:
Ratio of I/O for coherence:
Ratio of local vs remote work:
Ratio of fusion vs physical writes:

BAD

GOOD

11.8
51.7
0.7
0.2
0.0
0.9
1.6
0.2
0.0
0.1
0.3
33.4
9.5
0.0
6.7
1.8
1.2
4.3
0.0

2.2
11.2
0.0
0.2
0.2
0.4
0.4
0.2
0.0
0.1
0.3
7.5
3.9
0.0
2.5
0.7
1.3
4.4
0.0

Digging into RAC performance

problems (contd)
Global Enqueue Service Statistics
--------------------------------Ave global lock get time (ms):
Ave global lock convert time (ms):
Ratio of global lock gets vs global lock releases:
GCS and GES Messaging statistics
-------------------------------Ave message sent queue time (ms):
Ave message sent queue time on ksxp (ms):
Ave message received queue time (ms):
Ave GCS message process time (ms):
Ave GES message process time (ms):
% of direct sent messages:
% of indirect sent messages:
% of flow controlled messages:

BAD

GOOD

0.2
0.0
1.2

0.0
0.0
1.1

16.5
29.4
1.9
0.1
0.1
49.7
50.3
0.0

1.7
2.5
0.3
0.1
0.0
63.4
36.6
0.0

Implementing RAC in production

Smooth transition from single instance to RAC

Change ORACLE_HOME

Relinking the RAC Option ON/OFF

CLUSTER_DATABASE = TRUE/FALSE

Start/Stop the second instance

Start gradual movement of clients from one

instance to another

Relinking the RAC Option

1.

Login as the Oracle software owner and shutdown all

database instances on all nodes in the cluster.

2. cd $ORACLE_HOME/rdbms/lib
3. make -f ins_rdbms.mk rac_on (rac_off)

If this step did not fail with fatal errors then proceed to step
4.
4. make -f ins_rdbms.mk ioracle

Reference

Metalink Note:211177.1 RAC Survival Kit: Rac On / Rac Off - Relinking the
RAC Option

Metalink Note:183340.1 Frequently Asked Questions About the

CLUSTER_INTERCONNECTS Parameter in 9i.

http://www.fi.muni.cz/~kripac/orac-nbd/

The Oracle-on-Linux VMware Cookbook

http://www.oracle.com/technology/tech/linux/vmware/cookbook/index.html

Build Your Own Oracle RAC 10g Cluster on Linux and FireWire
http://www.oracle.com/technology/pub/articles/hunter_rac10g.html
Note:135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)

Thank You

zyumbyulev@mobiltel.bg