Real Application Testing: Consolidated Database Replay Feature Applies to: Oracle Server - Enterprise Edition - Version: 11.2.0.2.

0 and above Information in this document applies to any platform. What is being announced? Oracle Database 11g introduced Real Application Testing database option to enable businesses to assess impact of system changes with minimal effort and proactively identify and remediate any issues that were identified in testing. Real Application Testing option is comprised of two components SQL Performance Analyzer (SPA) and Database Replay. SPA helps predict the impact of a system change on unit SQL response time and Database Replay provides the best load testing solution using real production workloads. Oracle Database Release 11.2.0.2.0 and above supports now supports "Consolidated Database Replay" a feature that helps business assess the impact of database and workload consolidation to a single database. Consolidated Database Replay supports execution of multiple replays on a single database. The goal of this note is to provide information on how to use the Consolidated Database Replay functionality. This note contains information on mandatory patches and also includes usage examples, a FAQ, and syntax documentation. Consolidated Database Replay allows replay of multiple captured production workloads from the same or different systems. The captured workload information includes concurrency, think time, and transaction dependencies, etc. The captured workloads can then be replayed on a single database maintaining the production characteristics so that any potential issues can be identified and remediated in test before deploying the change to production. This will help customers assess how their database consolidation is going to work and accurately size their infrastructure. Note: The use of Real Application Testing thru API/EM interfaces including the Consolidated Database Replay functionality requires licensing of Real Application Testing. Please check the Licensing Information manual on OTN for details.

What do you need to do? Apply the following mandatory patches to the replay system to use the Consolidated Database Replay feature. This information applies to all platforms except Windows. Replay database Version 11.2.0.2.0 What patch you need to apply? 13947480 Download Information Comments

One-off patch can be downloaded or requested from MOS

For capture side patches see Doc ID 560977.1 For Replay side you do not need additional patches from 560977.1. Patch 13947480 includes patches recommended from 560977.1

11.2.0.3.0

13947480

One-off patch can be downloaded or requested from MOS

For capture side patches see Doc ID 560977.1

Note 1: The consolidated database replay feature is only supported on Oracle Database 11.2.0.2.0 and higher. Note 2: Consolidated Database Replay functionality is only supported through API in 11.2.0.2.0 and 11.2.0.3.0. EM Cloud Control will support this functionality in a future release.

1) Step 1: Capture: Capturing a Database Workload. Please see Oracle documentation on how to capture a database workload. Consolidated Database Replay feature does not require any special steps for capturing a database workload, the steps for capturing the database workload are exactly the same as before for a single capture. Consolidated Database Replay can use multiple workload captures based on any of the following: Oracle Database Release 9.2.0.8.0 and above, and multiple platforms (OS). You can capture

on any number of databases and any or different platform supported for Oracle Database and do one single consolidated database replay starting with 11.2.0.2.0 Database on any Oracle Database supported platform. For example: You can capture workload on 9.2.0.8.0 HP-UX and 10.2.0.4.0 AIX. Then you can perform consolidated replay on Oracle Exadata 11.2.0.2. Test System Setup. Before a workload can be replayed, the application data state should be logically equivalent to that of the capture systems at the start time of each workload capture. This minimizes replay divergence during replay. Since we are consolidating different databases into one single database and server, you should restore each schema that was captured through an Oracle supported method such as Data pump export/import, RMAN, etc. For example, if you captured: 1) Workload from CRM schema on 10.2.0.4.0 on Solaris server 2) Workload from HR schema on 10.2.0.5.0 on Linux server 3) Workload from SALES schema on 11.2.0.2.0 on Solaris server and are doing a Consolidated Database Replay on 11.2.0.2.0 database, then you would restore the following on the 11.2.0.2 test database: 1) CRM schema into CRM schema 2) HR schema into HR schema 3) SALES schema into SALES schema You cannot restore everything into one single schema and use the Consolidated Database Replay. You cannot remap the schema to a different schema and use the Consolidated Database Replay 1) Preprocessing Consolidated Database Workload.

For Consolidated Replay, you have to preprocess each captured workload into its own directory. Please see Oracle documentation on how to preprocess a database workload. Consolidated Database Replay feature does not require any special steps for preprocessing a database workload, the steps for preprocessing the database workload is exactly the same as previously used before the release of Consolidated Replay. Do not mix different workloads into one directory. For example if you captured: 1) Workload from CRM schema on 10.2.0.4.0 on Solaris server 2) Workload from HR schema on 10.2.0.5.0 on Linux server 3) Workload from SALES schema on 11.2.0.2.0 on Solaris server and you are doing a Consolidated Database Replay on 11.2.0.2.0, then you would do the preprocessing on 11.2.0.2.0 as follows: A) CRM Workload

create or replace directory crm as '/u01/test/cap_crm'; ensure your captured workload is in directory /u01/test/cap_crm' execute dbms_workload_replay.process_capture('CRM');

B) HR Workload create or replace directory hr as '/u01/test/cap_hr'; ensure your captured workload is in directory /u01/test/cap_hr' execute dbms_workload_replay.process_capture(HR');

C) SALES Workload create or replace directory sales as '/u01/test/cap_sales'; ensure your captured workload is in directory /u01/test/cap_sales' execute dbms_workload_replay.process_capture(SALES'); Note: - You need to do the preprocess only once and the preprocessed workload can be re used again.

2) Replaying a Consolidated Database Workload.

1) Create a root directory where all the preprocessed workload will be available for replay. mkdir '/u01/test/cons_dir'; create or replace directory cons_workload as '/u01/test/cons_dir 2) Copy each of the preprocessed workload directories into the root directory. cp r /u01/test/cap_crm /u01/test/cons_dir cp r /u01/test/cap_hr /u01/test/cons_dir cp r /u01/test/cap_sales /u01/test/cons_dir 3) Create database directory for each workload. create or replace directory crm as '/u01/test/cons_dir/cap_crm'; create or replace directory hr as '/u01/test/cons_dir/cap_hr'; create or replace directory sales as '/u01/test/cons_dir/cap_sales';

4) Set root replay directory as created in step 1 execute dbms_workload_replay.set_replay_directory('CONS_WORKLOAD'); 5) Create a replay schedule and add individual captures.

execute dbms_workload_replay.begin_replay_schedule('CONS_SCHEDULE'); select dbms_workload_replay.add_capture('CRM') from dual; select dbms_workload_replay.add_capture('HR') from dual; select dbms_workload_replay.add_capture('SALES') from dual; execute dbms_workload_replay.end_replay_schedule; 6) Initialize Consolidated Replay.

exec dbms_workload_replay.initialize_consolidated_replay('CONS_REPLAY','CONS_SCHEDULE');

7) Remap connections strings. During workload capture, connection strings used to connect to the production system are captured. In order for the replay to succeed, you need to remap these connection strings to the replay system. The replay clients can then connect to the replay system using the remapped connections. After the Initialize step, query the dba_workload_connection_map to find out what connection string needs to be remapped. DBA_WORKLOAD_CONNECTION_MAP displays the connection mapping information for workload replay. Each row defines one connection mapping for a particular workload replay. The column schedule_cap_id shows which capture a particular connection belongs to. schedule_cap_id identifies a workload capture added to a replay schedule. It starts with 1. Please check Oracle documentation. Use DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION procedure to remap the connections. For example Query dba_workload_connection_map for capture connections

select conn_id,schedule_cap_id,capture_conn,replay_conn from dba_workload_connection_map; exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id => 1,SCHEDULE_CAP_ID=>1,replay_connection =>'CRM'); exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id => 1,SCHEDULE_CAP_ID=>2,replay_connection =>'HR'); exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id => 1,SCHEDULE_CAP_ID=>3,replay_connection =>'SALES'); -- Verify connecting remapping is done. select conn_id,schedule_cap_id,capture_conn,replay_conn from dba_workload_connection_map;

Note: - for replay_connection CRM, HR, SALES are services that you have defined on the test system. 8) Prepare Consolidated Replay. exec dbms_workload_replay.prepare_consolidated_replay(synchronization=>'OBJECT_ID');

9) Start replay clients. wrc system/oracle mode=replay replaydir=/u01/test/cons_dir Note: - The replaydir path is pointing to the root directory holding the consolidated captures. You will need to start multiple replay clients. You would need at least 1 replay client per capture. To estimate the total number of replay clients you need to start, execute the following command for each workload in your replay

(a) (b) (c)

wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_crm' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_hr' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_sales'

To estimate the replay clients for the consolidated replay, add the output of the above (a+b+c) and execute the following replay client command (a+b+c) times in background wrc system/oracle mode=replay replaydir=/u01/test/cons_dir & (repeat a+b+c) 10) Start consolidated replay. execute dbms_workload_replay.start_consolidated_replay; 11) Once the replay is finished, generate the replay report. sqlplus / as sysdba -- replay report set echo off head off feedback off linesize 200 pagesize 1000 set long 1000000 longchunksize 10000000 VARIABLE rep_id number; BEGIN SELECT max(id) INTO :rep_id FROM dba_workload_replays; END; / spool report_cons.html select dbms_workload_replay.report( :rep_id, 'HTML') from dual; spool off

12) Generate AWR and ADDM reports for the Replay sqlplus / as sysdba set linesize 200 pagesize 1000 column name format a20 spool awr_id_usecase1.txt select id,name,AWR_BEGIN_SNAP,AWR_END_SNAP from dba_workload_replays order by id; spool off @$ORACLE_HOME/rdbms/admin/awrrpt

Frequently Asked Questions (FAQ) 1) How do I preprocess different workloads? See the heading above Preprocessing Consolidated Database Workload. 2) How many WRC replay clients should I start? To estimate the total number of replay clients you need to start, execute the following Command for each workload in your replay
wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_crm' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_hr' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_sales'

Add together the recommended number of clients from the above. This will tell you how many WRC replay clients you have to start at the minimum. 3) Can I use EM interface (DB Console, Cloud Control, Grid Control) when replaying consolidated workloads? No, at this time we only have API support for replay consolidation. However, workload capture can be done using the supported EM interfaces. A future release of EM Cloud Control 12c will support the replay consolidated feature. 4) Is there a Replay Compare Period Report for Consolidated Replay? No. There is no Replay Compare Period Report for Consolidated Replay. Replay Compare Period Report is only available for non consolidated replays. 5) How do I do performance analysis? This is a very generic recommendation.

While the Consolidated Replay is running we recommend using Enterprise Managers performance page and Top Activity page to monitor the consolidated replay. Enterprise Managers performance page and Top Activity page provides details for key database, and system performance statistics plus helpful insights to any potential contention. After the replay has completed, generate AWR, ADDM reports for the consolidated replay period. For RAC and Exadata be sure to use the Global AWR and ADDM reports. On a high level look at DB Time for consolidated replay. Compare it with the total DB Time for all the replays. Look at CPU utilization, memory utilization, I/O metrics, and network metrics. Ensure that these meet your business requirements. Keep the changes that were done including hardware, storage, database version and interpret the results in context with the changes and consolidation that was done.

6) What synchronization modes are supported for Consolidated Database Replay? Only synchronization = object_id or false is supported for Consolidated Replay feature. 7) What are the best practices to do Consolidated Database Replay? A) Before doing the consolidated Database Replay ensure that each individual replay completes as expected. B) Follow best practices for Database Replay in NOTE:560977.1 which includes using SPA
first to do unit SQL response time testing

8) How do I debug Database Replay

NOTE: 1287620.1 - Database Replay Diagnostic information

Details about new procedures and functions. 1) SET_REPLAY_DIRECTORY Procedure. This procedure sets a directory that contains multiple workload captures as the current replay directory. Syntax DBMS_WORKLOAD_REPLAY.SET_REPLAY_DIRECTORY ( replay_dir IN VARCHAR2); Parameters Parameter replay_dir Description Name of the OS directory containing the captures for a workload consolidation

2) BEGIN_REPLAY_SCHEDULE Procedure. This procedure initiates the creation of a reusable replay schedule. Syntax DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ( replay_dir_obj IN VARCHAR2, schedule_name IN VARCHAR2); Parameter replay_dir_obj Description Directory object that points to the replay directory that contains all the capture directories involved in the schedule

schedule_name Name of the schedule to be replayed

Usage Notes

Only one schedule can be in creation mode at a time. Calling the subprogram again before end_replay_schedule will raise an error. Prerequisites: o The workload capture was already processed using the PROCESS_CAPTURE Procedure in the same database version. o The user must have copied the capture directory appropriately. o The database is not in replay mode. o The SET_REPLAY_DIRECTORY Procedure has already been called.

3) ADD_CAPTURE Function. This function adds the given capture to the current schedule. The directory has to be a valid capture processed in the current database's version. It returns a unique ID that identifies this capture within this schedule. Syntax DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ( capture_dir_name IN VARCHAR2) RETURN NUMBER;

Parameters Parameter Description

capture_dir_name Name of the OS directory containing the capture under the replay top-level

Parameter

Description directory

Note: - use of any other parameter in the function is not supported in 11.2 releases. 4) END_REPLAY_SCHEDULE Procedure. This procedure wraps up the creation of the current schedule. The schedule is now saved and associated with the replay directory and can be used for a replay. Syntax DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; Usage Notes The BEGIN_REPLAY_SCHEDULE Procedure must have already been called.

5) INITIALIZE_CONSOLIDATED_REPLAY Procedure. This procedure puts the database state in INIT for a multiple-capture replay. It uses the replay_dir which has already been defined by the SET_REPLAY_DIRECTORY Procedure, pointing to a directory that contains all the capture directories involved in the schedule. It reads data about schedule schedule_name from the directory, and loads required connection data into the replay system. Syntax DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ( replay_name IN VARCHAR2, schedule_name IN VARCHAR2); Parameters Parameter replay_name Description Name of the workload replay. Every replay of a processed workload capture can be given a name. (Mandatory)

schedule_name Name of the schedule to be replayed. It must have been created through the BEGIN_REPLAY_SCHEDULE Procedure for the replay directory replay_dir

Usage Notes

Prerequisites

Workload capture was already processed using the PROCESS_CAPTURE Procedure in the same database version. Database state has been logically restored to what it was at the beginning of the original workload capture. The SET_REPLAY_DIRECTORY Procedure has been called.

6) PREPARE_CONSOLIDATED_REPLAY Procedure. Similar to the PREPARE_REPLAY Procedure, this procedure puts the database in a special "Prepare" mode for a multiple-capture replay. Syntax DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY ( synchronization IN BOOLEAN, connect_time_scale IN NUMBER DEFAULT 100, think_time_scale IN NUMBER DEFAULT 100, think_time_auto_correct IN BOOLEAN DEFAULT TRUE, capture_sts IN BOOLEAN DEFAULT FALSE, sts_cap_interval IN NUMBER DEFAULT 300);

Only synchronization = OBJECT_ID or FALSE is supported for Consolidated Replay feature. When synchronization is OBJECT_ID, a more advanced synchronization scheme is used. Every action that is replayed will be executed only after the relevant COMMITs have finished executing. The relevant COMMITs are the ones that were issued before the given action in the original workload capture and that had modified at least one of the database objects the given action is referencing (either implicitly or explicitly). This OBJECT_ID scheme has the same logical property of making sure that any action will see the same data it saw during capture. It ensures that SCNs for different captures will not clash. OBJECT_ID synchronization allows more concurrency during replays for the actions that do not touch the same objects/tables. 7) START_CONSOLIDATED_REPLAY Procedure. This procedure starts the replay of a multiple-capture capture. Syntax DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY; Usage Notes Prerequisites:

The call to the PREPARE_REPLAY Procedure was already issued. Enough number of external replay clients (WRC) that can faithfully replay the captured workload has already been started. The status of such external replay clients can be monitored using V$WORKLOAD_REPLAY_CLIENTS.

Who should I contact for more information?

Please create a Service Request with Oracle Support.

If any of these patches are not available for the specific platform or database version (11.2.0.2 and above) or if it conflicts with existing applied patch, please create an SR with Oracle Support, provide the details of the operating system platform, database version, current list of patches applied using opatch lsinventory.