Oracle-to-Oracle Data Transfer Options

Sudhanshu Kumar 1st Feb 2010

Data Extraction & Transfer Options

Non Real Time Real/Near-Real Time
Transportable Tablespaces Pulling Tablespaces

Oracle Replication

Change Data Capture

Data Pump Direct Access using dblink (SQL*Net) File Extracts

Transportable Tablespaces
Pros The quickest way to transport an entire tablespace to a target system is to simply transfer the tablespace's underlying files. Cons For a tablespace to be transported to a target system, it must be selfcontained. The source database tablespace needs to be made read-only until the copying of tablespace ends. Cross-platform issues - Endian-ness of operating system ASM (will need RMAN to pull out tablespaces into file structures)

Data Pump
Data Pump , introduced in 10g, replaces the older Export-Import tool. Data Pump can be used in below modes Transportable tablespace Just a Datapump approach of TTS discussed in previous slide Table/Schema dump Discussed below External Tables Discussed below Network Link (Dblink) Discussed in next slide Cons Slower than Transportable tablespaces. Tests Using EXPDP & IMPDP (Parallel=4) Uses Table dump mode Rows - 65 million Export - 00:02:30 FTP - Depends on n/w speed 10 GB file Can use shared volume/disk Import - 00:01:44 ODI LKM Oracle to Oracle (Data pump) - Uses External Tables mode 1 million records 00:00:17 5 million records 00:01:45 10 million records 00:04:14 20 million records 00:07:23 40 million records 00:13:53 65 million records 00:26:34

Data pump (Network Link Mode)

Uses Dblink between source & target i.e. data transfer happens through SQL*Net Pros In comparison to copy data directly using dblink method which shows exponential increase in time over data growth, this method shows consistent linear increase. Cons Slower than Transportable tablespaces Slower than Data Pump (dump file) Slower than Data Pump (External Tables) Tests Using Network Link (Parallel=4) 2 million records 00:01:22 4 million records 00:02:27 8 million records 00:04:45 16 million records 00:09:00 32 million records 00:17:09

Pulling Tablespaces
New tool in Oracle Database 10g that simplifies the transportable tablespace approach so that it involves only the execution of a single packaged procedure. In this approach, the user "pulls" the tablespaces from the source system, using a supplied package, DBMS_STREAMS_TABLESPACE_ADM. This package uses Data Pump to transport tablespaces and transfers the data files to the target system's format. It also performs any required endian conversion automatically. Cons Limitation in terms of flexibility because its a packaged approach.

Change Data Capture

Synchronous using Triggers Asynchronous Redo log files HotLog Distributed HotLog AutoLog Pros Completeness Performance minimal impact on source systems Easy Interface Minimal overhead cost Cons Additional structures needs to be created on Source systems e.g. triggers for synchronous. Source system needs to operate in archivelog mode. Needs additional maintenance and monitoring compared to other methods

Direct Access using dblink

Pros Simplest data transfer mechanism. Cons There should be some way to identify new records i.e. timestamps Not the fastest way for moving large data across Scalability Shows exponential rise in timings when copying large data set Tests Test performed on moving data in single thread: 0.3 million records - 00:00:09 1.2 million records - 00:00:34 4.75 million records - 00:02:18 9.5 million records - 00:06:35 Linear progression from 0.3 till 4.75 million records, but it becomes exponential from 4.75 to 9.5 million records !!! Running in 5 threads: 9.5 million records - 00:00:50 20 million records - 00:02:46 Running in 10 threads: 80 million records - 00:09:28

File Extracts
Source systems with batch feeds will not be able to provide a near real time feed. Frequent extracts (hourly) can be taken, but this may be an overhead on source systems . Usually source systems only provide a daily extract. Two methods SQL Loader External Tables Cons One of the slowest methods for data transfer.

Oracle Replication using MViews

There can be different ways to perform replication. Actually it comes under CDC. We will focus here on replication using Materialized Views only. Near real time

Cons Additional structure needs to be created on source database side i.e. MV logs.

Comparison of Methods
File Extracts
Real Time (RT)/ Non-Real Time (NRT)

Database Link

Data Pump External Tables

Data Pump Network Link

Transportable Tablespaces

CDC

NRT

NRT

NRT

NRT

NRT

RT

Performance impact on source system

Changes in source systems for implementing the method Performance of data extraction and transfer

Significant

Significant

Significant

Significant

Minimal

Minimal the impact is caused only by additional logging and log transfer

No

No

No 20 million in 7 mins 23 secs

No 16 million in 9 mins

Yes

Yes

Not Tested

9.5 million in 6 mins 35 secs Timestamp Based if timestamps not updated, changes are not identified Simple one step only defined by the SQL Select statement

Not Tested

Not Tested

Reliability of change data identification

Timestamp Based

100% - all the changes are identified via the redo log files

Complexity of the method

Medium

Medium

Simple

Medium

Complex CDC config, change tables on different instance, cannot use complex joins