Extract

Interface
Staging Tables Interface Tables

Load

(Source Instance)

Extract

Transform

Load

Destination Instance

Standard APIs

Validation and Reconciliation

Abstract: The Data Migration process is about,

1) Extracting the Data from Source Instance 2) Validating the Data at Destination Instance 3) Loading the Data at Destination Instance.

Scenario:
Data Migration from 11.5.5 to 11.5.10.2 instance. Huge Volume of Data. Production Go-live instance availability - Time limitation. Loaded data in terms of quality and quantity Others Execution sequence, strategy, reporting etc.

Best Practice:
In terms of learning's gathered during the data migration process which can provide basics for a smooth execution of a DM project. 1. DM Strategy to be dependent on the number of records to be migrated against each components. E.g.: If we have around 50 records to be migrated, its better to get that migrated manually for a component. 2. A Separate USER to migrate the data. This will help identifying the records easily, which are migrated.

3. The development/test instances should have enough capacity to mock the data migration activity for identifying and estimating the timings required for the data migration process.
4. The sequence of execution of components needs to be identified. Separate investigation to be performed to identify the components that can run in parallel. Note: When planning to run the components in parallel, take care that the instance is capable enough to support the parallel activity. Else it might produce negative results. Static and Dynamic component wise segregation will provide better picture. 5. Identify the components which needs to be migrated during early phases of data migration process. Its always better to have each component tested at least twice along with all the other components. Say for e.g.: if 6 test cycles are there, make sure to get all the components identified, developed and tested at least by 4th test cycle.

Best Practice:
6. For better control, Extraction of records from Source instance (or OU etc.) , Validation and Loading to be planned separately. The validation of extracted records especially by business would provide better confidence.

7. If any kind of validation programs are run against the extracted records, especially when
validated against the set ups, its always advisable to send out the reports to the set up team who might have missed some set ups which can be done, before loading the records. Note: this will allow to get a better number/percentage of load. Make sure that this process is in place as part of the data migration activity. 8. Always design the validation programs in such a way that they can be re executed multiple times without much effort (doing any back end updates. Performing any kind of back end updates during go live is too tough and DBAs wont be willing to do that. 9. Performance is a critical area which needs focus for a dynamic go live activity. Take care specifically when we got large volume of data to be migrated. Each component needs to be tuned to produce best results. 10. Identify the Data Base Triggers (standard as well as custom) which are defined against the base tables where we populate the data. This helps us to disable the triggers which are not required while during the Data migration activity. Take good care to Enable them back once DM activity is finished.

Best Practice:
11. Take care to cross verify data pertaining to the specific data migration especially when using standard interface tables. Note: standard interface tables may have some other data also which is already there existing. In the sense it should not be a surprise to the DM team, to note that some other data was also present there in the interface tables along with the Migration data which might also get loaded along. This might create difficulty in generating DVR (Data Validation reports) reports especially when reports make use of created_by user as Data Migration User. Plans should be in place for what exactly has to be done. Plans can be there to back up these other records to a back up table and remove this from interface table and move it back to interface once DM activity is over. 12. Cross component training needs to be done within team members Execution perspective. This allows arrangement in executing the components in case of personal emergencies. Note: DM is a very dynamic activity where we should not loose execution time in between, once the dependent tasks are executed. 13. When large volume of data needs to be migrated for a component, its better to do data migration in parts for that component as different sets of data, and make sure that each cycle is complete. Eg: When large volume of data is there, plan to do it in different cycles which involves same set of process with different (logical) sets of data. And make sure to cross verify each cycles data in the interface tables, error tables, staging tables etc to be in better control. Note: Some times it happens like, although we ran a standard import, it may not even pick up the records or it might change the statuses of records to some strange values which we may not have ever seen. 14. When large volume of data is involved, plan to take back ups of Standard interface tables, error tables and clean these tables so that the further runs would be faster. And also can think of re analyzing/gather stats the interface tables each time which improves performance.

Best Practice:
15. Test cycles for data migration should be meant to test the set ups, instance, data quality, load against extracts, the time taken for executing each component and ultimately the time taken for the whole migration process. Each test cycle should provide inputs to further improvement of data quality and better load timings and better number/percentage of data load. This may also include the learnings for parallel executions of components which can be run independently. 16. The more test cycles we do the better we achieve. 17. Reporting is a very important activity related to data migration. Define the reporting in such a way that, the report shows up no ID fields, until and unless asked for. For e.g.: rather than Org_id use Org_name. The success/failure counts should match the extract counts. 18. If huge volume of data is spooled in for reports, think of alternatives like having data reporting as different sets. For e.g.: data reporting in terms of OU wise, use sql*plus to spool the data etc. 19. Another identification is that data migration projects are the ones which well suited for six sigma, because DM projects play around with numbers(Extracted, loaded etc). 20. Execution team should be structured in such a way to have back up resources for executing the same component. This is very much required for components with huge volume of data and which are time consuming to load. (Load includes loading to base tables as well as reporting. Any thing which takes more than 12 hours should have an additional back up resource to execute the same component. Plan the back up resource so that the primary execution is done by the critical resource and the back up resource is just to carry the load forward).

Take Away / Benefits and Business Value Generated

With these we could achieve,

(1) Very Good percentage of loaded data Over all 99%.

(2) Very Good Quality of data loaded Which was very much appreciated by the client.

(3) Very Good Load Time for a Dynamic Go Live.