Database Tuning to reduce the Run-time for EOD/BOD jobs

Client Environment:

Our Client, a Bank, has more than 70 databases with different oracle versions
and operating system platforms. These systems are being used for Core banking
(retail and corporate banking ) applications. Core banking production database
which are running 24*7*365 environment. This environment also has lots of
standby databases available at local as well as remote locations. It also has a
dedicated set of reporting databases that are being created and refreshed at
regular intervals. Flexcube (I-flex’s core banking application) is being used for
running the Core banking application.

Problem statement:

End of day process (EOD) runs every day night in Retail banking and corporate
banking. After finishing both processes Begin of day process (BOD) runs in same
databases and respective report databases bring refresh. Over a period of time it
was observed that the total run-time for the EOD/BOD process had gone to 05
Hours. This in turn was resulting in delaying generation of critical reports required
for regulatory purposes. There are heavy penalties for missing these deadlines.

Client wished to reduce these run times by 50%.

Analysis:

We started to take statspack report during EOD process and found some queries
were fetching high buffers get. During whole EOD process we found only one
wait event significantly shoot higher called “db file scattered read”. It indicated
those full tables scanning were going in the EOD process. We found culprit
queries and send to application team for tuning.

We also found there was no significant CPU usage in said process.

We made some queries to identify hot tables (heavily accessed) which were
being accessed via full table scan. We executed queries during EOD process
and gathered explain plan details. After getting explain plans and studying the
structure of tables we found global non-partitioned indexes created on same
tables (hot tables). This was identified as the root cause. The key issue being,
due to certain limitation of non-partitioned global index, while fetching data the
indexes associated with these tables were being ignored, thus resulting in Full
table scans.
Our Solution:

There are 2 solutions for this issue, First one being, we can convert those global
indexes in to Local partitioned index or second option being, we can put the
partitioned tables in to parallel mode.

We decided to test the effectiveness of both these solutions in UAT/Reporting

environment. We created local prefix partition indexes instead of global non-
partition indexes on those table and executed same queries and generate
explain plans. The new explain plans indicated that all queries were fetching data
using the newly created indexes and did not perform full table scan.

We tested the second option of putting access to these partitioned tables in

parallel. Usually this results in higher CPU usage. The Server had 8 CPU’s, so by
enabling parallel execution, we were able to perform a faster fetch for the data.

Both the scenario’s were present to the client. As the first option required
approval from the application vendor, we got approval for second option and put
those tables in to parallel clause. The first option will be implemented once the
approval from application vendor is received.

Client Benefit:

After changing above structure, next EOD/BOD process finished within 2.30 to 3
hours.