Samer Barakat
Senior Architect, Applications Performance
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.
3 Copyright Oracle 2012, Oracle its affiliates. All reserved. rights Copyright 2012, and/or and/or its affiliates. All rights reserved.
<Insert Here>
Introduction
Introduction
Context within Performance Management
Configuration Requirements & Best Practices Operational Best Practices Patching & Upgrades Performance Testing & Capacity Planning
Workload Management
People Management
Agenda
Introduction to Performance Triage Approach Tuning Application Modules Tuning the Concurrent Manager Tuning the Database Tier
Performance Triage
the problem
If it is a product issue, PASS ON the right information to support/dev through the
regular channels
Try to identify a TEMPORARY WORKAROUND to alleviate the issue while you get a
permanent fix
components in-play
Refer to the architecture
TCP Socket%
PLSQL Lock Timer
Keep a watchful eye on OS metrics and resource consumption in all 3 tiers
10
11
12
14
15
16
Doc 244040.1 Recommended performance patches for all the modules and tech stack components are consolidated in this note
17
techstack components to the latest certified levels For example (as of Aug/2012), for R12 => Database 11.2.0.3, Forms 10.1.2.3, OC4J 10.1.3.5, JDK 6, etc
http://blogs.oracle.com/stevenChan/resource/certifications.html
18
Set the execution frequency as well view history of purge programs R12: programs tagged with the Purge program type
19
Copyright 2012, Oracle and/or its affiliates. All rights reserved.
20
parameter to No Identification of stuck workflows is resource intensive Start a separate background engine to handle stuck/timed out processes with a low frequency i.e. once a day
Use deferred activities to improve online response times for flows such as
static calls
Edit & add heavily used ITEM_TYPES
itemtypeList_t (WFSTD,FNDFFWF)
Generates wffncal2b.pls in utl_file_dir to re-create pkg Disable retention on Workflow Queues
DBMS_AQADM.ALTER_QUEUE(queue_name=>:b1,retention_time=>0);
Partition runtime tables for higher scalability MOS Doc: 260884.1 For high volume batch processing in RAC use ITEM_TYPE to Node affinity
23
MOS Docs 849060.1 & 130511.1 Use streamlined workflows Consider seeded workflow: Line Flow Generic: Performance Removes unnecessary activities and sub-processes, reducing WF data significantly Use Deferred tax & credit check For online booking speed, consider deferring scheduling MOS Doc 421314.1 Configurator should be setup to use a separate pool of JVMs
24
26
tables Index trx flexfields for headers & lines used in validation interface_line_attributeN columns
Parallelize Batch flow By supplied parameters where possible
e.g. Autoinvoice Manually by data sets if needed e.g. Rev Rec, AutoReceipts/Remittance, Auto-accounting (R12)
27
Complement with user participation tests and batch load tests with frequent and critical
jobs
References
http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html
28
Copyright 2012, Oracle and/or its affiliates. All rights reserved.
29
CM General Best Practices Workload Management Strategy CM Configuration Best Practices Parallel Concurrent Processing (PCP) Best Practices Concurrent Manager Tracing Best Practices
30
Review long-running auto resubmitted jobs Review short/long-running jobs selective parameters? Dont purge jobs that the users then have to run again Trim the fat: Review and eliminate concurrent jobs that are not required and/or are not being used by users.
31
Purge the FND tables on a regular basis using the Purge Concurrent Request and Manager Data program Patch 7530490 for R12 or 7834670 for 11i (MOS Doc ID 822368.1)
High throughput: Keep *.out and *.log on fast disks. Use Separate disks if throughput seems slow especially on NFS Filers
Truncate the reports.log file in log directory. Watch for 2GB limit on any output files (Doc ID 842850.1) Use specialization rules and work shifts to bind specific jobs to specific time windows . Avoid scheduling resource intensive batch requests during peak activity. Reschedule some programs to run when the concurrent managers have excess capacity. Define Workload Management Strategy based on job average duration and system usage profile.
Manage CM Schedules:
32
Avoid Short Sleep Times *Enough for a couple of minutes of work **Set cache size to at least twice the number of workers Specialization rules and work shifts Bind specific jobs to specific time windows
33
Profile Option: Concurrent: Active Request Limit can be used restrict the number of concurrent requests that may be run simultaneously by each user
Specialize if there are too many jobs in a specific category
To maximize throughput for jobs which spawn parallel workers (i.e. Auto Invoice, Payroll), consider reducing the sleep time of the Conflict Resolution Manager (CRM) Default is 60s, consider 5 or 10 seconds
Define dedicated queues for short and frequent requests and increase the sleep times for managers which do not require near real-time job execution
Reducing sleep time to a very low value many cause excessive CPU utilization. Set the profile Concurrent:Wait for Available TM to 1 (second) to minimize TM latency. The profile sets the total time to wait for a TM before switchover to next available TM Set Sleep time on Transaction Managers to 30 minutes. this avoids constant polls to check for shutdown requests.
34
Consider dedicating certain concurrent managers to process either short or long running programs to avoid queue backup Add more queues if all queues are running at maximum capacity only if resources permit. Tune the sleep time for queues with a large number of target processes. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table
Cache Size:
Increase the cache size (number of requests cached) to at least twice the number of target processes
Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time.
35
36
OM, Receivables Super User, Create Accounting WF background engines Node to ITEM_TYPE assignment
Use Affinity : Direct some workload components to specific RAC nodes to maximize scalability by minimizing inter-node communication and synchronization work.
37
12.1.3 introduced option to define node affinity at the program level Concurrent -> Program -> Define
38
39
Shipping
Education Contracts
Payroll
HR (with self-service) Benefits OTL (self-service)
iProc, Expenses
DBI
40
Program Level
Not recommended Enable the trace/profiling option in the concurrent program definition
Request Level
Ensure Concurrent: Allow Debugging is set to Yes
Tracing Documents
280391.1 How to Retrieve SID Information For a Running Request
41
42
43
Configuration Best Practices CBO Statistics Gathering Performance Management Common corrective actions to maximize scalability 11g New Features
44
11gR1 Certified with 11i and R12 MOS Docs 452783.1 & 735276.1
45
For dedicated server processes plan for PGA allocation of 30-40MB per shadow process (correlated to the process init.ora parameter) For shared server processes, user connections allocate almost all of their memory from the SGA, so more SGA memory will be needed, and much less PGA will be required.
46
system (if file systems are being used for the DB files) Configure & verify direct IO is used (if file systems are being used for the DB files) Improves performance/scalability by simulating raw devices, avoiding file system cache traffic Ensure your IO subsystem can handle your peak IO load & IOPS Consider RAID, S.A.M.E. configuration with sufficient spindles Measure with ORION tool or in 11g
DBMS_RESOURCE_MANAGER.CALIBRATE_IO MOS Doc 727062.1
Reduce the IO workload
Tune expensive SQL Trim the workload of unnecessary tasks Do 3X less IO on average with 11g Advanced Compression Option
47
Copyright 2012, Oracle and/or its affiliates. All rights reserved.
sizes for tablespaces containing large transaction tables/indexes. OATM supports uniform extent allocation and auto allocate extent (AUTOALLOCATE ) management OATM provides efficient space utilization. This is achieved by supporting locally managed tablespaces as opposed to the dictionary managed tablespace OATM Helps avoid frequent space management operations. Migration utility can be used to migrate a schema at a time
48
49
PARALLEL_INSTANCE_GROUP
11gR1 - Can be done as above or via services 11gR2 - PARALLEL_FORCE_LOCAL =true Tune sequence cache sizes for hot sequences (e.g. 1,000 or higher) in order to
minimize index key contention for the Apps surrogate keys. Make sure to use FNS_STATS to gather statistics.
50
Do not gather statistics excessively on entire schemas or the entire database such as
nightly or weekly
Use the GATHER_AUTO option to gather incrementally Use only FND_STATS or the Gather Schema / Table Statistics Concurrent Programs Do NOT USE the analyze or dbms_stats command directly. It is not
supported, and can result in sub-optimal plans. When Gather Schema Statistics concurrent program is used fnd_stats does the bookkeeping for the run.
51
52
Statistics are used by Oracle to estimate the combined selectivity of the predicates.
Multi-Column Stats can be seeded by calling
FND_STATS.LOAD_EXTNSTATS_COLS. FND_STATS Schema/Table stats will take care of the gathering stats on Multi-Column stats.
11g Extended Stats FND_STATS Patch 9470196. Also, in patch set 9962815.
53
54
56
Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 173 (recursive depth: 2) Rows Row Source Operation ------- --------------------------------------------------7674758 SORT AGGREGATE (cr=153846532 pr=225987 pw=0 time=7174772779 us) 7675089 NESTED LOOPS (cr=153846532 pr=225987 pw=0 time=6919378560 us) 7675089 SORT UNIQUE (cr=123146176 pr=162809 pw=0 time=6083080435 us) 55878643 TABLE ACCESS BY INDEX ROWID AR_TRX_COGS_GT (cr=123146176 pr=162809 pw=0 time=5577872837 us) 136390387 INDEX RANGE SCAN AR_TRX_COGS_N1 (cr=23439707 pr=37706 pw=0 time=2136416980 us)(object id.. 7675089 TABLE ACCESS BY INDEX ROWID RA_CUSTOMER_TRX_LINES_ALL (cr=30700356 pr=63178 pw=0 time=539908369 us) 7675089 INDEX UNIQUE SCAN RA_CUSTOMER_TRX_LINES_U1 (cr=23025267 pr=2820 pw=0 time=196155435 us)(object ..
58
59
60
61
62
63
(db file sequential read, db file scattered read, log file sync, free buffer waits, etc...)
Tune Top SQL is always the first step
Make sure that system has updated statistics. Maximize Memory availability, allocate generously to buffer cache, review AWR
logical IOs by Segment" Some of these waits are commonly caused by bad SQL execution plans
i.e. :latch: cache buffer chains, %buffer busy waits, read by other session
Make sure to use ASSM and OATM tablespace model. Increase INITRANS to alleviate ITL contention. Increase sequence caches, Work with support/consulting to evaluate if table/index partitioning will help . If the program allows, more frequent commits (batch size) can help by reducing
65
66
twp_upgrading_10g_to_11g_what_to_expect_from_optimizer.pdf
http://www.oracle.com/us/solutions/benchmark/apps-benchmark/
67
waiting hours for a complete trace to get the right trace data Expanded event syntax :
- alter session set events sql_trace [sql:a5ks9fhw2v9s1|56bs32ukywdsq]
68
69
Some others
Interval partitioning, Additional partitioning schemes Text indexing performance enhancements RAC scales better & AWR is more RAC aware
Many others..
70
1121043.1 Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite
1362660.1 Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3 301137.1 OSW Document 461053.1 OSWg Document
71
Other References
72
References
R12.1 documentation roadmap (790942.1) Oracle E-Business Suite Release 12.1 Info center (806593.1) Database preparation guidelines for R12.1 upgrade (761570.1) Patching FAQs (459156.1, 225165.1) Staged or shared APPL_TOP and distributed AD (734025.1, 384248.1, 236469.1) OAM Patch Wizard overview and FAQ (976188.1, 976688.1) AD Command Line Options for Release R12 (1078973.1) Recommended Performance Fixes (244040.1) R12 Upgrade Sizing & Best Practices (399362.1)
73
Additional Resources
74
Additional Resources
EBS 12.1.3 Data Model Comparison Report (1290886.1) EBS ATG Seed Data Comparison Report (1327399.1) EBS File Comparison Report (1446430.1)
White paper
75
Oracle Demos
Moscone West
End-to-End Management of Oracle E-Business Suite SOA-Based Integration for Oracle E-Business Suite W-063 W-064 W-065
New User Productivity Capabilities in Oracle E-Business Suite Oracle E-Business Suite 12.1 Technical Upgrade Best Practices
Advanced Architecture and Technology Stack for Oracle E-Business Suite
W-066
W-067
76
Related Sessions
Wednesday
10:15 am CON8809 - Oracle E-Business Suite 12.1 Upgrade Best Practices: Technical Insight Isam Alyousfi, Udayan Parvate CON9032 - Upgrading Your Customizations of Oracle E-Business Suite 12.1 Sara Woodhull CON9259 - Oracle E-Business Suite Internationalization and Multilingual Features Maher Al-Nubani CON9029 - Oracle E-Business Suite Directions: Slashing Downtimes with Online Patching Kevin Hudson Moscone West 3011 Moscone West 2016 Moscone West 2018 Moscone West 2016
10:15 am
10:15 am
11:45 am
77
Related Sessions
Wednesday
1:15 pm CON9027 - Personalize and Extend Oracle E-Business Suite Applications with Rich Mashups Gustavo Jimenez, Padmaprabodh Ambale CON9036 - Advanced Oracle E-Business Suite Architectures: Maximum Availability, Security, and More Elke Phelps CON9047 - Efficiently Scaling Oracle E-Business Suite on Oracle Exadata and Oracle Exalogic Isam Alyousfi, Nishit Rao Moscone West 2016
3:30 pm
5:00 pm
78
Related Sessions
Thursday
11:15 am CON8716 - Web Services and SOA Integration Options for Oracle E-Business Suite Rekha Ayothi, Veshaal Singh CON9030 - Recommendations for Oracle E-Business Suite Performance Tuning Isam Alyousfi, Samer Barakat CON3429 - Using Oracle ADF with Oracle E-Business Suite: The Full Integration View Siva Puthurkattil, Juan Camilo Ruiz, Sara Woodhull CON9020 - Integrating Oracle E-Business Suite with Oracle Identity Management Solutions Sunil Ghosh, Elke Phelps Moscone West 2016 Moscone West 2018 Moscone West 3003 Moscone West 2016
11:15 am
11:15 am
12:45 pm
79
Related Sessions
Thursday
2:15 pm CON9019 - Troubleshooting, Diagnosing, and Optimizing Oracle E-Business Suite Technology Gustavo Jimenez CON9031 - The Top 10 Things You Can Do to Secure Your Oracle E-Business Suite Instance Eric Bing, Erik Graversen Moscone West 2016 Moscone West 2018
2:15 pm
80
Q&A
81
Copyright 2012, Oracle and/or its affiliates. All rights reserved.
82
83