Gather stats fails after upgrade to Oracle 10G

Gather Schema Statistics or Gather Table Statistics or running FND_STATS fails with "ORA-20005: object statistics are locked". Example: ERROR: While GATHER_TABLE_STATS: object_name=AR.AR_REV_REC_QT***ORA-20005: object statistics are locked (stattype = ALL)*** This can happen with Advance Queue tables. In 10g, if a queue is created or recreated during the upgrade, automatic statistics gather gets locked (disabled) on these queue. The following statement can be used to check the tables which have statistics locked: select owner, table_name, stattype_locked from dba_tab_statistics where stattype_locked is not null; SOLUTION: Unlock gather stats running on those queues : exec dbms_stats.unlock_schema_stats('schema_owner'); exec dbms_stats.unlock_table_stats('table_owner','table_name'); Example: SQL> exec dbms_stats.unlock_schema_stats ('AR'); SQL> exec dbms_stats.unlock_table_stats('AR', 'AR_REV_REC_QT');

How to resolve cloning issue: BEGIN failed--compilation aborted at adcfgclone.pl line 26.

While running adcfgclone.pl as a part of database Tier clone, the below mentioned error occurs: -bash-3.00$ perl adcfgclone.pl dbTechStack Can't locate strict.pm in @INC (@INC contains: /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3/i686-linux-thread-multi /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3 /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3/i686-linux-thread-multi /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3 /ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl .) at adcfgclone.pl line 26. BEGIN failed--compilation aborted at adcfgclone.pl line 26. As indicated in the above error message, this is due to the PERL setting. We dint have the PERL5LIB set : -bash-3.00$ echo $PERL5LIB -bash-3.00$ which perl /d02/oradev/devdb/10.2.0/perl/bin/perl Resolution: $PERL5LIB was set to resolve the issue: Contents under $ORACLE_HOME/perl : -bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl total 16 drwxr-x--drwxr-x--drwxr-x--drwxr-x---

2 oradev dba 4096 Feb 2 oradev dba 4096 Feb 4 oradev dba 4096 Feb 4 oradev dba 4096 Feb

9 04:33 include 9 04:33 bin 9 04:33 lib 9 04:33 man

Contents under $ORACLE_HOME/perl/lib: -bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl/lib total 8 drwxr-x--- 3 oradev dba 4096 Feb 9 04:33 site_perl drwxr-x--- 44 oradev dba 4096 Feb 9 04:33 5.8.3 -bash-3.00$ PERL5LIB=/d02/oradev/devdb/10.2.0/perl/lib/5.8.3 -bash-3.00$ export PERL5LIB

-bash-3.00$ id uid=500(oradev) gid=502(dba) groups=502(dba) -bash-3.00$ echo $PERL5LIB /d02/oradev/devdb/10.2.0/perl/lib/5.8.3

Issue running adcfgclone.pl on Database Oracle home.

While running adcfgclone.pl as oracle user, the session just hangs without giving any error in the log file. The issue was that it wasnt able to connect to the database as: sqlplus "/as sysdba" Deep dive: Most generally this can occur due to improper shutdown of the database. While in the course of a database shutdown, if any of the oracle process is not completed killed and it holds a large shared memory segment, such issues can occur which would prevent from normal database operation. So next time when an instance tries to start it is not able to grab a large chunk of memory to get started. This can be checked as follows: After the database has been shut down , there shouldnt be any processes running as oracle on the server level: ipcs -m | grep <oracle user> eg: ipcs -m | grep oracle key shmid 0x185ee230 884763 owner perms oracle 640 bytes nattch 789785472 0 status

Here in the above example, a single oracle process holds 789785472 bytes of shared memory, even after shutdown. If the database instance is started, then we would expect abnormal behaviors on the database side, like the login issue that we are discussing now. This can cleaned as mentioned below:

ipcrm -m < shmid> eg: ipcrm -m 884763 This would free up the memory held. This resolved the issue.

Issue starting 10g Forms and reports server

Error:

Ping URL for proc: 887947305 failed while pinging url:/reports/rwservlet/pingserver?start=auto 12/11/01 08:18:46 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_BI_Forms~default_island~1 (887947305:8511) 12/11/01 08:19:10 [2] [libopmnoc4j] AJP13: Receiving response code: 500 while configured success code is: 200 12/11/01 08:19:10 [2] [libopmnoc4j] Ping URL for proc: 887947305 failed while pingingurl:/reports/rwservlet/pingserver?start=auto

Search String:

REP-56105

tail f OC4J~OC4J_BI_Forms~default_island~1

ConfigFileName: /u01/oracle/product/ais10g/forms90/server/forms/server/formsweb.cfg REP-56105: Engine rwEng-0 died with error

SOLUTION:

Bug no: 3557723

Work Around:
Disable pingserver command in $ORACLE_HOME/opmn/conf/opmn.xml - Take the backup of opmn.xml file @ cd $ORACLE_HOME/opmn/conf @ cp opmn.xml opmn.xml.PINGAUTO @ - Open file opmn.xml @ - Take out the following lines from opmn.xml @ < category id="urlping-parameters"> @ < data id="/reports/rwservlet/pingserver?start=auto" value="200"/>

@ < /category> @ - Bounce the opmn @ cd $ORACLE_HOME/opmn/bin @ ./opmnctl stopall ; ./opmnctl startall

Error: CACHE mismatching

SOLUTION:

Also Check for the domain name in /u01/oracle/product/ais10g/webcache Webcache.xml ----change the domain name.

Issue: Concurrent request running from a very long time

. Application version: 11.5.10.2 multi-node (4 nodes) Database: 11g RAC Scenario: Business critical concurrent request is running from more than 2 hours which was supposed to complete within 1 hour. The standard manager and the manager responsible to execute the critical concurrent request (custom manager) were overloaded with many requests. The server on which these managers run had very high server load and all CPUs were utilized. After diagnosis, we had to kill few requests which were running from more than 2 hours and were creating load on the server. These requests were in fact not performing anything and had gone zombie. The server load came down and CPUs available were set free to execute new requests/processes. However the business critical request was still running and wasnt moving ahead at all. The database session associated was also not executing anything. Challenge: Since this being a critical business request, we couldnt terminate the request coz we had already lost few hours running it. At the same time the database session also wasnt performing any execution and the whole request was not going anywhere even though it showed running normal status. Trick: Not recommended, however can be a life saver at times. ---- Forcibly change the status of the request to pending normal. ---- Kill the database session of the request. Use the below scripts:

update applsys.fnd_concurrent_requests set phase_code = 'P', status_code = 'I', actual_start_date = null, crm_release_date = null, controlling_manager=null, logfile_name = null, logfile_node_name = null, outfile_name = null, outfile_node_name = null, crm_tstmp = null where request_id in ('request-id'); alter system kill session 'sid, serial#, @instance-id'; Hence a new database session will get created as soon as the old one is killed and the request starts running. Since the server on which the request is supposed to get processed also had no issues now, the request went through very fast without any issues ;)

Login issues in Oracle Apps R12

Application version: 12.0.4 running on 11g RAC database. None of the users are able to login to the Oracle Apps R12 URL. The URL just hangs without giving any information. All the application services were up. Couldnt find any significant load on the CPU and the number of concurrent requests running at the time was normal. The http error_log had the below error: oc4j_socket_recvfull timed out The Application log has the below error: 12/04/26 03:15:13.511 html: Servlet error java.lang.OutOfMemoryError: Java heap space 12/04/26 03:50:35.503 html: Servlet error The opmn log had the below error: 12/04/26 03:15:00 java.lang.OutOfMemoryError: Java heap space 12/04/26 03:15:02 java.lang.OutOfMemoryError: Java heap space 12/04/26 03:15:06 java.lang.OutOfMemoryError: Java heap space

12/04/26 03:15:13 Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space oracle.jbo.DMLException: JBO-26066: Error during rollback Solution: We would need to do changes to -Xms ( for start heap memory ) and -Xmx ( for maximum heap memory ) of the opmn.xml. Changing the Xmx to 512 MB and Xms to 128 MB will help. >>>> Increase Java Heap Size for the OC4J instance as per the available memory (RAM) & restart the instance. You can do it with AS Control -> OC4J Instance -> Administration Tab -> Server Properties -> Java options. Update -Xms -Xmx java options. Once updated, stop and start the OC4J instance to pick up the changes.

Login issues in Oracle EBS 11i after Clone ( from hotbackup)

Error in login page:

ORA-01110: data file 98: '/disk09_250/oradata/mine/a_txn_data14.dbf' ORA-26040: Data block was loaded using the NOLOGGING option

SOLUTION:

1. Compile jsp: cd $FND_TOP/patch/115/bin perl ojspCompile.pl --compile --flush

2. Access the form directly using the URL: http://sanop.mine.com:8020/forms/frmservlet?form_params=config=PRODMINE

3. Run the request: "Synchronize WF LOCAL tables" - Parameters: ALL "Synchronize WF LOCAL tables" - Parameters: FND_USER

OACORE fails to start after increasing the JVM processes in Oracle R12
Issue: OACORE processes are not starting. Recent changes: Increased the JVM processes in the context file. Error:
adoacorectl.sh: exiting with status 206

Solution: 1) Stop the middle tier processes: adopmnctl.sh stop 2) Kill any runaway/zombie processes. 3) Remove the following jms lock files: $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/forms_default_group_1/jms.state.lock, $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/jms.state.lock $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/oafm_default_group_1jms.state.lock 4) Restart middle tier: adopmnctl.sh start

Restore FNDLIBR executable

Restore FNDLIBR executable which has been deleted accidentally: 1. Invoke adadmin on concurrent manager node 2. Select option 2. Maintain Applications Files menu 3. Select 1. Relink Applications programs 4. Enter FND when it prompts for Enter list of products to link ('all' for all products) [all]

5. When prompted for Generate specific executable for each selected product [No] ? YES select YES & from list of executable select FNDLIBR. This will create new FNDLIBR executable.

Move Concurrent Processing Server from one node to another in Oracle Apps 11i
Example: Oracle E-Business Suite (11.5.10.2) with 2 nodes: Node 1: Database and the Concurrent Processing Node 2: Forms, Reports and Web Server.

Goal: To Move the concurrent processing from Node1 to node 2.

Requirement: Shared appl_top / Shared application tier file system. If not follow the Step 2 below. How to perform: 1. This is recommended to be performed on a NON-PRODUCTION system first with necessary backups. 2. Create a shared APPL_TOP by following Metalink Note 233428.1 for sharing the application tier files system and allow the node 2 to share for concurrent manager node ( Node 1). 3. Edit context file on the Forms and Web node2 and change following context variables: s_isConc from "NO" to " YES " s_concstatus from "disabled" to " enabled " On Node1: make the below changes s_isConc from "YES" to "NO" s_concstatus from "enabled" to "disabled" 4. Run autoconfig on all the APPS nodes.

Concurrent managers are down with status: System Hold, Fix Manager
Issue: All the concurrent managers are down with a status message: system hold, fix manager. The administer Concurrent manager screen has Actual: 0 and Target: 1 for all managers.

Solution:
-

Ensure all application services are down.

Please check the system profile: Concurrent: GSM Enabled. This should be set to Y Execute: afdcm037.sql Path : $FND_TOP/patch/115/sql Description of the script: PL/SQL script to create libraries for FNDSM and FNDSVC and create Managers for Preexisting Nodes. Future nodes will be handled by db trigger. - Relink the below mentioned executables: adrelink.sh force=y fnd FNDLIBR adrelink.sh force=y fnd FNDSM adrelink.sh force=y fnd FNDFS adrelink.sh force=y fnd FNDCRM

- Execute cmclean.sql to clean out the concurrent manager tables. - Start all the application services and restest the issue.

ORA-00257: archiver error. Connect internal only, until freed

While trying to connect to the database, we get the below mentioned error: ERROR:

ORA-00257: archiver error. Connect internal only, until freed. Cause: The error is caused by the fact that the database archiver process is unable to archive the current online redolog due to lack of space in the destination for the archivelogs. The flash_recovery_area is full. Applicable to 9i, 10g and 11g database versions. Solution: The basic idea is to free space so as to make room for new archiving to happen. This can be addressed in multiple ways. Allow more space in the DB_RECOVERY_FILE_DEST with the DB_RECOVERY_FILE_DEST_SIZE parameter : alter system set db_recovery_file_dest_size=5G ; Imagine a situation where the space increased in step 1 gets filled, ie, 5 GB. Set an alternative archiving destination, so that when the primary destination is full, archiving is automatically performed to the alternate destination: Following changes in parameter file can enable the same: log_archive_dest_1='LOCATION=use_db_recovery_file_dest NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_2' log_archive_dest_2='LOCATION=/other_dest_for_archiving' log_archive_dest_state_1='enable' log_archive_dest_state_2='alternate' db_recovery_file_dest='/db01/mine/oracle/product/10.1.0/db_1/flash_recovery_area' db_recovery_file_dest_size=5G

Check the archive_log_dest and verify if the path mentioned exists and is valid. Take a backup of the existing archive files (recommended) OR it can be deleted to make more free archiving space. This can be done at the server level where files can be manually deleted. However incase of ASM storage with rman, below mentioned way is recommended (which would perform backup and then deleting):

Example : rman target sys/sys RMAN> backup archive log all ; RMAN> delete archive until time 'trunc(sysdate)';

Unable to authenticate session in Oracle apps R12 (12.0.4): Login Issue

Unable to authenticate session in Oracle apps R12 (12.0.4): Login Issue The Release 12.0.4 instance was cloned from another identical instance. When we try to login, the login page does not display Have tried clearing the cache (_pages). Have tried recompiling jsps and bouncing however the problem still persists. Cleared all the Apache and related logs and found an interesting error. It is in the following log file: $INST_TOP/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log 08/08/21 15:17:00.601 10.1.3.0.0 Started 08/08/21 15:17:02.769 html: 10.1.3.0.0 Started 08/08/21 15:17:33.132 html: oracle.apps.fnd.security.LeakDetectionFilter.initialized:1219349853132 08/08/21 15:17:33.132 html: LeakSetting: global:true,session:false,aggressive:false,stderr:false,appslog:false,corelog:false ignoring:null 08/08/21 15:17:34.859 html: chain failed javax.servlet.ServletException: java.lang.RuntimeException: Unable to create user session. Please contact your System Administrator. at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:780) Solution: Enable FND debug The FND debug log had the below error:

Oracle error -1654: ORA-01654: unable to extend index APPLSYS.FND_LOGINS_N2 by 16 in tablespace APPS_TS_ARCHIVE has been detected in FND_SIGNON.AUDIT_USER. Check if the APPS_TS_ARCHIVE Tablespace is full. Increase the size of tablespace APPS_TS_ARCHIVE / add a datafile to the tablespace. This will resolve the issue.

ICX: Session Timeout and autoconfig variable s_oc4j_sesstimeout in 12.0.4

Recently we came across a forms time out issue. All the user forms sessions were getting disconnected after an Hour. The FORMS_TIMEOUT variable was set to 120 mins though. This should practically disconnect the forms session after 2 hours. Here the reason for the issue is because of the mismatch of the ICX and OC4J timeout session parameters. OC4J session timeout, if not set, defaults to 20 minutes. This will cause the oc4j session to timeout before the Apps/ Forms session timeout which is defined by profile option "ICX: Session Timeout" Profile option "ICX: Session Timeout" and autoconfig variable "s_oc4j_sesstimeout" in context file should match. Example: In case you need the session to timeout after 2 hrs then the profile option "ICX: Session Timeout" and autoconfig variable "s_oc4j_sesstimeout" should be set to 120.

Unable to apply RDBMS patch (Opatch) on 10G ENV after postclone and upgrade from 9i.
UNABLE TO APPLY RDBMS PATCH (Opatch) ON 10G ENV after postclone and upgrade from 9i. SOLUTION: Some options of opatch: Opatch apply invPtrLoc Opatch apply no_inventory Which opatch

~/Oraclemine/10.2.0/OPatch/opatch (Oracle06) 5005469 - -bash $ opatch apply -invPtrLoc $ORACLE_HOME/oraInst.loc Invoking OPatch 10.2.0.2.0 . Oracle interim Patch Installer version 10.2.0.2.0 @ Copyright (c) 2005, Oracle Corporation. All rights reserved.. . Oracle Home: /mine/oracle/oraclemine/10.2.0 Central Inventory: /mine/oracle/oraclemine/10.2.0/oraInventory from : /mine/oracle/oraclemine/10.2.0/oraInst.loc OPatch version: 10.2.0.2.0 OUI version: 10.2.0.2.0 OUI location: /mine/oracle/oraclemine/10.2.0/oui Log file location: /mine/oracle/oraclemine/10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_21-58-34-PDT_Tue.log . ApplySession applying interim patch '5005469' to OH/mine/oracle/oraclemine/10.2.0' List of Homes on this system: OPatchSession cannot load inventory for the given Oracle Home /mine/oracle/oraclemine/10.2.0. Possible causes are: No read or write permission to ORACLE_HOME/.patch_storage Central Inventory is locked by another OUI instance No read permission to Central Inventory The lock file exists in ORACLE_HOME/.patch_storage The Oracle Home does not exist in Central Inventory . Apply Session failed: ApplySession failed to prepare the system. No detail available. System intact, OPatch will not attempt to restore the system . OPatch failed with error code 73 =========================================================================== I have checked that patch_lock file is not available on $ORACLE_HOME/.patch_storage and no lock on Inventory. Also required permissions are exisitng for .patch_storage and oraInventory. . lsinventory gives the following info =========================================================================== (oracle06) 5005469 - -bash $ opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc Invoking OPatch 10.2.0.2.0 . Oracle interim Patch Installer version 10.2.0.2.0 @ Copyright (c) 2005, Oracle Corporation. All rights reserved.. .Oracle Home : /mine/oracle/oraclemine/10.2.0 Central Inventory : / mine /oracle/ oraclemine /10.2.0/oraInventory from : / mine /oracle/ oraclemine /10.2.0/oraInst.loc OPatch version : 10.2.0.2.0 OUI version : 10.2.0.2.0 OUI location : / mine /oracle/ oraclemine /10.2.0/oui Log file location : / mine /oracle/ oraclemine /10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_22-00-16-PDT_Tue.log . List of Homes on this system: Inventory load failed... OPatch cannot load inventory for the given Oracle Home. Possible causes are:

Oracle Home dir. path does not exist in Central Inventory Oracle Home is a symbolic link Oracle Home inventory is corrupted LsInventory Session failed: OracleHomeInventory::load() gets null oracleHomeInfo .OPatch failed with error code 72 =========================================================================== Looks like $ORACLE_HOME is not attached to the local inventory. Attaching the ORACLE_HOME to the inventory. Run runClone.sh as mentioned under "Run 10g Cloning" step . This will resolve the issue

How to recover a cloned database.

Source Instance:

oracle@mine01$ sqlplus "/as sysdba" SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:26:33 2007 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to: Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 Production

SQL> !pwd /d02/oracle/proddb/9.2.0/admin/PROD_mine01/udump

SQL> alter session set tracefile_identifier=mine; ( this would set the trace file name as mine) Session altered. SQL> alter database backup controlfile to trace; Database altered. oracle@mine01$ ls -ltr *MINE* -rw-rw-r-- 1 oracle dba 67001 Apr 21 03:28 prod_ora_23305_MINE.trc

uuencode prod_ora_23305_MINE.trc prod_ora_23305_MINE.trc|mailxsandesh.achar@mine.com ( This would send an email to your mailbox with the trace file attached) Get this controlfile to target instance.

Target Instance: oradev@ mine02# ls -ltr *MINE* -rwxrwxrwx 1 oradev dba 67001 Apr 21 03:29 prod_ora_23305_MINE.trc

oradev@mdsuaor09# cp prod_ora_23305_MINE.trc mine.sql This would copy the trace file to a new file named mine.sql and all the required changes can be done to mine.sql as mentioned below: vi mine.sql esc:se nu check where the create controlfile line starts ( double copy of create control file line)and delete till there. esc:1,60d esc shift+g -------will take to last line esc:?CREATE --------press n and check if there are 2 sets of create controlfile if yes then bring the cursor to the last line of the wanted set escd shift+g change like this : CREATE CONTROLFILE SET DATABASE "DEV" RESETLOGS NOARCHIVELOG Now change the path of data files and save the file .

esc:%s,oracle/proddata,oradev/devdata,g

The entire practice above is to create a new set of controlfile and below mentioned steps would check if all datafiles have been copied properly:

oradev@ mine02# grep / mine.sql > mine.sh vi mine.sh esc:%s,',,g esc:%s/,//g esc:%s,^,ls -ltr ,g save the file . chmod +x mine.sh ./mine.sh 2> mine.log check the log file and see if there any data file missing.( mine.log file size shud be zero ) At this point we are ready with the control file and we are also sure that all data files have been copied to the target instance. oradev@ mine02# sqlplus "/as sysdba" SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:59:48 2007 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to an idle instance. SQL> startup nomount;

Now open another session and keep monitoring the alert log file.

SQL> @mine.sql

Control file created. SQL>recover database using backup controlfile; Meanwhile go to the source instance and go to alert log file and do a tail -500 and check for the path of the log file using the sequence number asked for recovery. Change the path accordingly and give it for the recovery. SQL> recover database using backup controlfile; ORA-00279: change 7254772161098 generated at 04/20/2007 14:08:25 needed for thread 1 ORA-00289: suggestion : /d02/oradev/devdb/9.2.0/dbs/arch1_88922.dbf ORA-00280: change 7254772161098 for thread 1 is in sequence #88922

Specify log: {<RET>=suggested | filename | AUTO | CANCEL} /d04/oradev/devdata/log4a.dbf Log applied. Media recovery complete. ( In this case the sequence number is 88922, hence check in source instance for the log file path for this sequence number . This will be there in alertlog file. Then change the path of log file according to the target instance and give it. Sometimes this will ask for multiple files with different sequence number)

SQL> select * from v$recover_file; no rows selected SQL> alter database open resetlogs; Database altered. SQL> select * from v$tempfile; no rows selected Check the alert log file while performing the below task. There should be no errors.

SQL> shutdown immediate; Database closed.

Database dismounted. ORACLE instance shut down. SQL> exit Disconnected from Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production oradev@ mine02# sqlplus "/as sysdba" SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 04:26:54 2007 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to an idle instance. SQL> startup; ORACLE instance started. Total System Global Area 1487899976 bytes Fixed Size 732488 bytes Variable Size 973078528 bytes Database Buffers 503316480 bytes Redo Buffers 10772480 bytes Database mounted. Database opened. SQL> exit At this point there shouldn't be any errors in the alertlog. Start the listener: oradev@ mine02# lsnrctl start DEV Then Drop temp tablespace and create new temp tablespace. Then add temp files into it. Change sys and system passwords if required.

OPMN and process monitoring in 10G Application server

OPMN and process monitoring Oracle Process Manager (PM) is the centralized process management mechanism in Oracle Application Server and is used to manage Oracle AS processes. It starts, stops, restarts and detects death of these processes. The Oracle AS processes that PM is configured to manage are specified in the opmn.xml file. The start and stop functions are executed by the OPMN as per the user commands. OPMN (Oracle Process monitoring and Notification services ) are a set of processes that manage mid-tier Application Server components like Oracle HTTP Server (Apache) and OC4J containers. OPMN consists of the Process Manager and the Notification Server. Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup and other related notifications between components in Oracle Application Server. It operates according to a publish-subscribe model: an Oracle AS component receives a notification of a certain type as per its subscription to ONS. When such a notification is published, ONS sends it to the appropriate subscribers. Four parameters determine the behavior of the Oracle Process Manager and Notification services process in managing the iAS middle tier comprising of OC4J instances and the Apache HTTP server. They are a) restart-on-death b) ping timeout c) ping interval d) reverse-ping timeout The settings for these parameters need to be governed by the heap sizing of the OC4J container JVMs, the latencies involved with garbage collection algorithms and the response times of the HTTP server. The way in which the parameters affect the functioning of OPMN is as follows: - OPMN pings the process managed by it,ie, OC4J instances and http server periodically and expects a response within a certain time out period. OPMN would try this 3 times before it declares the process as dead (due to no response) and then it would kill and restart the particular process. OPMN manages the failover and availability of the processes it manages. Reasons for Oracle HTTP Server not responding to OPMN ping could be: - due to a high load of concurrent requests which requires many http server attention. - because of timeouts between the various modules of Apache and the servers servicing client requests - bulk data received from any of the modules. - all new connections are not getting processed due to the server being fully occupied. - it can spawn, with many connections remaining in the CLOSE_WAIT state - due to thrashing when NFS hiccups cause files that need to be served to be unavailable to it - because of synchronization issues with the various mutexes that it needs to support for the proper functioning of its modules.

Reasons for OC4J containers not responding to OPMN ping could be: While the container is processing servlet or EJB logic within the JVM within which it runs, new objects get created in its heap memory area all the while. When the garbage collection thread starts to run, it looks for objects that it can release to the heap memory pool based upon several algorithms that depend upon the kind of references the objects have to themselves. Since the collection is "generational", i.e., objects having references are promoted to an older generation and presumed to have a longer lifetime, objects with weak references are candidates for "cleaning" up, and their occupied memory gets released to the global heap of the JVM. In this way, memory is reclaimed back into the the heap memory pool and made available for use in the creation of newer objects. The forays made by the garbage collector to reclaim memory in the heap are governed by several algorithms and every such collection takes a finite amount of time during which no other work of application processing is possible. And when the collection is over the entire heap, the full GC consists of a mark/sweep/compact cycles that "mark" the memory to be reclaimed, "sweep" the memory into the corresponding generations and "compact" the holes created when the memory is reclaimed, in order as to create contiguous memory for future object creation. These strategies consume more time, as to be expected, and can result in a delay in the container responding to an OPMN ping cycle. During such full GC scans, OPMN can and will kill and restart the container, causing it to lose the state of the application or request it was processing at that time. Since full GC scans (referred to to as stop-the-world scans) can happen at any time during the lifetime of a request or an application, there is always the danger of OPMN killing a perfectly functioning container on the assumption that it was "hung" since it was "unresponsive". Heap memory settings for OC4J containers Changes are made in the opmn.xml file in the <java-option> sections for each OC4J instance: -Xms ( for start heap memory ) and -Xmx ( for maximum heap memory ). The recommended settings for the -Xmx value are 512MB as typical applications need that much of memory to avoid java.lang.OutOfMemoryError exceptions as seen from experience. Start with an -Xms value of 128MB to prevent side effects of "Too many files open" errors when this setting is higher, as garbage collection kicks in only later for higher -Xms values, resulting in open file handles not getting released by the GC.
<java-option>-server -Xms128M -Xmx512M

Thread pool sizing In the server.xml file, set the thread-pool sizes as follows for optimum operation of the thread pool:
<global-thread-pool min="40" max="40" queue="80" keepAlive="-1"/>

This sets the min and max thread-pool sizes to the same value and the keepAlive parameter to "-1" recommended for production environments, this will ensure that idle threads are never destroyed to allow for

thread reuse without the overhead for new thread creation. The min, max and queue values can be left at the default as specified here.
Redundancy and load balancing More than one OC4J instance can be started to accommodate the higher volume of concurrent requests that the container may need to handle. This is set through the "numProcs" parameter in the opmn.xml file and this parameter takes the value of 1 by default, to start a single OC4J instance. For multiple instances, the "numProcs" parameter can be adjusted to different values ( 2 for two instances, and so on ) and PM needs to be restarted with this value for the modules under its control. Very often, the applications that are being run may be process or memory intensive and may require one to adjust the value of the "numProcs" parameter to effect load-balancing via multiple instances.

DMZ setup on Oracle EBS R12

EXTERNAL SERVER SETUP

Environment Name: CRP3

Machine: Database/Conc/Admin Node: minez Internal web/Form Node: minej External Web Node: mine8

Pre-Clone Steps on Internal Server Run adpreclone.pl on MINEJand MINEZ with applcrp3 and oracrp3 users. As oracrp3 user: cd $ORACLE_HOME/appsutil/scripts/CRP3_MINEZ perl adpreclone.pl dbTier As applcrp3 user: cd $ADMIN_SCRIPTS_HOME perl adpreclone.pl appsTier

Take a backup of /d21/applcrp/CRP3 folder on MINEJ Restore the Backup into MINE8 server under /d21 mount point.

Setup Host File: Put the following entries in the Hosts File (/etc/hosts) 10.218.17.36 10.221.15.167 minez.sonapglobal.com minez minej.bn.sonap.net minej.bn

Change the owner Ship of /d21/oracrp3 folder and the file under it to applcrp3 cd /d21 chown R oracrp3:dba oracrp3

Creating External Web Tier Note: Your steps should be more specific than the examples shown.

Create XML file for External Server: su applcrp3 cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin perl adclonectx.pl contextfile=/d21/applcrp3/CRP3/inst/apps/CRP3_minej/appl/admin/CRP3_minej.xml

Enter the APPS password: sonapcrp3 Target System Hostname (virtual or normal) [mine8]: Do you want the inputs to be validated (y/n) [n]? : Target System Database SID: CRP3 Target System Database Server Node [mine8]: minez Target System Base Directory: /d21/oracrp3/CRP3 Target System Forms ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.2]: Target System Web ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.3]: Target System APPL_TOP Mountpoint [/d21/oracrp3/CRP3/apps/apps_st/appl]: Target System COMMON_TOP Directory [/d21/oracrp3/CRP3/apps/apps_st/comn]: Target System Instance Home Directory [/d21/oracrp3/CRP3/inst]:

Username for the Applications File System Owner [applcrp3]: Group for the Applications File System Owner [dba]: Target System Root Service [enabled]: Target System Web Entry Point Services [enabled]: Target System Web Application Services [enabled]: Target System Batch Processing Services [disabled]: Target System Other Services [enabled]: Do you want to preserve the Display [minej:0.0] (y/n)? : n Target System Display [mine8:0.0]: Do you want the the target system to have the same port values as the source system (y/n) [y]? : n Target System Port Pool [0-99]: 1 Choose a value which will be set as APPLPTMP value on the target node [1]: 2 New context path and file name [/d21/oracrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml]: Cross Check if the Context File generated is correct or not, check with following command if the respective components are enabled on MINE8: grep i status $CONTEXT_FILE

Run adpreclone.pl to add the MINE8 server as a node to CRP3: su - applcrp3 cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin perl adcfgclone.pl appsTier /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml Enter the APPS password: Check the logfile for any error.

CONTEXT_FILE configuration: Modify the following CONTEXT_FILE parameters: Context Existing Value File Variabl New Value

e s_applc sf s_applt mp s_appl ptmp s_forms host s_chron osURL s_exter nal_url s_webe ntryhost /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/l ogs/appl/conc /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/t emp /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/p temp mine8 http://mine8.sonapglobal.com:8001/oracle_s mp_chronos/oracle_smp_chronos_sdk.gif http://mine8.sonapglobal.com:8001 mine8 /d21/applcrp3/CRP3/conc /crp3_appltmp /crp3_applptmp crp3ext http://crp3ext.sonapglobal.com:8001/oracle_s mp_chronos/oracle_smp_chronos_sdk.gif http://crp3ext.sonapglobal.com:8001 crp3ext http://crp3ext.sonapglobal.com:8001/OA_HTM L/AppsLogin

s_login_ http://mine8.sonapglobal.com:8001/OA_HTM page L/AppsLogin

Run AutoConfig on al l the Nodes.

Configuring MINE8 for DMZ

Run the script txkChangeProfH.sql for the Profile option setup: # sonapssh minez $ su applcrp3 $ sqlplus appUpdate Hierarchy Types/sonapcrp3 @$FND_TOP/patch/115/sql/txkChangeProfH.sql SERVRESP Run AutoConfig on all nodes.

Update Node Trust Level

To change the value of the Node Trust Level profile option value to External for a particular node, perform the following steps: 1. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL 2. Select the System Administrator Responsibility 3. Select Profile / System 4. From the 'Find system profile option Values' window, select the server and get the valie MINE8 into it. 5. Query for %NODE%TRUST%. You will see a profile option named 'Node Trust Level'. The value for this profile option at the site level will be Normal. Leave this setting unchanged. 6. Set the value of this profile option to External at the server level. The site level value should remain set to Normal

Update List of Responsibility To change the value of the Responsibility Trust Level profile option at the responsibility level for a particular responsibility, perform the following steps:

7. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL 8. Select System Administrator Responsibility 9. Select Profile / System 10. From the 'Find system profile option Values' window, select the responsibility that you want to make available to users logging in via the external web tier 11. Query for %RESP%TRUST%. You will see a profile option named 'Responsibility trust level'. The value for this profile option at site level will be Normal. Leave this setting unchanged. 12. Set the value of this profile option for the chosen responsibility to External at the responsibility level. The site-level value should remain Normal. 13. Repeat for all responsibilities that you want to make available from the external web tier.

List of Responsibilities which can be enabled on External Server is as followed:

Product Name iSupplier

Externally Accessible Responsibilites

Additional Profile Options POS: External URL

POS Supplier Guest User

POS: Internal URL

Plan to Pay Supplier View Plan, Source, Pay Supplier View Source to Pay Supplier View Supplier Profile Manager Procure to Pay Supplier View Oracle Sourcing Sourcing Supplier PON: External Applications Framework Agent PON: External login URL Oracle iProcurement Self Registered Employee Default Responsibility Self Registered New User Default Responsibility

Enable Oracle E-Business Suite Application Server Security

1. Set the value of Application Server Security Authentication (s_appserverid_authentication) to SECURE, in the CONTEXT_FILE on all the nodes. 2. Run AutoConfig on each Applications middle tier to complete the configuration. 3. After AutoConfig completes successfully, restart Oracle HTTP Server and OC4J processe

Increase JVM Size Change the following JVM parameter in the CONTEXT_FILE as mentioned in the below table: Note: Take a backup of Context File before Changing.

Variable s_oacore_jvm_start_options

Exisiting Value -server -verbose:gc -Xmx512M Xms128M XX:MaxPermSize=160M

New Value -server -verbose:gc Xmx1024M Xms521M XX:MaxPermSize=256M

Run Autoconfig

Run AutoConfig from ADMIN_SCRIPTS_HOME sonapng adautocfg.sh.

Enable SSL Login Note:Include a subset of test steps that will confirm that the customization has been installed properly. REFERENCE

Metalink Document: Oracle E-Bsonapness Suite R12 Configuration in a DMZ Document ID: 380490.1

Oracle EBS R12 with servlet forms mode and more

Forms Servlet mode in Oracle EBS R12: By default, Oracle EBS R12 has forms 10g Stack in servlet mode. Now the question is how is the client server connections handled? In Servlet mode we have a java servlet called the forms listener servlet which is responsible for the communication between the Forms java client and the oracle application serverforms services. Unlike Socket mode, the Forms Servlet mode doesnt require an additional forms port to handle communication between client and the application server. The forms servlet architecture operates through the http server port. All forms connections happen through the http port. Hence desktop clients cant access forms directly in servlet mode. The servlet architecture is designed to work with industry standards, and also to be fully supported for advanced network configurations with no additional ports or SSL configuration. In Oracle Applications R12 customers who want load balancing must use the servlet implementation in which forms servers can be balanced across multiple application servers by using a hardware load balancer along with the 10g Application Server. The way to find if the servlet mode/socket mode is implemented: Check the forms configuration file : appsweb.cfg pointed by environment variable"FORMS_WEB_CONFIG_FILE". The entries: serverURL= connectMode= Would be as mentioned below:

In servlet mode

In Socket mode

serverURL=/forms/lservlet connectMode=servlet

serverURL=(should be blank) connectMode=Socket

This can also be verified in the context file in 11i and R12:

context variable s_forms_servlet_serverurl s_forms_servlet_comment s_frmConnectMode

value for Servlet 11i: /forms/formservlet r12: /forms/lservlet (no value, blank) r12: servlet

value for Socket (no value, blank) # r12: socket

We can change the forms servlet mode to socket and how we do it is: 1. 2. 3. Source the environment on the application tier. Stop all the application tier services. Execute :

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \ -contextfile=$INST_TOP/appl/admin/mycontext.xml \ -mode=socket \ -port=9095 \ -runautoconfig=No

Note: context file is the full path of the context file of the instance. Mode is the mode to which the form has to be changed. Port number used to run Forms in socket mode. The default port number is 9095.A port number is not needed if servlet mode is used. Recommended value for autoconfig is NO.

4. 5.

Run autoconfig on apps tier. Start all the application tier services. 6. Check whether Forms Server is running: $INST_TOP/admin/scripts/adformsrvctl.sh status 7. Log in to Oracle Applications and launch a Forms-based application. 8. Open the Sun Java Console (from Tools Menu in Internet Explorer). 9. Check whether the "mode" directive displayed in Sun Java Console when launching formsbased applications is set to socket. 10. The direct launch URL for Forms Socket Mode is: <web_protocol>://<web_host>.<web_domain>:<web_port>/OA_HTML/frmservlet Note: If we need to change the forms mode from socket to servlet mode then all steps remain same except for the script which we run and the verification step:

1.

$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \ -contextfile=$INST_TOP/appl/admin/mycontext.xml \ -mode=servlet \ -runautoconfig=No

2.

No Port number is required as it would use the http port. Verification of servlet mode should reflect:

The "mode" directive displayed in Sun Java Console when launching forms- based applications, is set to http,native Direct Forms Servlet Launch is: web_protocol>://<web_host>.<web_domain>:<web_port>/forms/frmservlet

How to Find version

1. How to find Apps Version (11i/R12/12i) >> Connect to database as user apps SQL> select release_name from apps.fnd_product_groups; Output would be like 12.0.4 or 11.5.10.2 2. Web Server/Apache or Application Server in Apps 11i/R12 >> Log in as Application user, set environment variable and run below: $IAS_ORACLE_HOME/Apache/Apache/bin/httpd version Output for 11i should be like : Server version: Oracle HTTP Server Powered by Apache/1.3.19 (Unix) Server built: Jan 26 2005 11:06:44 (iAS 1.0.2.2.2 rollup 5) Output for R12 should be like : Server version: Oracle-Application-Server-10g/10.1.3.0.0Oracle-HTTP-Server Server built: Dec 4 2006 14:44:38 3. Forms & Report version (aka developer 6i) in 11i >> Log in as Application user, set environment variable and run below query: $ORACLE_HOME/bin/f60run | grep Version | grep Forms Output would be like : Forms 6.0 (Forms Runtime) Version 6.0.8.25.2 (Production) Check fourth character in version 25 which means Forms 6i patchset 16 (25-9)

4. Forms & Report version in R12/12i >> Log in as Application user, set environment variable and run below query: $ORACLE_HOME/bin/rwrun | grep Release Output should be like: Report Builder: Release 10.1.2.2.0 You can safely ignore warnings 5. Oracle Jinitiator in 11i/R12/12i >> Log in as Application user, set environment variable and run below query: grep jinit_ver_comma $CONTEXT_FILE (Default is Java Plug-In for R12/12i ) 6. Oracle Java Plug-in in 11i/R12/12i >>Log in as Application user, set environment variable and run below query: grep plugin $CONTEXT_FILE 7. File Version on file system >>adident Header <filename> Or strings <file_name> | grep Header Here adident is AD Utility (Oracle Apps) and strings is Unix utility 8. Version of pld file >> *.pld are source code of *.pll which are inturn source of *.plx. *.pll is in $AU_TOP/resource and to find its version check adident Header $AU_TOP/resource/<filename>.pll IGSAU012.pll: $Header IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $ Or strings $AU_TOP/resource/<filename>.pll | grep -i header

FDRCSID($Header: IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $); 9. OA Framework Version >> http:// hostname.domainName:port/OA_HTML/OAInfo.jsp (Only for 11i); Log in as Application user, set environment variable and run below query: adident Header $FND_TOP/html/OA.jsp adident Header $OA_HTML/OA.jsp output for both should look like $Header OA.jsp 115.60 2006/03/31 00:47:28 atgops1 noship $ 120.21 means OA Framework Version (coming soon..) 115.60 means OA Framework Version (coming soon..) 115.56 means OA Framework Version (coming soon..) 115.36 means OA Framework Version 5.7 115.27 means OA Framework Version 5.6E 115.26 means OA Framework Version 5.5.2E 10. Discoverer Version for 11i (3i or 4i) >> Log in as Application user, set environment variable and run below query: $ORACLE_HOME/bin/disc4ws | grep -i Version Discoverer Version for 11i or R12 (10g AS) >> Check under Application Server Section as 10g AS Discoverer is on standalone. 11. Workflow Version with Apps >> Connect to Database as apps user SQL> select TEXT Version from WF_RESOURCES where NAME = WF_VERSION; Output like 2.6.0 means workflow version 2.6.0 12. Oracle Single Sign On >> Connect to database which holds SSO repository: SQL>select version from orasso.wwc_version$; 13. Oracle Internet Directory >> There are two components in OID (Software/binaries & Schema/database) >>> To find software/binary version

$ORACLE_HOME/bin/oidldapd version output should look like oidldapd: Release 10.1.4.0.1 - Production on thu sep 11 11:08:12 2008 Copyright (c) 1982, 2006 Oracle. All rights reserved. >>> To find Schema Version/ database use ldapsearch -h <hostname> -p <port> -D cn=orcladmin -w <password> -b \ -s base objectclass=* orcldirectoryversion and output should be like version: 1 dn: orcldirectoryversion: OID 10.1.4.0.1 or Run following query in database SQL> select attrval from ods.ds_attrstore where entryid = 1 and attrname = orcldirectoryversion; Output should be like OID 10.1.4.0.1 14. Application Server >> Oracle Application Server (Prior to Oracle WebLogic Server) If application server is registered in database (Portal, Discoverer) check from database: SQL> select * from ias_versions; Or SQL>select * from INTERNET_APPSERVER_REGISTRY.SCHEMA_VERSIONS; AOC4J (Oracle Container for J2EE) Set ORACLE_HOME cd $ORACLE_HOME/j2ee/home java -jar oc4j.jar -version 15. Oracle Portal >> SQL> select version from portal.wwc_version$; 16. Database Component

I) Oracle Database To find database version: SQL> select * from v$version; Or All component version in database: $ORACLE_HOME/OPatch/opatch lsinventory -detail 17. Oracle Enterprise Manager >> Metalink Note 605398.1: How to find the version of the main EM components 18. Unix Operating System Solaris -> cat /etc/release Red Hat Linux -> cat /etc/redhat-release