How to resolve All Concurrent Requests stuck in pending phase or Some

Concurrent Manager 's status show Target node/queue unavailable?

How to resolve All Concurrent Requests stuck in pending phase or Some Concurrent
Manager 's status show Target node/queue unavailable?

History

1. 2009-03-23 Initialization

Reference

--------

Note:182154.1 Subject:Summary of Possible Reasons and Solutions for the Problem

Where All Concurrent Requests Stuck in Pending Phase

Scope

--------

Oracle Application 11i,Not Test in R12

Possible Reasons and Solutions

--------

1.Check /etc/hosts file to verify it has correct syntax:

<IP address> <hostname.domain> <hostname>

and for one hostname don't have multiple entries.

2.Before starting the concurrent managers using adcmctl.sh,Make sure all concurrent
managers process FNDLIBR have been shutdown at the OS level.

for example:
>ps -ef|grep $USER|grep FNDL
if there still have FDNLIBR processes still running.
>kill -9 PID
PID form output of the command ps in previous step.

3.Changing status of Concurrent Requests from Pending to Inactive.

At form [Administer Concurrent Managers(FNDCPVCM)],select the concurrent manager

which pending rrquests is not zero and click button [Requests] in the lower right corner.
in the pop up windows,recording the request IDs,and then using specific Request in Find
Request window search these request,(notice the field [Select the number of Days to
view] should be set a properly value.),in Requests window,click [Hold Request] to
change the phase of these request to [Inactive] and status to [On Hold] .

if ALL concurrent managers been shown in form [Administer Concurrent

Managers(FNDCPVCM)] with zero or null pending requests,then restart CM,after the
CM processes have been started normally, then in request windows,you can click [Move
Hold] to change the phase of these request back to [pending].

4.Check the value of system profile [Concurrent:GSM Enabled] at Site level.

if the value of [Concurrent:GSM Enabled] is Y,changing it to N and restart concurrent

managers,if concurrent managers can start normally,changing it back to Y and restart CM
again.

5.Make sure customize env file has been added in env file which used in Concurrent
Manager start/stop scripts adcmctl.sh.

>cat $COMMON_TOP/admin/scripts/$CONTEXT_NAME/adcmctl.sh|grep ENVFILE=

if your environment have user defined customize env files,please sure they have been
added in env file used in adcmctl.sh.

6. Relink executable program under $FND_TOP/bin using adadmin.

adadmin-->2. Maintain Applications Files menu-->1. Relink Applications programs

when ask specific products,input:FND
Enter list of products to link ('all' for all products) [all] : FND

7.Rebuild the concurrent manager views. As applmgr run the following from the OS:

This is non-destructive.

Concurrent Manager views can be rebuilt by running the following command at the
command line:

Ensure that concurrent managers are shutdown.

FNDLIBR FND FNDCPBWV apps/apps SYSADMIN 'System Administrator'

SYSADMIN

Restart the concurrent managers.

8.The Profile Option 'Concurrent: OPS Request Partitioning' may be set incorrectly. This
profile option should always be set to OFF, regardless of whether you are running
OPS(RAC) or not, because the profile is obsolete.

9. The System Profile Option: Concurrent Active Requests is possibly to 0.

a. Log into Oracle Applications as SYSADMIN.

b. Select System Administrator responsibility.

c. Navigate to PROFILE > SYSTEM.

d. Query for %CONC%ACTIVE%.

e. Change the profile option for 'Concurrent: Active Request Limit' to Null (blank).

f. Exit Oracle Applications and log in again for the change to take affect.

g. Run a new concurrent request.

10. The Concurrent managers were brought down, while an outstanding request was still
running in the background. In which case, update the

FND_CONCURRENT_REQUESTS table as follows:

sql>

update fnd_concurrent_requests

set status_code='X', phase_code='C'

where status_code='T';

sql> commit;

STATUS_CODE

SQL> select lookup_code,meaning

2 from fnd_lookups

3 where lookup_type = 'CP_STATUS_CODE'

4 order by lookup_code;

LOOKUP_CODE MEANING
--------------- --------------------

A Waiting

B Resuming

C Normal

D Cancelled

E Error

G Warning

H On Hold

I Normal

M No Manager

P Scheduled

Q Standby

R Normal

S Suspended

T Terminating

U Disabled

W Paused

X Terminated

Z Waiting

PHASE_CODE

SQL> select lookup_code,meaning

2 from fnd_lookups

3 where lookup_type = 'CP_PHASE_CODE';

LOOKUP_CODE MEANING

-------------- --------------------

C Completed

I Inactive

P Pending

R Running

11. The control_code for concurrent_queue_name = 'FNDCRM' is 'N' in the

FND_CONCURRENT_QUEUES table, which means 'Target node/queue unavailable'.

This value should be NULL (CRM is running; target and actual process amount are the
same), or 'A' ('Activate concurrent manager' control status).

Set the control_code to 'A' in fnd_concurrent_queues for the Conflict Resolution

Manager:

a. Logon to Oracle Applications database server as 'applmgr'.

b. Verify the Applications environment is setup correctly ($ORACLE_HOME and

$ORACLE_SID).

c. Logon to SQL*Plus as 'APPS' and run the following SQL statement:

update fnd_concurrent_queues

set control_code = 'A'

where concurrent_queue_name = 'FNDCRM';

commit;

d. Verify the status of the concurrent managers through the Concurrent -> Manager
-> Administer form.

If the CRM is still not active, bounce (deactivate, activate) the Internal Concurrent
Manager. This is done through the Concurrent > Manager > Administer form from the
'System Administrator' responsibility. It can also be done through the CONCSUB
command at the command level.

Setting the control_code to 'A' in the fnd_concurrent_queues table for the Conflict
Resolution Manager indicates that this concurrent manager is to be activated with the
parameter values specified through this table for this manager (MAX_PROCESSES,
CACHE_SIZE, etc).

Concurrent Manager [Conflict Resolution Manager] and [Scheduler/Prereleaser

Manager] cannot be Restart or Active in the form [Administer Concurrent
Managers(FNDCPVCM)],

so we need update control_code to do this job.

queue name of Conflict Resolution Manager is FNDCRM.

queue name of Scheduler/Prereleaser Manager is FNDSCH.

Control_code

SQL> select lookup_code,meaning

2 from fnd_lookups

3 where lookup_type = 'CP_CONTROL_CODE'

4 order by lookup_code;

LOOKUP_CODE MEANING

--------------- --------------------

A Activating

B Activated

D Deactivating

E Deactivated

N Target node/queue unavailable

O Suspending concurrent manager

P Suspended

Q Resuming concurrent manager

R Restarting

T Terminating
U Updating environment information

V Verifying

X Terminated

12. What is the cache size? Try increasing the cache size then stop/restart the concurrent
managers.

If concurrent requests are rarely prioritized and there are managers that service short-
running requests, consider setting the cache size to equal at least twice the number of
target processes. This increases the throughput of the concurrent managers by attempting
to avoid any sleep time.

For example:

If more than one manager or worker processes the same type of requests with only a
small cache size, it may be unable to process any jobs in a single processing cycle,
because other processes have already run the cached requests. When this happens, it is
important to note that the manager will sleep before refreshing its cache. To increase
manager throughput where there are sufficient requests of the required type in the queue,
increase the cache size to improve the chance of the manager finding work to process and
thus avoid having to enter a sleep phase.

TIP: Ensure that the system is not resource-constrained before attempting to increase the
rate of concurrent processing in this way, otherwise,

these changes may actually reduce concurrent processing throughput because jobs take
longer to run.

Help of Cache size

Cache Size (Concurrent Manager only)

Enter the number of requests your manager remembers each time it reads which requests
to run. For example, if a manager's workshift has 1 target process and a cache value of 3,
it will read three requests and wait until these three requests have been run before reading
new requests.

In reading requests, the manager will only put requests it is allowed to run into its cache.
For example, if you have defined your manager to run only Order Entry reports then the
manager will put only Order Entry requests into its cache.

If you enter 1, the concurrent manager must look at its requests list each time it is ready
to process another request.
By setting the cache size at a higher number, the concurrent manager does not have to
read its requests list each time it runs a request. However, the manager does not recognize
any priority changes you make for a particular request if it has already read that request
into its cache. Further, even if you give a higher priority to a new request, that new
request must wait until the buffer is empty and the manager returns to look at the requests
list. That request may have to wait a long time if you set the buffer size to a high number.

You should use cache size to tune your concurrent managers to work most efficiently for
you site's needs. If your organization tends to reprioritize jobs going to a certain manager,
that manager should have its buffer size set fairly low.

Suggestion: Enter a value of 1 when defining a manager that runs long, time-consuming
jobs, and a value of 3 or 4 for managers that run small, quick jobs.

13.Concurrent Manager [Output Post Processor] can not be started with Actual is 0 and
Service Info %%service threads running.

Select [Output Post Processor] in Administer and click [Processes] button in lower right
corner,verify [Start Date],if it early than restart date,you should Terminate it first in main
windows and the restart it.

14.OAM Generic Collection Service:HOSTNAME is showing status:Deactivated in

FNDCPVCM and "The target node/queue unavailable" in OAM.

OAM access path:OAM-->Site Map -->Application Services -->Generic Services -->

Select OAM Generic Collection Service -->View Details.

using below sql to verify TARGET_NODE=<Right Node> and

CONTROL_CODE=NULL,if not,update.

SQL> select CONCURRENT_QUEUE_NAME, CONTROL_CODE, TARGET_NODE,

NODE_NAME from FND_CONCURRENT_QUEUES where concurrent_queue_name
like 'OAMGCS_%';

15.if "The target node/queue unavailable" had been shown after cloning or Adding
new node.

>sqlplus apps/appsuat

SQL*Plus: Release 10.1.0.5.0 - Production on Mon Mar 23 15:48:04 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production

With the Partitioning, OLAP and Data Mining options

SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;

PL/SQL procedure successfully completed

SQL> update icx_parameters set session_cookie_domain=NULL;

1 row updated

SQL> commit;

Commit complete

SQL> exit

after execute FND_CONC_CLONE.SETUP_CLEAN,firstly run Autoconfig on ALL DB

tier and then all APP tier.

16.from note:738405.1,if the Internal Concurrent Manager starts, but all others managers
have a status of 'Target queue/node unavailable'.

and the log file of internal Concurrent Manager shows the following errors:

Internal Concurrent Manager found node to be down .

you should check if ping command can be used in OS level.

>ping hostname

17.Note:104541.1 Troubleshooting (Concurrent Manager Generic)

18.Note:105133.1 Concurrent Manager Questions and Answers Relating to Generic

Platform

19.Note:134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent

Manager Tables

@END

The new version of this doc can be find in:http://docs.google.com/Doc?

id=dd9hpwxq_14dhrvqv3z