Remedy - AR System Server - UDM Job is stuck with "In Progress" and it's step remains in "Queued" status - Related to Java Plugin server connectivity

Knowledge Article

Article Number

000331144

Old Article Number

000120859

Article Type

Solutions to a Product Problem

Title

Remedy - AR System Server - UDM Job is stuck with "In Progress" and it's step remains in "Queued" status - Related to Java Plugin server connectivity

Summary

UDM jobs remains in queued state and giving error

ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect <LB NAME>:<port number>

Product

Remedy AR System Server

Component

AR System

Applies to

BMC Remedy AR System Java Plugin Server - 9.x

Problem

The reported scenario is:
- UDM jobs are stuck in "In Progress" status and their step is stuck in "Queued" state.
- In the AI job there is a transformation which contains 3 steps:
     1. Excel Input.
     2. Get variable.
     3. AR OutPut to the staging form.

When running the DMT (UDM) Job, it remains in "In Progress" and it's step is getting stuck at "Queued".

Checked further by performing these steps too:
1. Clear out the records from DMT:SYS:SequencingEngine form with Parent_Job_GUID = "DO NOT REMOVE" .
2. Rebuild the Sequence table from this form - DMT:SYS:StagingFormDependency .
3. Found one entry for each job in CAI:Events form where status was WARNING .
4. Clear the records from DLD:ThreadManager form.
5. Observed Source and Target Server Name field values in CAI:Events form which holds app load balancer name.

Cause

As per the Error System is looking for a Load Balancer name which was not resolvable over the network network

Solution

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Checking the arjavaplugin.log it was observed multiple errors related to the connectivity from the main Java plugin server.

Example:
................................
...etc...
ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect <{LOAD-BALANCER NAME}>:<{PORT}>
   at com.bmc.arsys.apitransport.ApiProxyJRpcBase.connectionTry(ApiProxyJRpcBase.java:136)
   at com.bmc.arsys.api.ProxyJRpc.getRpcClient(ProxyJRpc.java:148)
   at com.bmc.arsys.api.ProxyJRpc.<init>(ProxyJRpc.java:67)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
   at java.lang.reflect.Constructor.newInstance(Unknown Source)
.. etc....
................................

Steps to perform:
1 Create an entry in the UDM:RAppPassword form containing the Load Balancer name. See RApp Password form for more information
2 Check and if required add in the Hosts file in the OS and entry containing the IP address of the Load Balancer (Short name and FQDN)
Note: If the environment is a Server Group edit the Hosts file on all servers' OS of the group.
3 Restart the AR System application service and run the job again.

Additional article(s):
Remedy - Server - Error:"ERROR (91): RPC call failed; {AR servername}:{AR Port} Connection reset" on CAI:Events running a UDM job

Attachment(s):