Remedy AR System Server - Connection refused errors (ARERR 90) ERROR (90)

Knowledge Article

Remedy AR System Server - Connection refused errors (ARERR 90) ERROR (90)
ERROR (90): Cannot establish a network connection to the AR System server
Remedy AR System Server
Remedy AR System Server
Remedy AR System Server 7.x, 8.x, 9.x and higher

This specific error message can be observed by several odd behaviors, but it is always because a component (Mid-Tier, plugin server, Smart IT, etc.) is unable to set a network connection to AR System Server.  Here is a sample error message:
 

Connects to <arservername>:<arport> through Java Rpc failed with: ERROR (90): Cannot establish a network connection to the AR System server; <Message>: <arservername>:<arport>

Some examples of behaviors include, but are not limited to:
  1. Mid-Tier, SmartIT, or other client presenting a "AR Server is Not Reachable" message
  2. A plugin not working at all (Like Approvals, LDAP, Email or other) "ERROR (90): Cannot establish a network connection to the AR System server; ONC/RPC program not registered"
  3. A component not registering with AR Server (like File Deployer or Flashboards).
You can confirm these errors actually relate to the message above by reviewing the clients/plugin's logs.  Examples would include:
  1. armonitor.log
  2. arfiledeployer.log
  3. arjavapluginsvr.log
  4. aremail.log
  5. armidtier.log
This is always caused by the server not responding to the client or rejecting the connection. Usually due to a miss-configuration of the OS or AR Server config.
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

NOTE: This will often require collaboration from a network engineer and a system engineer along with the Remedy admin.

There are several tools that you can utilize to help troubleshoot a ARERR 90 message.
  1. Ping - The ping tool is a common tool included with both Windows and Linux based systems.  You can use this to ensure that the server is reachable and resolvable.  If a ping doesn't work, then a AR Client will not be able to communicate with AR Server.
  2. Telnet - The Telnet can be used to verify if firewall rules or other routing issues are impeding operations of AR Server.  If you're not able to telnet from the client machine to the server machine on the AR Server port, then a AR Client will not be able to communicate with AR Server.
  3. Netstat - The netstat tool can be used server side to ensure that the AR Server process is actually listening on the expected port using the expected network interface.  If the server is not listening then no client will be able to connect.
NOTE:  BMC will not provide support for these third party tools.  If you need help installing, managing, or operating them, please reach out to your OS vendor or System Admins.  Other tools can be used in place of these common tools, like "Visual Ping", as per your organizations requirements.

NOTE:  Keep in mind AR Server (as well as all clients) use the OS itself to manage the networking calls, so while these third party tools like ping and telnet won't solve a issue with connectivity for you it can be very helpful to help narrow the scope of the issue and identify a weak point.  AR Server must be resolvable, available, and connectable from the client machine before you'll be able to connect with a AR Client.

Please check the following configuration files to ensure that the correct server name and port is used in all instances (other config files could be used depending on the product you're observing a issue with):
  1. armonitor.conf
  2. ar.conf
    1. Server-Connect-Name: <FQDN of AR Server>
    2. Domain-Name: <Domain Name>
  3. AREmail/emailDaemon.properties
  4. flashboards/server.conf
  5. midtier/web-inf/classes/config.properties

You also might need to check the system host file, hostname file, and other system config files.  Additionally check any DNS entries and load balancer configurations that are being used.

Some common issues are:
  1. misspelled server names
  2. miss-configured ip/dns entries
  3. case sensitive DNS/hostname entries
  4. firewall blocking ports
  5. wrong ports being used
Another common troubleshooting step is to eliminate any network appliances, like load balancers, from the system and point clients directly to a server.  This is especially important when you have downtime in production.  It is often better to for-go load balancers and point clients directly to servers and get users back up and running first, then worry about integrating load balancers and other appliances back into the system later.
 
Also, please review our documentation on ARERR 90 here:
https://docs.bmc.com/docs/ars2002/error-messages-1-to-99-909638815.html

Please see the following KA's for additional information for other Remedy products:
AR System Clients: https://communities.bmc.com/docs/DOC-76397
AR System Mid-Tier: https://communities.bmc.com/docs/DOC-86903
Remedy AR System Server
Remedy AR System Server
Remedy AR System Server 7.x, 8.x, 9.x and higher
Remedy AR System Server
Remedy AR System Server
Remedy AR System Server 7.x, 8.x, 9.x and higher
Attachment(s):