Remedy - Server - AR Server 9.x not starting up - Error encountered in bundle: com.bmc.arsys.messaging

Product

Remedy AR System Server

Component

AR System Server

Applies to

ARS 9.x

Problem

Remedy Server unable to startup.

Contents from arerror.log:

AR System Assignment Engine 9.1.00 201610070720(c) Copyright 1991-2016 BMC Software, Inc.
CMDB Server Starting
CMDB Server Started.
Thu Jan 19 16:17:48.958 2017 Server binding to TCP port servername/10.85.128.187 on assigned port 37000
Action Request System Server Started.
AR Server Startup Notification Error
Extension loaded: com.bmc.arsys.alertws 9.1.0.SNAPSHOT ARSYS.ALRT.WEBSERVICE
Extension loaded: com.bmc.arsys.archiveddatamanagement 9.1.0.SNAPSHOT ARSYS.ADM.FLT.API
Extension loaded: com.bmc.arsys.serveradmin 9.1.0.SNAPSHOT REMEDY.ARDBC.SERVER.ADMINISTRATION
Extension loaded: com.bmc.arsys.webservice2 9.1.0.SNAPSHOT ARSYS.ARF.WEBSERVICE
Extension loaded: com.bmc.arsys.server 9.1.0.SNAPSHOT com.bmc.arsys.produse
Extension loaded: com.bmc.arsys.assignmentEngine 9.1.0.SNAPSHOT BMC.ARS.ASSIGNMENTENGINE
Extension loaded: com.bmc.cmdb 9.1.0.SNAPSHOT bmc.cmdb.cmdbEngine
Error encountered in bundle: com.bmc.arsys.messaging
Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=com.bmc.arsys.messaging, config=bundleentry://163.fwk1456208737/OSGI-INF/blueprint/application_context.xml))
Unable to create application context for [com.bmc.arsys.messaging], unsatisfied dependencies: none
TID 58 initTransport Normal ARServer shutdown process Initiated.
Error encountered in bundle: com.bmc.arsys.messaging
Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=com.bmc.arsys.messaging, config=bundleentry://163.fwk1456208737/OSGI-INF/blueprint/application_context.xml))
Unable to create application context for [com.bmc.arsys.messaging], unsatisfied dependencies: none
TID 57 initTransport Normal ARServer shutdown process Initiated.
Error encountered in bundle: com.bmc.arsys.messaging

Contents from ardebug.log:

Thu Jan 19 16:42:48.483 2017 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localBroker' defined in OSGi resource[bundleentry://163.fwk1456208737/OSGI-INF/blueprint/application_context.xml|bnd.id=163|bnd.sym=com.bmc.arsys.messaging]: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Port value out of range: -1
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
   at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
   at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
   at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
   at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
   at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762)
   at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
   at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
   at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
   at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Port value out of range: -1
   at java.net.ServerSocket.<init>(ServerSocket.java:232)
   at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
   at org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:132)
   at org.apache.activemq.transport.tcp.TcpTransportFactory.doBind(TcpTransportFactory.java:56)
   at org.apache.activemq.transport.TransportFactorySupport.bind(TransportFactorySupport.java:40)
   at org.apache.activemq.broker.BrokerService.createTransportConnector(BrokerService.java:2327)
   at org.apache.activemq.broker.BrokerService.addConnector(BrokerService.java:316)
   at org.apache.activemq.broker.BrokerService.addConnector(BrokerService.java:306)
   at com.bmc.arsys.messaging.data.MessageBroker.getBroker(MessageBroker.java:128)
   at com.bmc.arsys.messaging.data.MessageBroker.init(MessageBroker.java:98)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:497)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1696)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1635)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1564)
   ... 14 common frames omitted

Cause

Remedy Server unable to successfully perform DELETE SQL statements due to blocks on specific table by inactive Oracle Sessions.

Solution

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Enable SQL logging manually by modifying ar.cfg/ar.conf and setting the following parameter

Debug-mode: 1

You can also set the location and name of the log files using this parameter.

SQL-Log-File:

Optionally if the parameter is not defined then SQL log file name will be arsql.log located in the 'db' directory of the remedy server installation.

Restart the remedy server. If the server is restarting multiple times check to see if the last SQL statement encountered in the remedy SQL log file is the same one. If the statement is an UPDATE, INSERT or DELETE, have DBA check to see if the table/record is being blocked perhaps by another process or inactive processes.

DBA can terminate the process.

If the issue is still not resolved provide the arerror.log, ardebug.log and arsql.log to BMC Support. Recommendation is to use the remedy maintenance utility located the the ..ARSystem/arsystem directory. Choose the 2nd to last option 'Zip Logs'.

For more details on the utility:
https://docs.bmc.com/docs/display/public/ars91/Running+the+Maintenance+Tool

Knowledge Article