Armonitor process crashes however, the child processes still running (arserver, arplugin, Java plugin server....)
The reason being, any armonitor child process will now be owned by init (process id 1).
Bringing down armonitor does "not" bring down arserver
Looking at the arsystem script it can be seen that the armonitor it is stopped using:
${JAVA_HOME}/bin/java -Djava.lib.path=. -classpath ${ARMONITOR_LIB} com.bmc.arsys.armonitor.ARMonitorAdmin stoparmonitor "$@"
However, since ARmonitor is not running and when trying to stop arsystem the following error is observed:
...........................................
./arsystem stop
Action Request System shutdown in progress.
ARMonitorJ might not be running, Remote exception - Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at java.rmi.Naming.lookup(Naming.java:101)
at com.bmc.arsys.armonitor.ARMonitorAdmin.processRMICommands(ARMonitorAdmin.java:84)
at com.bmc.arsys.armonitor.ARMonitorAdmin.main(ARMonitorAdmin.java:38)
...........................................
The child process does not close, since the arsystem script stops there and does no further checks for child processes.
For example:
Using grep for arserver i.e
ps -elf |grep arserver
4878 1 80 0 - 2992310 futex_ Mar08 ? 00:44:43 /usr/java/jdk1.8.0_45/jre/bin/java -Xms8589934592 -Xmx8589934592 -cp ./../lib/com.bmc.arsys.boot-9.1.02-SNAPSHOT.jar -XX:OnOutOfMemoryError=./kill-server -Dcom.bmc.arsys.boot.flavor=server -Dcom.sun.management.jmxremote -XX:MaxMetaspaceSize=512M -XX:MetaspaceSize=256M -Xss2M -Dlogback.configurationFile=file:../conf/logback_server.xml -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl -javaagent:../lib/spring-instrument-4.1.9.RELEASE.jar -XX:NewRatio=2 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCompressedOops -XX:HeapDumpPath=../Logs -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=file:../db/arserverjvmcrash_PID%p.log -Dorg.eclipse.equinox.http.jetty.autostart=false -Djetty.home=../jetty com.bmc.arsys.boot.Main -s {AR_Server_name} -i
Notice that init now owns the process for arserver,( process id 1) in above statement.
If we try start arserver, ie:
./arsystem start
The following error appears:
...............................................
INFO - ******************************************************************************************
INFO - Initializing ARMonitor Version 9.1.02 201611252320
INFO - ******************************************************************************************
INFO - 172.22.239.172
INFO - Binding of armonitor on port [ 7319 ] is successful.
INFO - Initializing ProcessController, to start and monitor all the processes listed in armonitor configuration file.
WARN - Duplicate entry found for process - /usr/java/jdk1.8.0_45/jre/bin/java -jar /opt/bmc/ARSystem/bin/arserver.jar -s clm-aus-018184 -i /opt/bmc/ARSystem -l /etc/arsystem/{AR_Server_Name}
WARN - Duplicate entry found for process - /usr/java/jdk1.8.0_45/jre/bin/java -jar /opt/bmc/ARSystem/bin/arserver.jar -s clm-aus-018184 -i /opt/bmc/ARSystem -l /etc/arsystem/{AR_Server_Name}
INFO - Read and start all processes listed in armonitor.conf.
INFO - Starting Server-Process : /usr/java/jdk1.8.0_45/jre/bin/java -jar /opt/bmc/ARSystem/bin/arserver.jar -s clm-aus-018184 -i /opt/bmc/ARSystem -l /etc/arsystem/{AR_Server_Name}
INFO - ProcessMonitor starting process [ ARSERVER ].
INFO - Server port socket listener initializing..
INFO - Server Statup Notification Port [ARSERVERPORT] :34464
INFO - Fri Mar 10 07:47:59.879 2017 RestApiActivator:starting bundle...
INFO - Fri Mar 10 07:47:59.959 2017 Restapi bundle started.
INFO - Registering CMDB REST API @ /cmdb ...
INFO - Registered CMDB REST API
INFO - Fri Mar 10 07:48:15.081 2017 Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=com.bmc.arsys.messaging, config=bundleentry://158.fwk1018547642/OSGI-INF/blueprint/application_context.xml))
INFO - org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localBroker' defined in OSGi resource[bundleentry://158.fwk1018547642/OSGI-INF/blueprint/application_context.xml|bnd.id=158|bnd.sym=com.bmc.arsys.messaging]: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Port value out of range: -1
INFO - at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[bundlefile:na]
INFO - at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) ~[bundlefile:na]
INFO - at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) ~[na:na]
INFO - at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60) ~[bundlefile:2.0.0]
INFO - at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325) ~[bundlefile:2.0.0]
INFO - at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85) ~[na:na]
INFO - at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290) ~[bundlefile:2.0.0]
INFO - at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137) [bundlefile:2.0.0]
INFO - at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
INFO - Caused by: java.lang.IllegalArgumentException: Port value out of range: -1
INFO - at java.net.ServerSocket.<init>(ServerSocket.java:232) ~[na:1.8.0_45]
INFO - at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231) ~[na:1.8.0_45]
INFO - at org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:132) ~[na:na]
INFO - at org.apache.activemq.transport.tcp.TcpTransportFactory.doBind(TcpTransportFactory.java:56) ~[na:na]
INFO - at org.apache.activemq.transport.TransportFactorySupport.bind(TransportFactorySupport.java:40) ~[na:na]
INFO - at org.apache.activemq.broker.BrokerService.createTransportConnector(BrokerService.java:2327) ~[na:na]
INFO - at org.apache.activemq.broker.BrokerService.addConnector(BrokerService.java:316) ~[na:na]
INFO - at org.apache.activemq.broker.BrokerService.addConnector(BrokerService.java:306) ~[na:na]
INFO - at com.bmc.arsys.messaging.data.MessageBroker.getBroker(MessageBroker.java:128) ~[na:na]
INFO - at com.bmc.arsys.messaging.data.MessageBroker.init(MessageBroker.java:98) ~[na:na]
INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
...............................................