After restarting the system it was observed that the system appeared stable and functioning for a number of hours. And then they receive the following: ............... Wed Aug 23 19:45:04.034 2017 Exception in thread "ActiveMQ BrokerService[broker_0] Task-321" Wed Aug 23 19:45:04.035 2017 java.lang.OutOfMemoryError: unable to create new native thread Wed Aug 23 19:45:04.036 2017 at java.lang.Thread.start0(Native Method) Wed Aug 23 19:45:04.036 2017 at java.lang.Thread.start(Thread.java:714) ............... Wed Aug 23 19:45:04.037 2017 at java.lang.Thread.run(Thread.java:745) ............... Wed Aug 23 19:45:04.051 2017 Error while broadcasting messages to servergroup. Message[UserReleasedMessageData[ messageType=UserReleased userName=webservices cacheId=1276 super.toString()=ServerGroupMessageData [messageType=UserReleased, callType=null, super.toString()=AbstractMessageData [serverNameOfMessageOrigin={servername}, delayOutsideMsgDelivery=false]]]] ERROR (8505): Error sending message through Java Messaging Service; Server group unable to create new native thread ............... Wed Aug 23 19:48:04.349 2017 Exception in thread "ActiveMQ BrokerService[broker_0] Task-320" Wed Aug 23 19:48:04.352 2017 java.lang.OutOfMemoryError: unable to create new native thread Wed Aug 23 19:48:04.354 2017 at java.lang.Thread.start0(Native Method) ............... |
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers. Apply the relevant HotFix for these versions: Version - 9.1.02.004 HotFix: SR_9102P4_20190130_CU_ALL or higher. Location: ftp://ftp.bmc.com/pub/ARRecommendedFixes/Server/9.1.02.004/ Version - 9.1.03 HotFix: SR_9103_20180322_CU_ALL or higher. Location: ftp://ftp.bmc.com/pub/ARRecommendedFixes/Server/9.1.03/ Perform the additional checks/changes: 1.Change the following parameter in arserverd.conf - explicity in bytes Example: From : jvm.minimum.heap.size=8g jvm.maximum.heap.size=8g To : jvm.minimum.heap.size=8589934592 (explicity in bytes) jvm.maximum.heap.size=8589934592 2. Map the JVM Options parameter removing the duplicates and maintaining the sequence of options. Example: jvm.option.1 = -javaagent:../lib/spring-instrument-4.1.9.RELEASE.jar jvm.option.2 = -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl jvm.option.3 = -Dlogback.configurationFile=file:../conf/logback_server.xml jvm.option.4 = -Xss2M jvm.option.5 = -XX:MetaspaceSize=384M jvm.option.6 = -XX:MaxMetaspaceSize=512M jvm.option.7 = -Dcom.sun.management.jmxremote jvm.option.8 = -Dcom.bmc.arsys.boot.flavor=server jvm.option.9 = -XX:OnOutOfMemoryError=./kill-server jvm.option.10 = -Djetty.home=../jetty jvm.option.11 = -Dorg.eclipse.<servername>.http.jetty.autostart=false jvm.option.12 = -XX:ErrorFile=file:../db/arserverjvmcrash_PID%p.log jvm.option.13 = -XX:+HeapDumpOnOutOfMemoryError jvm.option.14 = -XX:HeapDumpPath=../Logs jvm.option.15 = -XX:+UseCompressedOops jvm.option.16 = -XX:+UseConcMarkSweepGC jvm.option.17 = -XX:+UseParNewGC jvm.option.18 = -XX:NewRatio=2 jvm.option.19 = -XX:+DisableExplicitGC 3 .From Remedy Administration Console -> Centralized Configuration -> com.bmc.arsys.server -> Check the parameters: Broker-Max-Memory-Usage 1024 Broker-Max-Temp-Storage 16384 Broker-Send-Failed-No-Space-After-Timeout 30 Broker-Temp-Storage-Directory /remedy/bmc/ARSystem/db/activemq-data/broker_0/tmp_storage 4. Change the ulimit –n to 50000 5. Allow the following ports on each server: 1099 = RMI port required by ActiveMQ (not configurable). 61617 = JMS broker port for ActiveMQ (Default-messaging-port). 40001 = Port on which the RMI Registry listens for ehcache (Peer-listener-port). 40002 = Port on which the remote objects bound in the registry receive calls for ehcache (Peer-Remote-Object-Port). 10001 = Shutdown port (not configurable). 61500 = Port for JMX connections (Jmx-port). Doc-Link: https://docs.bmc.com/docs/ars91/setting-ports-and-rpc-numbers-609073765.html |