Knowledge Article
Remedy - AR System Server - How to collect Java heap dumps (and thread dumps) for AR Server troubleshooting - ARS 9.x - INCLUDES VIDEO
Remedy - AR System Server - How to collect Java heap dumps (and thread dumps) for AR Server troubleshooting - ARS 9.x - INCLUDES VIDEO
Remedy AR System Server
Remedy AR System Server
BMC Remedy AR System Server 9.x
Support has asked for heap and thread dumps from AR Server (9.x and above) - how can these be collected? |
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers. Note: VisualVM has also been distributed in Oracle JDK 6-8 as Java VisualVM. It has been discontinued in Oracle JDK 9 onwards so JVisualVM does not come bundled in later versions of Java JDK. If you are using a more updated Java JDK greater than version 9 please check the following article to implement VisualVM: Remedy - AR System Server - How to install and run VisualVM with latest Java JDK versions Please see attached PDF document, showing an example of using JVisualVM (in this case, executed on a Windows machine) to collect heap dumps, thread dumps and application snapshots. For non-Windows environments, you can run JVisualVM from a remote Windows machine and point to the Unix system running ARServer Perform the following steps to create and prepare files to send to Support: 1. Install Java SDK (JDK) on the system you will perform the steps on. NOTE: This system is typically a Windows system other than the AR Server system. 2. View the ar.cfg (conf) or use the Centralized Configuration UI to check the Jmx-port setting. a. The default is 61500 3. View the arserver.config/arserverd.conf file to ensure the following settings are present. Note that the exact option number is not critical: a. jvm.option.7=-Dcom.sun.management.jmxremote b. jvm.option.14=-XX:+HeapDumpOnOutOfMemoryError c. jvm.option.15=-XX:HeapDumpPath=./Logs Note: - For Windows: the file is called arserver.config and it is located in ....\BMC Software\ARSystem - For Unix/Linux: the file is called arserverd.conf and it is located in .../ARSystem/bin 4. Ensure that the AR System server's system drive (C:\ on Windows) contains at least enough free diskspace for each heap dump you take. Each dump will be roughly the size of the Arserver Java memory used; so if ARServer Java process takes 6GB and you are taking 4 dumps, ensure that you have more than 24 GBs available plus some extra room.
5. Launch JVisualVM from the JDK's bin directory. 6. Click on File / Add JMX Connection... a. Connect to the ARServer host using the Jmx port. b. The Connection string will look like: service:jmx:rmi:///jndi/rmi://<server Name>:<jmx port>/ARServer For example: service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer
d. Check the box to no require SSL e. Click OK to connect 7. Double-click on the new Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer (pid 8504) 8. Click on the Monitor tab 9. Click on the Heap Dump button. This will generate a large .hprof file on the AR Server system. a. You will either get an option to save the dump or it will open it up in a new tab. 1. If you get prompted, Provide the full path and filename of the dump file. 2. Otherwise, if the heap dump opens in a new tab, look at the navigation pane on the left for this new dump, right-click on it and choose Save-As 10. Click OK to continue. 11. Click on the Threads Tab. 12. Click on the Thread Dump button to generate a dump of thread activity. 13. Repeat step 12 several times to capture multiple Thread Dumps Support needs multiple thread dumps so that we can identify possible trends in thread utilization
14. Right-click on the Remote Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer (pid 8504) a. Select Application Snapshot b. This will generate a "point in time" snapshot of the environment details and thread dumps to a single file c. This does not include the .hprof files that are on the AR Server system d. In some cases, you will need to generate an Application Snapshot from a large timeframe, often 24 hours 1. To change interval to 24 hours from the default: Choose “Tools | Options” and enter 1,440 for Charts Cache 2. After sufficient data is collected, select the node in the navigation tree, right-click, and choose “Application Snapshot” 15. Right click on the new Snapshot items and choose Save As to create a .apps file. This will save to the local drive that you are running JVisualVM from. 16. Gather, Zip, and send to Support the following files: a. The .hprof files created in step 9 Note: These files will be on the AR System server system b. The .apps file created in step 15 Note: This files will be on the system from which you are running JVisualVM Other methods: You can alternatively use the jstack and jmap utilities if there is a JDK installation on the AR System server. The utilities are located under the JDK bin directory. In some cases the jvisualvm may not respond and using the command line versions would be the next option. These are command line utilities. Below are examples of the command syntax to use where you redirect the output to a file. The pid will be the process ID of the remedy server java process. Look for the Java with the java heap parameters and also referencing the SNAPSHOT jar file as in the example below. In some cases the '-F' parameter will encounter an error when executing the command. If this occurs just omit the -F. The 2nd parameter with jstack below is the lower case letter l. jstack -F -l [pid] > stack.log Example: jstack -F -l 1234 > stack.log jmap -dump:live,file=[dump_file_name] [pid] Example:jmap -dump:live,file=/opt/tmp/AddressBook-heapdump.bin 37320
Remedy Server java process: /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/bin/java -Xms8589934592 -Xmx8589934592 -cp ./../lib/com.bmc.arsys.boot-9.1.02-SNAPSHOT.jar Note: Even after zipping the files, they may be rather large. You may need to connect to ftp.bmc.com, create a directory under the 'incoming' directory, upload the files and alert Support to the location and filename(s). Please follow the steps explained in the following article on how to use BMC's FTP site: BMC Remedy How to upload files on ftp.bmc.com Contact BMC Support if you require any assistance to access/create a FTP directory in the BMC FTP server. Related Article(s) Remedy - Email Engine - v9.1.x How to capture Thread and Heap dump for Email Engine process (Windows) Online Documentation - JVM Monitoring |
Attachment(s):