How can I collect Java Heap and Thread Dumps for Smart IT while troubleshooting stability or performance issues

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

While troubleshooting stability or performance issues of the Smart IT Application, BMC Support may request a customer to create Java Heap or Thread Dumps of Smart IT’s Tomcat server.

Examples of situations where this information may be requested:

Heap Dump: To analyze OutOfMemory errors in Tomcat logs (e.g. the Heap Memory usage may have a constant increasing trend, eventually leading to all available Heap space being used, resulting in OutOfMemory errors and the Application to stop functioning properly)

Thread Dump: To analyze situations where either specific use cases are performing poorly, or there is a sudden general performance issue for the whole Smart IT Application (and where the performance issue cannot be explained by slowness of services that Smart IT depends on, i.e. AR Server, Smart IT DB, RSSO are found to be responding in a timely manner). In these situations, high CPU usage but stable memory consumption may be observed for the Tomcat process.

The VisualVM application can be used to create such Java Heap and Thread Dumps. The suggestion would be to install this client on a separate (Windows) machine and connect from there to the target Smart IT Tomcat Server. To allow VisualVM to connect to Smart IT’s remote Java Virtual Machine (JVM), some additional parameters need to be configured for Tomcat. Instruction to obtain VisualVM and on how to configure Tomcat are provided below.

Notes:

• There are also command line alternatives to create Thread and/or Heap dumps. AR Platform Knowledge Article 000248856 lists several of those options (in the Smart IT case, you would use the Process ID (PID) of the Tomcat process rather than the AR Server Java PID).
• In case (frequent) crashes of the Smart IT Tomcat Server are observed, the suggestion would be to add the following parameter to Tomcat’s Java/Catalina options:

VisualVM client installation
Preparing Smart IT’s Tomcat server to allow JMX connections

The information below includes the basic setup to enable JMX connections (e.g. without authentication or ssl). While this may be sufficient in a local network where access is restricted, do consider if further security settings for JMX access should be enabled, especially in case the server/port can be accessed from remote locations (see also: Apache Tomcat Security Considerations). 

Find a free port on the Smart IT Server (port 9010 is used in the example below).

1. In Windows Explorer, open Tomcat‘s ‘bin’-folder
2. Open the Tomcat monitor application by double clicking its executable – if the Tomcat Service name would ‘SmartITTomcat8’, the executable would name would be ‘SmartITTomcat8w.exe’
3. Go to the ‘Java’-tab and add the following lines in the ‘Java Options’ section (note: update the port and IP address as required):

4. Confirm the configured remote port is available to accept requests – e.g. from the command line, execute:

 

1. Go to Tomcat’s 'bin'-folder
2. Create a backup of the setenv.sh file, then edit the file to add the following line (update the port number and IP address as required):

3. Restart Tomcat
4. Confirm the configured remote port is available to accept requests – e.g. from the command line, execute:

Creating Heap and Thread Dumps using VisualVM

1. Open the VisualVM application
2. Right-click on ‘Remote’ and select ‘Add Remote Host…’

   

3. Enter the Host name of the Smart IT server in the Pop-up and click ‘Ok’
4. Right-click on the Remote host that was added and select ‘Add JMX Connection…’

   

5. In the dialog, enter the port number that was configured in Tomcat’s Java properties and press ‘Ok’

6. Double click on the new entry under the Remote host name to open the connection
7. If a Heap dump is what is requested, go to the ‘Monitor’-tab, click ‘Heap Dump’ and confirm the folder on the remote host (i.e. Smart IT Server) where the Heap Dump (.hprof file) will be saved:

8. If Thread Dumps are requested that capture a specific (poor performing) use case, or a general sudden performance issue, perform the following steps:

• Enable Smart IT Debug logging (if required, check the steps in point 3 of Knowledge Article 000225149 on how to enable this)
• Open Smart IT in the browser and access the Developer Tools (F12) to capture network traffic (for details on how to set this up, please refer to this document)
• In VisualVM open the ‘Threads’-tab and locate the ‘Thread Dump’ button
• Trigger the use case in the browser and as it starts executing, click on ‘Thread Dump’ in VisualVM:

• When the use case completes, save the F12 Network logs to .har file and from the Smart IT server, ZIP the following log files:
  • <tomcat>\logs\localhost_access_log.<date>.txt
  • ..\Smart_IT\Logs\smartit.log

• The Thread Dumps will show-up under the JMX connection in VisualVM. Right-clicking on the JMX Connection will give the option to save an ‘Application Snapshot’ that includes all Thread Dumps:

• Then Right-click on the newly created Snapshot to save it to disk on client side (.apps format):

• Share the Application Snapshot (.apps) file, .har network file, as well as the indicated logs from the Smart IT server for analysis.

See also: BEST FAQ on Remedy for Smart IT Performance Issues