TrueSight Infrastructure Management (TSIM) on a Linux-based Operating System crashes intermittently with "java.lang.OutOfMemoryError: unable to create new native thread" in TrueSight.log

Below are the steps to increase this limits value for the Linux user used to run the TSIM. Root user access may be required to make these changes:

1. Open the file /etc/security/limits.conf in edit mode
2. Add/Edit the following two parameters to the value 32000:
   • <tsim_user> soft nofile 65536
   • <tsim_user> hard nofile 65536
   • <tsim_user> soft nproc  32000
   • <tsim_user> hard nproc  32000
3. Save the file
4. Logoff all current putty sessions of the TSIM user and login again
5. Run the "ulimit -a" command to verify if the "max user processes" and "max open files" values have been updated

Post the above changes, few changes need to be made in some of the configuration files of the TSIM so as to ensure that the jserver and wildfly processes run with the ulimit settings set for the TSIM user. Below are the steps:

1. Take a backup of <Install_Dir>/TrueSight/pw/pronto/bin/pw file and place the backup file in a different location
2. Open the file <Install_Dir>/TrueSight/pw/pronto/bin/pw file in Edit mode
3. Search for the below line in the file:
   • my $set_ulimit = "ulimit -n 16096; ";
4. Edit this line and update the value as 65536 as shown below:
   • my $set_ulimit = "ulimit -n 65536; ";
5. Save and close the file
6. Take a backup of <Install_Dir>/TrueSight/pw/pronto/lib/perl/PWProcess.pm file and place the backup file in a different location
7. Open the file <Install_Dir>/TrueSight/pw/pronto/lib/perl/PWProcess.pm file in Edit mode
8. Search for the below line in the file:
   • my $set_ulimit = "ulimit -n 4096; ";
9. Edit the line and update the value as 65536 as shown below:
   • my $set_ulimit = "ulimit -n 65536; ";
10. Save and close the file
11. Restart all TSIM services
12. Post successful restart, run the "pw p l" command and note the PIDs for the processes "services" and "jserver"
13. Navigate to location "/proc/<services_PID>" and run command "cat limits":
14. Verify if the below two parameter values are updated as shown below:
    • Max processes             32000                32000                processes
    • Max open files            65536                65536                files
15. Navigate to location "/proc/<jserver_PID>" and run command "cat limits":
16. Verify if the below two parameter values are updated as shown below:
    • Max processes             32000                32000                processes
    • Max open files            65536                65536                files
17. If the updated values are reflected, then the changes are successful

If after increasing the ulimit settings set for the TSIM user the problem is still occurring, check if there are deprecated/decommissioned cells still configured, and if yes, make sure to delete deprecated/decommissioned configured Cells using mdelcell command.