This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.
Based on the environment set up rundashboardutil.cmd or rundashboardutil.sh should be considered in the solution.
Cause 1: Incorrect JAVA path in the rundashboardutil.cmd/rundashboardutil.sh file is one of the causes
Solution 1:
1. Correct the java path in the file:
\AtriumCoreHome\cmdb\utils\DashboardDataGenerator\rundashboardutil.cmd/rundashboardutil.sh
2. Run the Dashboard and Data source schedules manually.
\AtriumCoreHome\cmdb\utils\DashboardDataGenerator\rundashboardutil.cmd DATASOURCES
\AtriumCoreHome\cmdb\utils\DashboardDataGenerator\rundashboardutil.cmd KPI
** Considerations**
For UNIX run the following way:
./ AtriumCoreHome/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh DATASOURCES
./ AtriumCoreHome/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh KPI
For 21.3:
./ AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh DATASOURCES
./ AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh KPI
-----Additional guidelines to troubleshoot such issues:
1. CMDB:Dashboard_Utility_Run_History
2. CMDB:Dashboard_ScheduleManager
-
Schedules are created in the ScheduleManager form, there should be at least 1 schedule for KPI and one for DataSources. Double-check these first.
-
Then also make sure that there is no "running" job in the Utility_Run_History. You can safely delete all the records in this form.
-
You can check if the Escalation: "CMDB:SHM:Schedule_RunSchedule" is enabled and running ok
Cause 2: Variable or hardcoded values in rundashboardutil.cmd or rundashboardutil.sh
Solution 2: Check file rundashboardutil.cmd and confirm if the below parameters have variables or hardcoded values:
CMDB_VERSION
CMDBVERSION
AR_VERSION
If variable then update with hardcoded value. Below places can be referred to get the version of parameter.
--For CMDB_VERSION= %ATRIUMCORE_HOME%\cmdb\utils\DashboardDataGenerator\dashboarddatagenerator-%CMDB_VERSION%-SNAPSHOT.jar
--For CMDBVERSION=%ATRIUMCORE_HOME%\cmdb\sdk\bin\cmdbapi%CMDBVERSION%.jar
--For AR_VERSION=%BMC_AR_SYSTEM_HOME%\Arserver\api\lib\arcmnapp%AR_VERSION%.jar
Cause 3: Specific pool number not defined for the escalation
Solution 3:
In Remedy Developer Studio, add a pool number for the escalation: CMDB:SHM:Schedule_RunSchedule if there is contention with other escalations i.e. you see a delay in when the escalation runs.
See KA: 000318190 for how to add a pool number to any escalation.
Cause 4: ERROR (5430): Invalid Run Process Command
Solution 4:
NOTE** This is only relevant to 21.03 patch 5 (21.03.05) and later.
An error can occur in the API/FLTR/SQL pertaining to the whitelist process:
An application command failed.ERROR (5430): Invalid Run Process Command. Running non-whitelisted commands via Run Process action is not supported.; "/opt/bmc/AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh" KPI
To check the Run-Process-Command-Whitelist parameter:
1.) Click on the Home Page Flyout menu :
Applications > AR System Administration Console > System > General > Centralized Configuration
2.) To see the Select Run-Process-Command-Whitelist parameter, choose Component Type: com.bmc.arsys.server.shared > *
In one use-case, the Run-Process-Command-Whitelist parameter was set to:
./AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.cmd|../AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh|"../artools/arexport.sh"|"./artools/arexport.bat"|"pdp.sh"|"./artools/pdp.bat"|"remoteaction/remoteaction.sh"|"remoteaction\remoteaction.bat"|"DMLDataImport.bat"|"DMLDataImport.sh"|"/opt/bmc/ARSystem/db/mkdir.sh"
However, the first command is rundashboardutil.cmd and it doesn't have double-quotes. All other values in the whitelist parameter contain double-quotes.
The string has to be corrected and then the whitelist will accept and run the rundashboardutil.cmd.
In the solution for the test use-case, the issue is in a container, the following change was made to the Run-Process-Command-Whitelist parameter:
"/opt/bmc/AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh"|"../artools/arexport.sh"|"./artools/arexport.bat"|"pdp.sh"|"./artools/pdp.bat"|"remoteaction/remoteaction.sh"|"remoteaction\remoteaction.bat"|"DMLDataImport.bat"|"DMLDataImport.sh"|"/opt/bmc/ARSystem/db/mkdir.sh"
Note that after applying Solution 4, it was found that Solution 3 above was also necessary. Once Solutions 3 and 4 were performed, the Dashboard utility would run daily as expected.
If the above steps do not resolve the issue then as an additional Step you can run the rundashboardutil.cmd utility from the Command prompt and check if it gets triggered.
In one of the use case we could see the below Error:
In order to resolve this Error extract the dashboarddatagenerator-9.1.10-SNAPSHOT jar and check if DashboardDataGenerator file is present under com/bmc/cmdb/ui/dashboard folder.
If it is missing replace it from the other working environment.Cause 5: Escalation is not enabled
Solution 5:1. Check the AR System Server Group Operation Ranking form
2. Check the Operation for "Escalation"
3. Verify that the Rank1 is the same like Current Operation Owner
- Applications> AR System Management Console> AR System Server Group Console
- On left side select Current Operation Owner
4. Click on the Home Page Flyout menu Applications > AR System Administration Console > System > General > Centralized Configuration. Choose Component Type: com.bmc.arsys.server.shared > *
Make sure Disable-Escalation-Global is set to "F"
5. Run again the rundashboardutil.sh or rundashboardutil.cmd command and validate the "CMDB:Dashboard_ScheduleManager" form.
Cause 6: Java heap memory in KPI or Datasource dashboard
Solution 6:
The logs can be found in the ARSystem/db path based on the installation path and on the server wherever escalation is ranked 1
DashboardDataGenerator_DATASOURCES_xxx.log
DashboardDataGenerator_KPI_xx.log
Sample errors details
[ java.util.concurrent.ExecutionException ] Thread - 40 : Could not get restuls from CI Completeness processors: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
java.util.concurrent.ExecutionException ] Thread: 57 Error while waiting main reader to complete for class BMC.CORE:BMC_Product: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
For this in rundashboardutil.cmd or rundashboardutil.sh (in case of Linux environment) file udpate the memory parameter.
DASHBOARD_UTIL_JAVA_OPTIONS=-Xmx1536m
to
DASHBOARD_UTIL_JAVA_OPTIONS="-Xms512m -Xmx4096m"
The file has to be placed in the server. /opt/bmc/AtriumCore/cmdb/utils/DashboardDataGenerator/rundashboardutil.sh (linux)
Restart all platform servers
Cause 7: Performance issue with huge CI data.
<TrID: > Thread: 53 Completed processing class BMC_STORAGESUBSYSTEM Found 0 [10 Jun 2024 21:05:51,833] [ERROR] : <TrID: > [ com.bmc.arsys.api.ARException ] Thread - 40 : Could not fetch total number of instances for class BMC.CORE:BMC_BaseRelationship in BMC.ASSET dataset: ERROR (552): The SQL database operation failed.; terminating connection due to administrator command
<TrID: > [ com.bmc.arsys.api.ARException ] Thread - 18 : Could not update heart beat field: ERROR (552): The SQL database operation failed.;
Solution 7:
As a workaround, we excluded BMC_PRODUCT class from completeness KPI run as it had multiple errors
in DashboardDataGenerator_KPI log
A defect is logged DRD21-102929 to check the performance issue.
How to fetch Dashboard logs for review:
--Goto ITSM Homepage->Expand Application->AR System Server Group Console
-->Go to Logs->View Logs
->>Now select FTS0 server -->Check for logs having latest runid after keyword DashboardDataGenerator_DATASOURCES and DashboardDataGenerator_DATASOURCES_KPI.
>>You can get current utility run job id from form CMDB:Dashboard_Utility_Run_History