A) Useful High-Level questions to answer to help BMC Customer Support understand the problematic behavior

1) Is the performance problem confined to specific console actions (e.g. saving certain Component Templates, browsing Server Groups etc) or is it slowness across the board? If confined to specific actions, what are these actions and how severe is the performance issue? How does the performance of these operations differ from previous experience?

2) Is the performance problem consistent or does it appear to be intermittent? If intermittent, does there seem to be any pattern to the performance problem? e.g. always slower at certain times of the day or when certain jobs or Database Maintenance Tasks are running

3) Is the performance the same for members of the BLAdmins role as it is for members of other roles? Or is it specific to certain roles?

4) When did the performance problem(s) begin to occur in the BSA environment and were there any noticeable environment changes made at that time? e.g. BSA/TSSA Upgrade, Database migration, Network change, changes to the virtualization infrastructure on which the BSA/TSSA env resides etc.

5) Is the problem confined to specific BSA/TSSA consoles or do BSA/TSSA consoles installed on different servers/workstation all have the same issue?

6) Is the BSA/TSSA Console installed in the same subnet as the BSA/TSSA Application Server?

7) Is the BSA/TSSA Console running on user desktops or on a centralized Console Server e.g. Citrix

8) Is there anti-virus software running on the BSA/TSSA Console? Can that be temporarily disabled and the operation retested to determine whether this is affecting performance?

Often the user/system admin would disagree disabling the security software (such as McAfee) on the server. In such case, it would be helpful to run RAMMAP sysinternal on the Appserver/RCP console and capture the output file for review.

Alternatively one can capture process monitory log  to during the time stamp of issue, which help in finding interference of any anti-virus with the application process. 

If you observe large number of zombie processes present on the appserver/RCP, where slowness was observed, then it is very likely that the security software may be causing the issue.

9) If the BSA/TSSA environment uses an Oracle Database, is the BMC-recommended gather_schema_stats  job running weekly and has it been succeeding? This can be confirmed via a DBA, the Health and Value Dashboard or running a DB Diagnostic as described in the "Verifying that statistics are current" section of this  page.

10) Has the Oracle Out of Box gather statistics job been disabled as recommended in the Setting up an Oracle database for TSSA ? Running both the BMC-recommended gather_schema_stats job and the Out Of Box Oracle version, even at different times, has been known to cause performance problems which is why the BSA documentation recommends disabling the Out Of Box Oracle version.

11) If the BSA/TSSA environment uses a SQL Server Database, is the SQL Server sp_updatestats procedure scheduled to run weekly?  And does an index rebuild occur on a regular schedule?

12) Is the BSA/TSSA Database Cleanup Job  running regularly and successfully to completion in this environment?

13) Run the Database Diagnostic under Monitoring and diagnosing issues in the TrueSight Server Automation environment . This tests DB latency by running many queries and returning their response time to the expected response time.

14) Are the BSA/TSSA Appservers and BSA/TSSA DB in the same network location? If Virtual Machines, are the Application Servers and DB running on hosts in the same data center? We have seen issues where the DB Server is running on a host on a different, albeit close by, data center. Once the DB Server host was migrated to the same data center as the BSA/TSSA Application Servers, the performance issue was resolved. The VM team should be consulted to verify this.


B) What items should be gathered and provided to BMC Customer Support?

In addition to answering the above questions, the following data should be provided to BMC Customer Support to help with problem diagnosis.

1) Job Run Logs of recent BSA DB Cleanup Jobs. There may be multiple jobs configured to run on different days e.g. Daily Cleanup and Weekly Cleanup. Please gather the job run logs of each Cleanup Job. This will help BMC Support confirm what cleanup commands are being run and whether they are running to completion.

2) Database Reports:

• Oracle: Request that the site DBA provide Oracle AWR, ADDM and ASH reports for a time period during which the performance problems was reproduced. It is important that the reports are captured for a timeframe during which the problems were observed. Rather than a single report covering a 12 hour period, it is better to generate multi hourly reports which can then be correlated to Job Run logs, known times when Console performance was slow etc.
• SqlServer: Sql Server usually requires setting up a SQL Server Profiler session and capturing data, so that may require reproducing the problem again when the profiler session is running.

3) Health and Value Dashboard results zip file: The BSA Health and Value dashboard stores its results in the <File Server Storage>\dashboards\reports\latest directory. See below example. Please zip up the most-recent directory and provide the resultant zip file:

4) Debug Logging - setup the following Debug Logging on both the BSA Console Workstation and the BSA Configuration Appserver
 

a)  Console Logs - perform the following steps on the BSA Console workstation:

• Backup the existing <BSA_Console_Dir>/NSH/br/rcp.cf file, and then replace with the attached version.
• Close the BSA Console
• Rename the Workspace directory: %APPDATA%\Bladelogic\1_2_1\
• Restart the BSA console
• After the issue has been reproduced (step c below), capture and provide the following logs from the BSA Console Workstation:
• BLWorkbenchPlugin.log: From  %APPDATA%\Bladelogic\1_2_1\Workspace1\.metadata\.plugins\com.bladelogic.client.ui\
•  .log file from %APPDATA%\Bladelogic\1_2_1\1_2_1\Workspace1\.metadata
• Restore the original rcp.cf with the backup after the testing.

1. Have the DBA setup to create manual AWR snapshot on BladeLogic database.
2. Run the actions that are slow in that GUI, note what actions are slow.
3.  Create another manual AWR snapshot.
4. Send AWR report

1. Have the DBA setup a trace session on the BladeLogic database.