This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Pre-upgrade checklist apart from the ones contained in BMC official document of 20.02 upgrade PreRequisite.

1. --Run CDMChecker utility per instructions provided in the following documentation URL.
   https://docs.bmc.com/docs/ac91/using-the-cdmchecker-tool-679149883.html
   --Typically, the below commands (executed separately), can be handy in returning missing forms / fields
   [1] cdmchecker -m -u
   [2] cdmchecker -a -u
   --Grab a definition copy of the missing forms from another working environment using BMC Developer Studio, and then import it into the affected AR server. Below documentation link has instructions to import and export form definitions in BMC Developer Studio.
   https://docs.bmc.com/docs/ars2002/exporting-and-importing-definitions-909637926.html

2. Check SIM Plugin:

--Upgrade may fail due to SIM workflows getting triggered during CI operation which were looking for SIM's NOTIFY plugin and plugin is not up and running/missing on customer env.
Below link talks in detail of How to check if Notify plugin is correctly configured in Platform server.
https://confluence.bmc.com/pages/viewpage.action?pageId=237986434#PublishingServer/IMAPI/IIWSTroubleshootingandHelp-Notifyplugininformation
https://docs.bmc.com/docs/display/public/proactivenet95/Configuring%20the%20Notify%20ARDBC%20plug-in
--Make sure SIM's NOTIFY plugin is up and running (you can check this by checking if class manager has BMC.SIM classes added from CMDB UI), if not then disable all SIM workflow.
      Verify that Notify ARDBC plugin is up and running:
 
                                I.            Open the form 'NOTIFY:protocols' and retrieve entries. You should get 1 entry with version.
                              II.            Open the form 'NOTIFY:servers' and retrieve entries. You should get 1 entry. The port should be accessible for Publishing Server to open a TCP/IP connection.
 
                            III.            Enable tracing for Notify ARDBC plugin:
 
The plugin uses standard ARS logging facilities enable logging for SQL/Filter/API log and Plug-in Server Set the logging level to "ALL" for full logging.
 
Because in some cases logging might be insufficient or complex when a lot of plugins are loaded, one can set the environment variable NOTIFY_DEBUG before the plugin server is started (as a system environment variable on Windows for example).
 
NOTIFY_DEBUG True
 
> In that case a log file with highest detail (regardless of the Log Level Setting) will be written in the following location "C:\ardbcnotify.log" (On windows)
–OR- "/tmp/ardbcnotify.log" (On UNIX).
 
                            IV.            Enable Dispatcher logging by adding a line to the ar.cfg file such as:
 
Dispatch-Log-File:C:\temp\dispatch.log
 
This will capture the activity of the dispatcher server process, arsvcdsp.exe.
 
Restarting the Action Request System Server (to restart the arplugin, to reload the notify ARDBC Plugin) is required.

--You can disable all filters associated with the plugin if automated publish feature is not is use.
SIM:Notify:ClassInfo
SIM:Notify:AttribInfo
SIM:Notify:ReconInfo
SIM:Notify:EventFailure
--If you are using publishing feature, please follow below link to check plugin details.
https://docs.bmc.com/docs/display/public/proactivenet95/Configuring%20the%20Notify%20ARDBC%20plug-in
https://confluence.bmc.com/pages/viewpage.action?pageId=237986434#PublishingServer/IMAPI/IIWSTroubleshootingandHelp-VerifyingNotifyPluginServerAvailability

Note -This link will give you all the configuration details.

3.CMDB 20.02 upgrade works on importing a def file on form CMDB:JobRunInfo.--Make sure CMDB:JobRunInfo form does not have too many records as this delays the upgrade process and gets the installer stuck in one place without giving any error.
--You can check count on this form by directly querying onto ARSystem Database by running below query:
Select count(*) from CMDB_JobRunInfo
--If count is more than 50k and contains data older than 1 year as well then you can retain 6 months old data and delete older than this.
Note – The CMDB:JobRunInfo form keeps logs info on NE and RE job runs and if you have many errors in the RE or NE, your table grows in big numbers (millions of records per run) and this can affect def import of CMDB:JobRunInfo form by installer
 

These commands are present as below command lines in armonitor.cfg:
"C:\Program Files\BMC Software\AtriumCore\cmdb\server64\bin\arcmdbd.exe" -i "C:\Program Files\BMC Software\ARSystem" -m
"C:\Program Files\BMC Software\AtriumCore\cmdb\server64\bin\arrecond.exe" -i "C:\Program Files\BMC Software\ARSystem" -m
Verifying exe from Task Manager:
--In Linux, you can view this process by running the following command:
grep -i "arrecond".

5.Verify SHARE:Application_Properties 

• Check ARServer upgrade- correct version details and correct references in SHARE:Application_Properties
•  Verify no failed/running references in SHARE:Application_Properties 

Search with below queries to confirm on same
·       Property Value=%failed%
·       Property Value=%running%
·       Property Value=%action%

6. Verify if Web service is installed (you can verify Atrium Core installation Directory to see WSC directory)
 

• If no reference of install directory for web-service is present or if no reference of shared Tomcat then verify SHARE:Application_Properties form.
• If these components (Atrium Web Services and Web Services Registry) are present in SHARE:Application_properties form then delete it after taking back up of those Records.
• Take a backup of existing AtriumCoreInstalledconfiguration.xml file (Path - $BMCSOFTWAREINSTALLATION$\AtriumCore

• If web services are not being used, then need to remove all the references including “WSC” feature from AtriumCoreInstalledconfiguration.xml. Refer document ‘https://community.bmc.com/s/article/Atrium-CMDB-How-to-uninstall-Atrium-Core-Web-Services-Web-Services-Registry’ for same.

• Verify “failed” feature if any only delete the feature name
   

       FROM
        <property>
      <name>FAILED_INSTALL_FEATURES</name>
      <value>featureCMDB;featureAtrium_Webservices;featureAIS</value>
    </property>
        TO
          <property>
      <name>FAILED_INSTALL_FEATURES</name>
      <value> </value>
    </property> 

 -- Tomcat used for webservices is being run on Local User Account and not any service account.
 

This can be verified by navigating to Task manager->Performance->Open Resource Monitor->Network->Expand Listening Ports tab. Here WS port should be listed under tomcat exe and not under any other applications like arplugin.exe, armonitor.exe etc

Steps to uninstall webservices:
Use uninstaller.exe to uninstall Atrium Webservices and Web services Registry component.

• Choose Start > Settings > Control Panel on the Windows desktop.
• In the Control Panel dialog box, double-click Add/Remove Programs.
• In the Add/Remove Programs dialog box, select AtriumCore.
• Click Remove.
• When the uninstaller launches, select which features to uninstall.

Else follow below document for uninstalling above 2 components manually:
 

• https://community.bmc.com/s/article/Atrium-CMDB-How-to-uninstall-Atrium-Core-Web-Services-Web-Services-Registry

8. Check if CMDB Engine is Running/Started.

- Confirm by checking the arerror.log if CMDB is started post the latest restart of AR Service.
- If it is not running, then check for any felix deployer related errors in arerror.log or ardebug.log to find the root cause. The Felix Deployer is responsible for deploying the jar files in the ARSystem\deploy folder.

9. Full User Permissions to the JAVA and AtriumCore directories:

- Provide Full Control/All permissions to the user running the installer to the JAVA and AtriumCore directories.

10. Incase of failure on Patch upgrade for 20.02 patch 4 and 5, please check below:

--Operating-mode in Primary server should be 1

--Operating Mode in any secondary server should be 2. This config confirms that DB level changes have been done and now only file changes are needed on server level
Doc for reference - https://community.bmc.com/s/news/aA33n000000CfdjCAC/operating-mode