The BMC Helix CMDB Synchronization is slow.

Root Cause 1:  The Inline Normalization (enabled by default) is slow

The "Impact Normalization Feature" is not needed because Discovery CMDB Sync creates its own impact relationships (assuming it is using the default data model).

To confirm: Disable the Inline Normalization with the steps below and check if the sync is faster

In the CMDB UI, go to the AtriumCore Console->Normalization->Edit Configuration, disable 'Inline Normalization' the dataset of Discovery (BMC.ADDM by default).

Root Cause 2:  The specified RPC queue is too busy

To confirm: If there are connection errors and timeouts in cmdb sync logs, the root cause is possible.

Solution:  Change the RPC queue in the GUI to use 390698 or 390699, but care must be taken so that CMDB Sync is not "competing" with Normalization and/or Reconciliation resources.

Root Cause 3:  The number "Concurrent Workers" is not optimal

See this article: Discovery: What are the "Concurrent Workers" in the CMDB sync configuration page?

To confirm:

- If the number is 1 or more than 3, the root cause is possible.
- If there are connection errors and timeouts in cmdb sync logs, the root cause is possible (the number of worker is too high).

Solution: See this documentation page.

Extract: "start from the default [1] and gradually increase the number up to when sync rate keeps up with discovery rate"

Do not use *all* of the available RPC queue threads on the AR Server: The AR Server has other work to do on those RPC queues besides supporting CMDB Sync from Discovery.
A good guideline is to use at most half of the threads on an RPC queue.

Root Cause 4:  The "Batch Size" is not optimal

To confirm:

If there are connection errors and timeouts in cmdb sync logs, the root cause is possible (the number of worker is too high).

Solution: See this documentation page.

Extract: " You can set the batch size to anything between 1 and 1000. The default is 100."

Root Cause 5:  Performance issue on AR/CMDB side. The biggest bottleneck could be around the database,  AR Escalations, AR Filters, an overlay on a BMC.CORE form, customization, index issues

Root Cause 6: Defect DRUD1-50026: the implementation of the RFE DRDC1-23311 in the July TKU has created major performance issues for some big customers with fast pacing Kubernetes environments. 

Solution:

• • Deactivate Continuous Sync in the CMDB Sync connection(s)
  • Update to a version above the November 2024 TKU.
  • Deactivate the pattern CMDB.Kubernetes_Tags
  • Remove any workaround that could have been put in place, such as having set "No BMC_ComputerSystem CIs are synced to the CMDB." in the tab "Filtering" of the sub tab "CMDB" for "BMC_Tag". Possibly reactivate the pattern CMDB.Tags if interested in synchronizing tags other than kubernetes.
  • Reactivate Continuous Sync

Workaround 1:

- Disable the CMDB.Tags syncmapping pattern
- Destroy the CMDB Connection in the CMDB
- Recreate a connection in Discovery and wait for the end of the resync

Workaround 2: 

- Import the syncmapping CMDB.Tags from June 2024
- Destroy the CMDB Connection in the CMDB
- Recreate a connection
- Next TKU update: Possibly import the pattern again if the new TKU doesn't yet embed improvements related to synching Kubernetes tags. Note that at some point there is a risk that the June pattern version may not be activable again. Make sure to test the TKU update and June pattern version in a test environment first.

Workaround 3:

- in the CMDB Sync connection go to the tab "Filtering" then click on the sub tab "CMDB"
- select "BMC_Tag" in the dropdown list and tick the option "No BMC_ComputerSystem CIs are synced to the CMDB."

Note: this will not affect an ongoing synchronization, it will only be used for the next syncs. Also it will not have any positive effect on the time it takes to build or update the shadow copy, as these filters are used at the time of the synchronization itself, and not the preparation.

Optimizations for the workarounds:
Either delete or Mark As Deleted the unwanted tags in the CMDB before resynching Discovery to it:

- Mark A Deleted: this should be faster than destroying

- Deleting: Note that deleting them from the CMDB will most likely imply a lot of errors about the fact that instance ids cannot be found in the CMDB Discovery logs, and that this is expected.

Root cause 7: CMDB defect fixed in versions above CMDB 9.0

Root cause 8: Discovery-side performance issue (rare issue)

To confirm: Use this article: Discovery: Troubleshooting Discovery performance issues

Root cause 9: Kubernetes cluster hosts trigger around 13K CI's and relationships delete during CMDB Sync. This is because, these hosts reached the removal threshold. This is the expected behavior.

To confirm: On CMDB Sync page, check if Kubernetes specific hosts are taking more time to sync due to huge CI and relationship delete count.

Work-around:  In case this is impacting other CI's in syncing with CMDB, ensure that the Kubernetes cluster scans are not running in parallel with other scans. Schedule Kubernetes scans in batches so the hosts do not get queued up.