Discovery: CMDB sync of almost all hosts/devices fails with the error "Violates relationship cardinality (120039)"

Product

BMC Discovery

Component

BMC Discovery

Applies to

BMC Discovery

Problem

CMDB sync of almost all hosts/devices fails with the following error:

<CI Name>: Failed creation of 1 non-shared relationship: Failed creation of BMC_InIPSubnet BMC_IPEndpoint:<IP> - BMC_IPConnectivitySubnet:<subnet> : Violates relationship cardinality (120039)

Destroying and re-scanning the hosts/devices does not resolve the error.

When running a full resync to the same dataset or to an empty/new dataset, the same "Violates relationship cardinality (120039)" error is reported for many CIs:

Request (400): ERROR (120039): The operation violates the cardinality constraint of the relationship. : 1-Many cardinality will be violatedCannot create relationship instance because right endpoint is already in another relationship.Relationship Instance Id: <ID>

During the resync process, the following timeout error is also reported:

cmdb_sync.rest_exporter: ERROR: Failed POST http://<CMDB Server>:8008/api/cmdb/v1.0/instances/BMC.ADDM5: HTTP Error Service Unavailable (503): ERROR (93): Timeout during data retrieval due to busy server -- retry the operation. Please contact administrator if the problem persists

Cause

AR/CMDB performance issue.

Solution

During the full resync, when creating the relationship CIs (in batches of 100), the Discovery connection fails on the client API timeout because the AR server is busy, and a timeout error is reported. However, a few of these relationships do get created in the CMDB. When this happens, the CMDB transformer log reports:

cmdb_sync.resync: INFO: <CMDBServer> - New: Sync creation of 91386 relationships
cmdb_sync.rest_exporter: INFO: Sync CREATE 100 relationships to http://<CMDBServer>:8008 (0a9d683989698d8866160a610e013941 : <CMDBServer> - New).

cmdb_sync.rest_exporter: ERROR: Failed POST http://<CMDBServer>:8008/api/cmdb/v1.0/instances/BMC.ADDM5: HTTP Error Service Unavailable (503): ERROR (93): Timeout during data retrieval due to busy server -- retry the operation. Please contact administrator if the problem persists

cmdb_sync.target: INFO: <CMDBServer> - New: Check existence of 100 relationships after failed create (check attempt 1)...

Since a timeout is reported for the batch of CIs, Discovery believes the relationship creation has failed, and it tries again to resync these CIs. As a few of these CIs are already created during the previous attempt, the error "Violates relationship cardinality (120039)" is reported during the second attempt:

Request (400): ERROR (120039): The operation violates the cardinality constraint of the relationship. : 1-Many cardinality will be violatedCannot create relationship instance because right endpoint is already in another relationship.Relationship Instance Id: OIGAA5V0FF7DEAQSHWALQSHWALLF5R

The resync process eventually completes with errors.

cmdb_sync.resync: INFO: <CMDBServer> - New: End task RESYNC_TASK_COMMIT: state = RESYNC_STATE_COMPLETED message = 'Processed 72390 CIs, 110805 relationships (duration 0:26:43)' errors = [CMDBSyncCORBA.ErrorMessage(time='<>', short_message='muccmdbi02 - New: Failed Creation of <> relationships',

As these errors are present when the resync is completed, any subsequent sync of hosts/devices also ends with the same error.

SOLUTION:
Refer to the KA 000355123: Discovery: Extremely slow performance on CMDB sync/resync when creating relationships using REST API

To mitigate the issue, decrease the batch size to 50 in the Discovery CMDB connection configuration, and run a full resync. After this, the resync should complete (probably with fewer errors).

To troubleshoot more specific cases, see also:

- KA 000325031: Discovery: A single host fails with CMDB sync error 120039: The operation violates the cardinality constraint of the relationship
- KA 000383825: Discovery: CMDB sync of almost all hosts/devices fails with the error "Violates relationship cardinality (120039)"

Knowledge Article