BMC Helix IT Operations Management (ITOM) Deployment: ITOM Environment is down | Postgres pods are crashing, error in the postgres pods is "Error communicating with DCS"


To get the postgres pod logs:

kubectl get pod -n <NS> | grep -i postgres

kubectl logs postgres-bmc-pg-ha-0 -n <namespace>
kubectl logs postgres-bmc-pg-ha-1 -n <namespace>
kubectl logs postgres-bmc-pg-ha-2 -n <namespace>

Example:
kubectl logs postgres-bmc-pg-ha-0 -n itom

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa7babd7460>, 'Connection to 10.XX.0.XX timed out. (connect timeout=2.5)')': /api/v1/namespaces/<NS>/configmaps?labelSelector=application%3Dpatroni%2Ccluster-name%3Dpostgres-bmc-pg-ha
ERROR: Request to server https://10.XX.0.XX:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.XX.0.XX', port=443): Max retries exceeded with url: /api/v1/namespaces/<NS>/pods?labelSelector=application%3Dpatroni%2Ccluster-name%3Dpostgres-bmc-pg-ha (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa7babd7a00>, 'Connection to 10.XX.0.XX timed out. (connect timeout=2.5)'))")
2024-05-14 08:58:22,707 ERROR: get_cluster
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/patroni/dcs/kubernetes.py", line 966, in __load_cluster
    self._wait_caches(stop_time)
  File "/usr/local/lib/python3.9/site-packages/patroni/dcs/kubernetes.py", line 858, in _wait_caches
    raise RetryFailedError('Exceeded retry deadline')
patroni.utils.RetryFailedError: Exceeded retry deadline
ERROR: Error communicating with DCS
INFO: DCS is not accessible
WARNING: Loop time exceeded, rescheduling immediately.
ERROR: Request to server https://10.XX.0.XX:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.XX.0.XX', port=443): Max retries exceeded with url: api/v1/namespaces/<NS>/configmaps?labelSelector=application%3Dpatroni%2Ccluster-name%3Dpostgres-bmc-pg-ha (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa7babd7910>, 'Connection to 10.XX.0.XX timed out. (connect timeout=2.5)'))")
ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa7babd7190>, 'Connection to 10.XX.0.XX timed out. (connect timeout=2.5)')': /api/v1/namespaces/<NS>/pods?labelSelector=application%3Dpatroni%2Ccluster-name%3Dpostgres-bmc-pg-ha
2024-05-14 08:58:28,159 ERROR: Request to server https://10.XX.0.XX:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.XX.0.XX', port=443): Max retries exceeded with url: /api/v1/namespaces/<NS>/pods?labelSelector=application%3Dpatroni%2Ccluster-name%3Dpostgres-bmc-pg-ha (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa7b8acd7c0>, 'Connection to 10.XX.0.XX timed out. (connect timeout=2.5)'))")
ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
INFO: Local timeline=77 lsn=3C/B6011D90
.
.
INFO: following new leader after trying and failing to obtain lock
WARNING: Loop time exceeded, rescheduling immediately.
INFO: Lock owner: postgres-bmc-pg-ha-1; I am postgres-bmc-pg-ha-0

See also: BEST FAQ on resolving PostgreSQL Pod/Deployment issues in an OnPrem BMC Helix IT Operations Management (ITOM) deployment