BMC Helix IT Operations Management (ITOM) Deployment: Deployment manager is failing to upgrade elastic search logging from version 22.2.01 to 22.4 with "Error: UPGRADE FAILED: cannot patch "elasticsearch-logging-kibana" with kind PersistentVolumeClaim"

Following errors observed in the "helix-on-prem-deployment-manager/logs/deployment.log" file

*****************************************************************************
*                   Installing EFK-BMC-helix-logging for installer               *
*****************************************************************************
INFO: Successfully decrypted password.
INFO: Creating Secrets and ConfigMap for BMC helix logging services
INFO: Installing Elasticsearch for elasticsearch-logging                   
INFO: Updating override file.....
INFO: Successfully decrypted password.
INFO: Installing helm chart: elasticsearch-logging
/usr/bin/helm upgrade elasticsearch-logging /helix-on-prem-deployment-manager/commons/../bmc-helix-logging/efk/elasticsearch/elasticsearch-19.0.2.tgz -f /helix-on-prem-deployment-manager/bmc-helix-logging/efk/elasticsearch/chart_values-updated.yaml --set ingressApiVersion=true -n bmc-helix-logging --install --wait --timeout 20m0s >>/helix-on-prem-deployment-manager/commons/../logs/deployment.log 2>&1
I0316 02:42:51.257053   30219 request.go:621] Throttling request took 1.164487577s, request: GET:https://api.<>.com:6443/apis/batch/v1beta1?timeout=32s
Error: UPGRADE FAILED: cannot patch "elasticsearch-logging-kibana" with kind PersistentVolumeClaim: PersistentVolumeClaim "elasticsearch-logging-kibana" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value && cannot patch "elasticsearch-logging-data" with kind StatefulSet: StatefulSet.apps "elasticsearch-logging-data" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden && cannot patch "elasticsearch-logging-master" with kind StatefulSet: StatefulSet.apps "elasticsearch-logging-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
[230316-024302] ERROR: Failed to install helm chart: elasticsearch-logging.
[230316-024302] ERROR: Failed to install EFK-elasticsearch...............................
[230316-024302] ERROR: Failed to deploy Bmc Helix Logging 

The issue is identified as a defect #DRRE3-2460. The upgrade is not possible because of the major upgrade in Elasticsearch from v7 to v8 for security vulnerability fixes.

The workaround is to clean up the old deployment for logging from the bmc-helix-logging namespace and deploy the new version of BMC Helix Logging.

Please follow the below steps:

1. Take backup for secret "ims-serviceaccount":

kubectl get secret ims-serviceaccount -n <helix-platfrom-namespace> -o yaml > ims-serviceaccount.yaml

Note: Please review the backup file, and validate it is having value for "IMS_SERVICE_ACCOUNT_KEY." Example below: 

[root@example-controller-node1 ~]# kubectl edit secret ims-serviceaccount -n <bhom-namepsace>
apiVersion: v1
data:
  IMS_SERVICE_ACCOUNT_KEY: MzdkNDZjOTgtZDhiNC00ZGVkLWJhNjktNzcxNmNiNmZhYzUx
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"IMS_SERVICE_ACCOUNT_KEY":"MzdkNDZjOTgtZDhiNC00ZGVkLWJhNjktNzcxNmNiNmZhYzUx"},"kind":"Secret","metadata":{"annotations":{},"creationTimestamp":null,"name":"ims-serviceaccount","namespace":"<namespce>"}}

2. Using helm uninstall the deployment for bmc-helix-logging

helm delete elasticsearch-logging -n <logging-namespace>
helm delete fluentd -n <logging-namespace>

3. Delete the PVCs for bmc-helix-logging

kubectl delete pvc -n <logging-namespace> data-elasticsearch-logging-data-0 data-elasticsearch-logging-data-1 data-elasticsearch-logging-master-0 data-elasticsearch-logging-master-1 elasticsearch-logging-kibana

4. Then from the 22.4 deployment manager directory, update the helix-on-prem-deployment-manager/configs/deployment.config file as below:
keep only "BMC_HELIX_LOGGING=yes" and "INSTALL_MODE=yes" rest all the application mark as "no" (if other applications are already deployed successfully) and re-run the helix-on-prem-deployment-manager/deployment-manager.sh script.

Example:

cat helix-on-prem-deployment-manager/configs/deployment.config

#Install mode as fresh or upgrade
INSTALL_MODE=install

#Flag controlling infra services installation
INFRA=no

#Flag controlling individual infra services installation
_PTPOSTGRESS=no
_KAFKA=no
_REDIS=no
_RSSO=no
_VICTORIAMETRICS=no
_ELASTICSEARCH=no
_MINIO=no

# Do not make changes to service flags it will break dependency
#Flag controlling helix dashboard services installation
HELIX_DASHBOARD_SERVICES=no

#Flag controlling itsminsight services installation
ITSMINSIGHT_SERVICES=no

#Flag controlling aiops services installation
AIOPS_SERVICES=no

#Flag controlling monitor product installation
MONITOR=no

#Flag controlling intelligentintegrations services installation
INTELLI_INT_SERVICES=no

#Flag controlling intelligent automation product installation
INTELLIGENT_AUTOMATION=no

#Flag controlling bmc-helix-logging product installation
BMC_HELIX_LOGGING=yes

#Flag Controlling optimize installation
OPTIMIZE=no

5. Once logging is deployed  user will notice a blank value for IMS_SERVICE_ACCOUNT_KEY in secret ims-serviceaccount 

Example:

[root@example-controller-node1 ~]# kubectl edit secret ims-serviceaccount -n <bhom-namepsace>
apiVersion: v1
data:
  IMS_SERVICE_ACCOUNT_KEY: ""
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"IMS_SERVICE_ACCOUNT_KEY":""},"kind":"Secret","metadata":{"annotations":{},"creationTimestamp":null,"name":"ims-serviceaccount","namespace":"<namespce>"}}

6. Then edit secret ims-serviceaccount and copy the key IMS_SERVICE_ACCOUNT_KEY from backup ims-serviceaccoun.yaml file from steps-1, and update the IMS_SERVICE_ACCOUNT_KEY value and save the file.

7. Once the secret is edited successfully then restart all ims pods to take effect of the change

kubectl rollout restart deploy/ims -n <bmc-helix-platform-namepsace>

if the issue still persists then please open a support case and share the helix-on-prem-deployment-manager/logs/deployment.log file