Use the collect-diagnostic-logs.sh script (Attached to this Knowledge Article) to automatically gather diagnostic information from the cluster and package it into a support bundle.

The script:

• automatically detects whether kubectl or oc CLI is available
• collects cluster and namespace diagnostic data
• gathers logs from recently failed pods
• collects PostgreSQL health information if present
• packages all collected information into a ZIP file that can be shared with support teams.

The script must be executed from a system that has kubectl or oc CLI installed and configured to access the Kubernetes/OpenShift cluster
 

Script Name

collect-diagnostic-logs.sh

Steps to Run the Script

1. Copy the script to the system where kubectl or oc is configured.
2. Make the script executable:

chmod +x collect-diagnostic-logs.sh

3. Execute the script:

./collect-diagnostic-logs.sh

User Inputs Required

During execution, the script prompts for the following inputs:

ITSM Namespace (Optional)

Enter the ITSM namespace, or type No in case there is no ITSM OnPrem deployed:

• Provide the ITSM / Helix Platform Common Services namespace if ITSM is deployed.
• If the environment contains only ITOM, enter: No

ITOM Namespace (Mandatory)

Enter ITOM namespace (Required):

• Provide the namespace where ITOM components are deployed.
• This field is mandatory and the script will not proceed without it.

Example:

Enter ITSM namespace, or type No in case there is no ITSM OnPrem deployed: No
Enter ITOM namespace (Required): itomwmk

What the Script Collects

The script collects diagnostic information from several areas of the cluster.

1. Cluster Information

The script gathers cluster-level information to verify the overall health of the cluster.

Collected data includes:

• Kubernetes version
• Cluster API information
• Node inventory and roles
• Node conditions
• Node resource utilization

Commands executed:

kubectl version

kubectl cluster-info
kubectl get nodes -o wide
kubectl describe nodes
kubectl top nodes

This helps identify:

• Node readiness issues
• CPU or memory exhaustion
• Disk pressure

2. Namespace Resource Information

The script collects workload and configuration details from the ITSM and ITOM namespaces.

Collected resources include:

• Pods
• Deployments
• StatefulSets
• DaemonSets
• Services
• Persistent Volume Claims (PVCs)

Commands executed:

kubectl get pods

kubectl get deployments
kubectl get statefulsets
kubectl get daemonsets
kubectl get svc
kubectl get pvc

This helps verify:

• Application deployment status
• Replica configuration
• Service exposure and storage

3. Pod Resource Utilization

The script collects real-time CPU and memory usage for pods.

Commands executed:

kubectl top pods

oc adm top pods

This helps identify:

• High CPU usage pods
• High memory usage pods
• Performance bottlenecks

4. High Resource Utilization Detection

The script identifies pods where:

• CPU ≥ 90% of limit
• Memory ≥ 90% of limit

Logic used:

• Reads usage from kubectl top pods
• Reads limits from kubectl describe pod
• Calculates utilization percentage

For such pods, the script collects:

kubectl describe pod

This helps identify:

• Pods nearing resource limits
• Memory leaks or CPU spikes

5. Detection of Pods in Problematic States

The script detects pods that are not running correctly by inspecting container state reasons (not just STATUS column).

Detected states include:

ImagePullBackOff

ErrImagePull
CrashLoopBackOff
CreateContainerError
Init:CrashLoopBackOff
Pending

Commands used:

kubectl get pod -o jsonpath

kubectl describe pod
kubectl get pod -o yaml

This ensures accurate detection even when STATUS column is misleading.

6. Diagnostics Collected for Problematic Pods

For each problematic pod, the script collects:

• Pod description
• Pod YAML definition
• Logs from all containers
• Previous logs (-p)

Commands executed:

kubectl describe pod <pod>

kubectl get pod <pod> -o yaml
kubectl logs <pod> -c <container>
kubectl logs -p <pod> -c <container>

This helps diagnose:

• Image pull failures
• CrashLoopBackOff issues
• Init container failures
• Configuration errors
• Scheduling issues (Pending pods)

7. Restarted Container Diagnostics (Last 2 Days)

The script detects containers that restarted within the last 2 days.

Collected data:

• Pod describe output
• Container logs
• Previous logs

Commands executed:

kubectl describe pod

kubectl logs
kubectl logs -p

This helps identify:

• Recent crashes
• OOMKilled containers
• Application failures

8. PostgreSQL Health Checks

If PostgreSQL is present, the script collects diagnostics from:

• postgres-bmc-pg-ha-15
• postgres-bmc-pg-ha-17

Collected data includes:

• Disk usage
• Patroni cluster status

Commands executed:

kubectl exec <pod> -- df -h

kubectl exec <pod> -- patronictl list

This helps identify:

• Disk space issues
• Replication problems
• Cluster health

9. PostgreSQL Leader Detection

The script identifies the primary node using:

role=primary

This ensures correct execution of database checks.

10. PostgreSQL Replication Slot Verification

From the leader pod:

select * from pg_replication_slots;

This helps identify:

• Replication lag
• Inactive replicas
• WAL backlog

11. Support Bundle Creation

After execution, the script generates a compressed bundle:

k8s_support_bundle_<timestamp>.zip

Example:

k8s_support_bundle_20260408_213804.zip

The bundle contains:

• Cluster diagnostics
• Namespace data
• Pod logs and failures
• PostgreSQL diagnostics

Summary

The collect-diagnostic-logs.sh script automates the collection of diagnostic data from Kubernetes/OpenShift environments used in BMC Helix ITSM and ITOM deployments.

It helps troubleshoot issues related to:

• Cluster health
• Pod resource utilization
• Container crashes and restarts
• Image pull and startup failures
• PostgreSQL replication and storage

The generated ZIP file can be shared with support teams for faster analysis and resolution.

Manual Steps (If Script Cannot Be Used)

If the collect-diagnostic-logs.sh script cannot be executed in the environment, the required logs can be collected manually using the following commands.

To fetch the logs for a particular pod (Kubernetes and OpenShift):
kubectl logs <pod-name> -n <namespace-of-ITOM>
oc logs <pod-name> -n <namespace-of-ITOM>

If there are two containers associated with a pod then use the following to fetch logs for a particular container:
kubectl logs <pod-name> -c <container-name> -n <namespace-of-ITOM>

Please note that the user can redirect the output of the pod log to a text file for sharing with the support team if further troubleshooting is required. Here's an example:

For a pod:
kubectl logs <pod-name> -n <namespace-of-ITOM> > pod1.txt
oc logs <pod-name> -n <namespace-of-ITOM> > pod2.txt

For a particular container:
kubectl logs <pod-name> -c <container-name> -n <namespace-of-ITOM> > pod1_1.txt
oc logs <pod-name> -c <container-name> -n <namespace-of-ITOM> > pod1_1.txt

Watch the video which covers how to collect logs, how to check and collect Pod or Service Logs in BMC Helix IT Operations Management (ITOM) OnPrem Deployment