Kafka and Zookeeper services are not starting on Control-M/Server or Control-M/Enterprise Manager giving error "Fatal error during KafkaServer startup. Prepare to shutdown"

Knowledge Article

Article Number

000376637

Old Article Number

Article Type

Solutions to a Product Problem

Title

Kafka and Zookeeper services are not starting on Control-M/Server or Control-M/Enterprise Manager giving error "Fatal error during KafkaServer startup. Prepare to shutdown"

Summary

Product

Control-M/Server for UNIX and Microsoft Windows

Component

Control-M/Server for UNIX and Microsoft Windows

Applies to

Control-M/Server 9.0.20 and higher
Control-M/Enterprise Manager 9.0.18 and higher
Unix / Linux ; Windows

Problem

Kafka and Zookeeper services are not initializing causing all other microservices to remain inactive.
In the Kafka logs found in the
<home>/kafka/kafka_data/log directory the following messages are printed:

[2020-11-26 14:29:18,285] INFO Opening socket connection to server localhost/127.0.0.1:12184. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2020-11-26 14:29:18,389] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-11-26 14:29:18,389] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-11-26 14:29:18,395] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING

This means that Kafka cannot connect to Zookeeper and closes after a Timeout.
In the Zookeeper logs, the following messages are printed:

[2020-11-26 14:05:36,780] INFO Server successfully logged in. (org.apache.zookeeper.Login)
[2020-11-26 14:05:36,807] INFO binding to port localhost/127.0.0.1:12183 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-11-26 14:05:37,016] ERROR Last transaction was partial. (org.apache.zookeeper.server.persistence.Util)
[2020-11-26 14:05:37,018] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.EOFException

Cause

Corrupted zookeeper transaction file in the directory <home>/kafka/kafka_data/kafkadata/zookeeper1/version-2

Solution

Please note that the following procedures shouldn't be used in Control-M Workflow Insights environment which will need additional steps.

Note: this only for versions 9.0.21.x and lower. If you are on version 9.0.22, reach out to BMC Support.

Use the following steps to reinitialize Kafka and Zookeeper:

Control-M/Server

1. Stop the Control-M Server (shut_ctm && shut_ca) and ensure that all processes are down, except for the Control-M Server database which should remain running.

2. Check if there is any Control-M Server remaining process running, if there are processes that are still running, kill them
For Unix/Linux, please use:
ps -ef | grep deploy | grep ctm_server

For Windows, please check from the Task manager

3. Clean up kafkadata folder
For Unix/Linux:
Please navigate to directory <Control-M Server Home>/kafka/kafka_data/kafkadata/ and delete all subdirectories (you must keep broker0 and zookeeper1 directories, but can delete the files inside them)

For Windows:
Please navigate to directory <Control-M Home>\Control-M Common\kafka\kafka_data\kafkadata and delete all subdirectories (you must keep broker0 and zookeeper1 directories)

4. Run the apache_kafka_config.py configuration file as follows:
For Unix/Linux:
run_python <Control-M Server Home>/ctm_server/services/bin/apache_kafka_config.py install --kafkaPort <kfport> --zookeeperPort <zkport>
run_python <Control-M Server Home>/ctm_server/services/bin/apache_kafka_config.py sync

For Windows:
Please open the CMD.exe, and then run
run_python <Control-M Server Home>\ctm_server\services\bin\apache_kafka_config.py install --kafkaPort <kfport> --zookeeperPort <zkport>
run_python <Control-M Server Home>\ctm_server\services\bin\apache_kafka_config.py sync

Note: The default kafka port is 19092 and the default zookeeper port is 12181

5. Start Control-M server (start_ctm && start_ca)

Control-M/Enterprise Manager

1. Stop Control-M Enterprise Manager and verify that all processes are down, except for the Control-M Enterprise Manager database which should remain running.

2. Navigate to the "version-2" directory located under:
Unix/Linux:
From version 9.0.20.100 and earlier: <emhome>/ctm_em/kafka/kafka_data/kafkadata/zookeeper
From version 9.0.20.200 and higher: <emhome>/kafka/kafka_data/kafkadata/zookeeper

Windows:
<emhome>/Control-M Common/kafka/kafka_data/kafkadata/zookeeper

3. Search for any 0-byte files and delete them.
Example log.f6 is 0 bytes and needs to be deleted:
-rw-r----- 1 em920 controlm 67108880 Dec 27 13:06 log.b07
-rw-r----- 1 em920 controlm 67108880 Jan 6 08:52 log.b3f
-rw-r----- 1 em920 controlm 67108880 Nov 15 2021 log.e0
-rw-r----- 1 em920 controlm 0 Nov 15 2021 log.f6

4. Execute the following commands:
Unix/Linux: <emhome>/ctm_em/services/bin
em bmcpython apache_kafka_config.py install --kafkaPort <kfport> --zookeeperPort <zkport>
em bmcpython apache_kafka_config.py sync

Windows:
bmcpython apache_kafka_config.py install --kafkaPort <kfport> --zookeeperPort <zkport>
bmcpython apache_kafka_config.py sync

Windows Examples (with default ports):
bmcpython "C:\Program Files\BMC Software\Control-M EM\Default\services\bin\apache_kafka_config.py" install --kafkaPort 19092 --zookeeperPort 12181
bmcpython "C:\Program Files\BMC Software\Control-M EM\Default\services\bin\apache_kafka_config.py" sync

5. Start Control-M/Enterprise Manager and verify services start properly

NOTE: The default Kafka port is 19092. The default Zookeeper port is 12181.

Attachment(s):