After upgrading Control-M/Server to version 9.0.20 or installing a new instance on the same Unix/Linux machine, the new Kafka and Zookeeper services are in a Warning state in the Control-M Configuration Manager (CCM) On a host with both Control-M/Enterprise Manager and Control-M/Server running, you might see two processes running: one for Kafka and one for Zookeeper but that are related to Control-M/Enterprise Manager instead of being for Control-M/Server. In the ctm_em/log/services-configuration-agent.log file you can see messages like the following ones: ======================= 2020-10-02 06:45:19,032 INFO [taskScheduler-5] (RunningInstancesDataProvider:142) - Retrieving 0 pids with detect word deployment.service=apache-zookeeper 2020-10-02 06:45:19,033 INFO [taskScheduler-5] (RunningInstancesDataProvider:72) - Service apache-zookeeper cannot run retries 16 cooldown: false has pid: false 2020-10-02 06:45:19,033 INFO [taskScheduler-5] (ServiceActionStartProcess:155) - Heartbeats: 0 Pids: Desired State: Up Service: apache-zookeeper 2020-10-02 06:45:19,087 INFO [taskScheduler-5] (RunningInstancesDataProvider:142) - Retrieving 0 pids with detect word deployment.service=apache-kafka 2020-10-02 06:45:19,087 INFO [taskScheduler-5] (RunningInstancesDataProvider:72) - Service apache-kafka cannot run retries 16 cooldown: false has pid: false 2020-10-02 06:45:19,087 INFO [taskScheduler-5] (ServiceActionStartProcess:155) - Heartbeats: 0 Pids: Desired State: Up Service: apache-kafka 2020-10-02 06:45:19,231 INFO [pool-3-thread-55] (HeartbeatOnBehalfDataProviderBase:107) - Health Report: [{"instance": "zk-91b288f5-1", "service": "apache-zookeeper", "displayName": "Apache Zookeeper", "statusMessage": "[imok]", "status": "Up"}] Command: [[SERVICES_HOME]/bin/apache_zookeeper_health.py] 2020-10-02 06:45:19,232 INFO [pool-3-thread-55] (MessagingProviderBase:292) - MessagingProviderBase -> publish -> Messaging Provider Is Not In The Connected State. Message Publication Will Not Occur 2020-10-02 06:45:19,260 INFO [pool-3-thread-54] (HeartbeatOnBehalfDataProviderBase:107) - Health Report: [{"instance": "kafka-91b288f5-1", "displayName": "Apache Kafka", "service": "apache-kafka", "statusMessage": "OK", "status": "Up"}] Command: [python "[SERVICES_HOME]/bin/apache_kafka_health.py" --kafkaHost localhost] 2020-10-02 06:45:19,260 INFO [pool-3-thread-54] (MessagingProviderBase:292) - MessagingProviderBase -> publish -> Messaging Provider Is Not In The Connected State. Message Publication Will Not Occur 2020-10-02 06:45:28,906 INFO [initializationTimer] (MessagingProviderBase:160) - MessagingProviderBase -> initialize -> Retry Invoked 2020-10-02 06:45:28,906 INFO [initializationTimer] (MessagingProviderBase:125) - MessagingProviderBase -> initialize -> Invoked 2020-10-02 06:45:28,907 WARN [initializationTimer] (MessagingProviderBase:153) - MessagingProviderBase -> initialize -> Encountered An Exception -> Continuing [shouldThrowExceptionOnInitializationError = false] 2020-10-02 06:45:28,907 INFO [initializationTimer] (MessagingProviderBase:154) - MessagingProviderBase -> initialize -> Initialization Will Be Retried 2020-10-02 06:45:28,907 INFO [Thread-71096] (MessagingProviderBase:181) - at subscribing communication 2020-10-02 06:45:29,017 ERROR [Thread-71096] (NetworkClient:663) - [Consumer clientId=consumer-35547, groupId=Obmc_SGctm-srv_Rdefault_Ddefault_R0_M91b288f5_Sservices-configuration-agent_V1_Isca-91b288f5-1] Connection to node -1 failed authentication due to: Authentication failed: Invalid username or password 2020-10-02 06:45:29,018 ERROR [Thread-71096] (MessagingProviderBase:648) - Error subscribing to topics org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: Invalid username or password ===================== |
CTM-5074 has been created for this issue and is implemented in version 9.0.21 Apply the latest available version to fix the issue. Alternate option before version 9.0.21: The following instructions must not be applied on the Windows Operating System : Change the ports used by Control-M/Server to ones not used by Control-M/Enterprise Manager with the following steps: The default Kafka port is 19092. In Control-M/Server, please use any unused port, such as 29092. The default Zookeeper port is 12181. In Control-M/Server, please use any unused port, such as 22181. Stop the Control-M/Server processes with the following commands:
ps -fu < unix user>If there are some processes left, please kill them. Log in as your Control-M/Server and run the following commands
The following video demonstrates this solution: Additional Information: - Micro Services were first introduced to Control-M/Server in version 9.0.20, but for Control-M/Enterprise Manager since version 9.0.18. To use Micro Services, Control-M/Server uses its instances of Zookeeper and Kafka. If the upgrade (or installation) of Control-M/Server version 9.0.20 is performed in a machine where there is already the Control-M/Enterprise Manager at version 9.0.19 or higher, and this is kept down while installing/upgrading, then the ports selected to be used by the Control-M/Server's Kafka and Zookeeper are the default ones and will create a conflict with the ones used by Control-M/Enterprise Manager. This can be verified by checking the file <CTM Server Home>/ctm_server/services/config/services.yml: this file contains the ports assigned to Kafka and Zookeeper: zookeeper. connect: 'localhost\:12181/emkafka' zookeeper.properties: 'em.zookeeper.properties' kafka.bootstrap.servers: 'hostname\:19092' Check to see if the zookeeper. connect and/or the kafka.bootstrap.servers ports from that file match the ports found in the same file on the Control-M/Enterprise Manager machine (<EM Server Home>/ctm_em/services/config/services.yml). If both files from Control-M/Server and Control-M/Enterprise manager contain any of the same port values then neither instance of Kafka will work properly. - Customers viewing this solution may find value in the following self-help Connect with Control-M video. |