After updating Control-M/Server to version 9.0.20 or setting up a new instance on the same Unix/Linux machine, the newly added Kafka and Zookeeper services might show a warning in the Control-M Configuration Manager (CCM).

If both Control-M/Enterprise Manager and Control-M/Server are operating on the same host, you could see two processes: one for Kafka and one for Zookeeper. It's important to note that these processes may belong to Control-M/Enterprise Manager instead of Control-M/Server.

You can find messages related to this issue in the $EM_HOME/log/services-configuration-agent.log file:
=======================
2020-10-02 06:45:19,032 INFO [taskScheduler-5] (RunningInstancesDataProvider:142) - Retrieving 0 pids with detect word deployment.service=apache-zookeeper 
2020-10-02 06:45:19,033 INFO [taskScheduler-5] (RunningInstancesDataProvider:72) - Service apache-zookeeper cannot run retries 16 cooldown: false has pid: false
2020-10-02 06:45:19,033 INFO [taskScheduler-5] (ServiceActionStartProcess:155) - Heartbeats: 0  Pids:   Desired State: Up  Service: apache-zookeeper

2020-10-02 06:45:19,087 INFO [taskScheduler-5] (RunningInstancesDataProvider:142) - Retrieving 0 pids with detect word deployment.service=apache-kafka 
2020-10-02 06:45:19,087 INFO [taskScheduler-5] (RunningInstancesDataProvider:72) - Service apache-kafka cannot run retries 16 cooldown: false has pid: false
2020-10-02 06:45:19,087 INFO [taskScheduler-5] (ServiceActionStartProcess:155) - Heartbeats: 0  Pids:   Desired State: Up  Service: apache-kafka

2020-10-02 06:45:19,231 INFO [pool-3-thread-55] (HeartbeatOnBehalfDataProviderBase:107) - Health Report: [{"instance": "zk-91b288f5-1", "service": "apache-zookeeper", "displayName": "Apache Zookeeper", "statusMessage": "[imok]", "status": "Up"}] Command: [[SERVICES_HOME]/bin/apache_zookeeper_health.py]
2020-10-02 06:45:19,232 INFO [pool-3-thread-55] (MessagingProviderBase:292) - MessagingProviderBase -> publish -> Messaging Provider Is Not In The Connected State. Message Publication Will Not Occur
2020-10-02 06:45:19,260 INFO [pool-3-thread-54] (HeartbeatOnBehalfDataProviderBase:107) - Health Report: [{"instance": "kafka-91b288f5-1", "displayName": "Apache Kafka", "service": "apache-kafka", "statusMessage": "OK", "status": "Up"}] Command: [python "[SERVICES_HOME]/bin/apache_kafka_health.py" --kafkaHost localhost]
2020-10-02 06:45:19,260 INFO [pool-3-thread-54] (MessagingProviderBase:292) - MessagingProviderBase -> publish -> Messaging Provider Is Not In The Connected State. Message Publication Will Not Occur
2020-10-02 06:45:28,906 INFO [initializationTimer] (MessagingProviderBase:160) - MessagingProviderBase -> initialize -> Retry Invoked
2020-10-02 06:45:28,906 INFO [initializationTimer] (MessagingProviderBase:125) - MessagingProviderBase -> initialize -> Invoked
2020-10-02 06:45:28,907 WARN [initializationTimer] (MessagingProviderBase:153) - MessagingProviderBase -> initialize -> Encountered An Exception -> Continuing [shouldThrowExceptionOnInitializationError = false]
2020-10-02 06:45:28,907 INFO [initializationTimer] (MessagingProviderBase:154) - MessagingProviderBase -> initialize -> Initialization Will Be Retried
2020-10-02 06:45:28,907 INFO [Thread-71096] (MessagingProviderBase:181) - at subscribing communication
2020-10-02 06:45:29,017 ERROR [Thread-71096] (NetworkClient:663) - [Consumer clientId=consumer-35547, groupId=Obmc_SGctm-srv_Rdefault_Ddefault_R0_M91b288f5_Sservices-configuration-agent_V1_Isca-91b288f5-1] Connection to node -1 failed authentication due to: Authentication failed: Invalid username or password
2020-10-02 06:45:29,018 ERROR [Thread-71096] (MessagingProviderBase:648) - Error subscribing to topics
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: Invalid username or password
=====================
 

CTM-5074 has been created for this issue and is implemented in version 9.0.21
Apply the latest available version to fix the issue.


Alternate option before version 9.0.21:

The following instructions must not be applied on the Windows Operating System : 
Change the ports used by Control-M/Server to ones not used by Control-M/Enterprise Manager with the following steps:

The default Kafka port is 19092. In Control-M/Server, please use any unused port, such as 29092.
The default Zookeeper port is 12181. In Control-M/Server, please use any unused port, such as 22181.

Stop the Control-M/Server processes with the following commands:

• shut_ca

• shut_ctm

Make sure there are no processes left still running for the Control-M/Server :

ps -fu < unix user>

If there are some processes left, please kill them.


Log in as your Control-M/Server and run the following commands

• run_python <CTM_SERVER_HOME>/ctm_server/services/bin/apache_kafka_config.py install --kafkaPort <KAFKA_PORT> --zookeeperPort <ZK_PORT>

• run_python <CTM_SERVER_HOME>/ctm_server/services/bin/apache_kafka_config.py sync

• For example:
  run_python /home/ctm920/ctm_server/services/bin/apache_kafka_config.py install --kafkaPort 29092 --zookeeperPort 22181
  run_python /home/ctm920/ctm_server/services/bin/apache_kafka_config.py sync

Start Control-M/Server with the following commands:

• start_ca

• start_ctm

If the Control-M/Enterprise Manager Services continue to show as not started in the CCM, similarly stop and restart those processes with the following commands:

• stop_config_agent

• start_config_agent

 

The following video demonstrates this solution:

Additional Information:
- Micro Services were first introduced to Control-M/Server in version 9.0.20, but for Control-M/Enterprise Manager since version 9.0.18.
To use Micro Services, Control-M/Server uses its instances of Zookeeper and Kafka.
If the upgrade (or installation) of Control-M/Server version 9.0.20 is performed in a machine where there is already the Control-M/Enterprise Manager at version 9.0.19 or higher, and this is kept down while installing/upgrading, then the ports selected to be used by the Control-M/Server's Kafka and Zookeeper are the default ones and will create a conflict with the ones used by Control-M/Enterprise Manager.

This can be verified by checking the file <CTM Server Home>/ctm_server/services/config/services.yml: this file contains the ports assigned to Kafka and Zookeeper:
  zookeeper. connect: 'localhost\:12181/emkafka'
  zookeeper.properties: 'em.zookeeper.properties'
  kafka.bootstrap.servers: 'hostname\:19092'

Check to see if the zookeeper. connect and/or the kafka.bootstrap.servers ports from that file match the ports found in the same file on the Control-M/Enterprise Manager machine (<EM Server Home>/ctm_em/services/config/services.yml). If both files from Control-M/Server and Control-M/Enterprise manager contain any of the same port values then neither instance of Kafka will work properly.

- Customers viewing this solution may find value in the following self-help Connect with Control-M video.