- How to verify Control-M/Agent and Control-M/Server are properly communicating and configured correctly?
- What to check if the Control-M/Agent is unavailable in the Control-M Configuration Manager(CCM)?

Note: Control-M/Servers and Control-M/Agents must be running in pair. 
The setup of having more than one Control-M/Server communicate with a single Control-M/Agent is not supported. 
However, more than one Control-M/Agent can be installed on a machine and communicate with a single Control-M Server.

Below are the steps to verify Control-M/Server and Control-M/Agent communication configuration.

Step 1
Determine the 'Server to Agent Port Number' (SAPN) from Control-M/Server configuration:
- This is the port that Control-M/Server communicates to the agent. In other words, the port which the Control-M/Agent will be "listening" on, expecting an incoming message from the Control-M/Server. 
- The default value from installation is 7006 but can be changed to another free port number. 
- To find the SAPN port number, follow the procedures below on the Control-M/Server:
a. Executes 'ctm_menu'
b. Select the Parameter Customization option
c. Select Parameters for Communicating with Specific Agent Platforms
d. To view/modify communication parameters, enter the node ID of Agent Platform:   <agent>
The value in "Server to Agent port Number" is the SAPN.
 
Step 2
Determine the 'Agent to Server Port Number'(ASPN) from the Control-M/Server configuration:
- This is the port where Control-M/Server listens for incoming agent connections and which Control-M/Agent connects to Control-M Server,. 
- The default value from installation is 7005 but can be changed to another free port number. 
- To find the ASPN port number, follow the procedures below on the Control-M/Server:
a. Executes 'ctm_menu'
b. Select the Parameter Customization option
c. Select Communication and Operational Parameters
d. The value in 'Agent to Server Port Number' is the ASPN.

Step 3
Check if Control-M/Agent is running and listening on the SAPN listening port:
a. To check the listening port, run the following command at the Control-M Agent host:
 
 Unix:
 netstat -an | grep <SAPN>
 Windows:
 netstat -an | findstr "<SAPN>"

- The output should be similar to the below if the port is listening:

Protocol Recv-Q Send-Q Local Address Foreign Address status
tcp4        0            0            *.7006            *.*                        LISTEN

- Verify that the port number following the string '*.' is the SAPN (ie, 7006 in the above line). 
- If there is no such line in the netstat command output, then the Control-M/Agent services not listening on this port.  If this is the case, please proceed directly to "Verifying Configuration"(Step 4) in this document.

- From the netstat command output above, there is a process listening on this SAPN port. 
- Perform telnet to the Control-M/Agent hostname and SAPN port number, from within the same host using the below command to verify the port can be connected:

 telnet localhost <SAPN>

- Below shows the connected message.

Connected to localhost.xxx.com.
Escape character is '^]'.

- To exit from telnet,  enter '<CTRL> + ]' and follow by 'quit' to exit from the telnet interface.

b. To check if the Control-M/Agent process is running, execute the following.
 
 Unix:
 ps -ef | grep p_ctmag

 Windows:
 tasklist | findstr "p_ctmag"

- The following will be displayed if the Control-M/Agent is running.

<user>    <pid>     1  0 01:20 ?     00:00:02 /home/ctmag9/ctm/exe/p_ctmag

- If there is no entry returned, it means Control-M/Agent is not running and there is another application using the port number. 
- Change the Server to Agent port number (SAPN) by following the steps in the "Changing Agent Port Configuration"(Step 5) section of this document.

- If there is a connection refused message from the telnet command, or the telnet command is stuck and non-responsive, then there is likely a configuration issue with the configuration. 
Please proceed to the "Verifying Control-M/Agent Configuration" section (Step 4) of this document.

Step 4
Verifying Control-M/Agent Configuration:
- After logging in as the Control-M/Agent user on the Control-M/Agent host, execute the 'ctmagcfg' command to check the port settings as below.

1)      Agent-to-Server Port Number . . . : [7005]
2)      Server-to-Agent Port Number . . . : [7006]

- Make sure that the port numbers are correct and that the Control-M/Agent SAPN port is not used by any other application on the machine.
- This can be verified again by stopping the Control-M/Agent services (for Unix, '$Agent Home/ctm/scripts/shut-ag' or from Windows Services.msc)  and checking with "netstat -an" command to see if the Server to Agent Port Number (SAPN) is still active.

On Windows:
netstat -an | findstr "7006"
On Unix:
netstat -an | grep 7006

- If the port is used by another application, then change the SAPN port by following the steps described in the "Changing Control-M/Agent Port Configuration" (Step 5) of this document.
 
Step 5
Changing Control-M/Agent Port Configuration:
- After login in as the Control-M/Agent user or local administrator on the Control-M/Agent host, perform these steps.

a. To find a port that is free, use the "netstat -an" and look for a port, typically 10 ports higher.  This example will look at port 7106. 
By running "netstat -an | grep 7016" and not seeing any lines returned, this port is not in use by any application. This port can be set as a new Server to Agent Port Number(SAPN).
(NOTE: Check with the system administrator if unsure which port to use before proceeding) 

b, To change the port,
i. Stop the Control-M/Agent services
ii. Executes ctmagcfg
iii. Select Server-to-Agent Port Number option and enter '7016' to change the value. 
NOTE: DO NOT CHANGE the Agent to Server port number(ASPN). 
iv. Accept the defaults for the rest of the parameters or change only the port number and Control-M Server Authorized Servers entries.

c. Once the change is done, select the 's' option (Save) and Enter to save the change.

d. Start Control-M/Agent services (for Unix, '$Agent Home/ctm/scripts/start-ag' or from Windows Services.msc) 

e. Verify telnet command can connect to the new port 7016.

telnet localhost <SAPN>

f. Once connected use '<CTRL> + ]'   followed by "quit" to exit the telnet interface.

- If the connection still fails, check with the system administrator or contact BMC support for additional help.  There is probably an issue with the IP configuration on that machine or the product version.

g. If the SAPN is changed navigate to Control-M/Server host to update the Server-to-Agent configuration as well with the below steps:
i. Execute 'ctm_menu'
ii. Select the Parameter Customization option
iii. Select Parameters for Communicating with Specific Agent Platforms option
iv. To view/modify communication parameters enter the host ID of Agent Platform:   <agent>
v. Select the option for "Server to Agent Port Number" and modify it to reflect the new SAPN.
 
 

Step 6
Performing Control-M/Agent communication diagnostics:
- Control-M/Agent communication diagnostics can be performed by running 'ag_diag_comm' command on the Control-M/Agent host machine. 
- This command has to be executed under the Control-M/Agent user account for Unix and Administrator for Windows. 
- Once the command completes, verify all of the entries are correct and that the UNIX/System ping to Control-M Server has succeeded. 
- If the UNIX/System ping did not succeed, please approach the system administrator to verify that the hostname of the Control-M Server can be resolved on the Control-M/Agent host. 
- If the 'ping' command succeeds or the system administrator has verified the name resolution, then perform the 'ag_ping' command at the OS prompt.
i. If 'ag_ping' completes successfully, then proceed to verify Control-M/Agent connectivity from the Control-M/Server.
ii. If 'ag_ping' fails, then continue troubleshooting the Control-M/Server.

Step 7
Performing Control-M Server communication diagnostic:
- The following is to be executed on the Control-M/Server host with a Control-M/Server user account on Unix or Administrator on Windows.
a. Executes 'ctm_diag_comm {Control-M/Agent host id}'

i. If the 'CTMS ping to Agent or remote host' succeeds and the status is 'available', then the Control-M/Agent is ready for job submission.
ii. If  the ''CTMS ping to Agent or remote host'' fails or the status is 'unavailable'
- Please check with a system administrator and make sure that the Control-M/Server host can resolve the hostname of the Control-M/Agent. 
- Typically, this is a Name Resolution issue, and depending on the environment, local /etc/hosts file or DNS server host table will need to be updated.
- If there is a firewall in the environment, please alert the infrastructure team of the ports involved.
- The firewall communication requirement is as follows:
- Any port from the Control-M/Server needs to be able to initiate a connection to the agent on the SAPN. 
- Any port on the Control-M/Agent needs to be able to create a communication to the Control-M/Server on the ASPN. 
- If NAT is in use or has any other questions, please contact BMC support.

NOTE: 'UNIX ping to Agent or Remote host' may appear as 'Failed' on some hosts because the OS Ping executable is disabled for the account due to security hardening. If this is the case, this is fine and will not affect communication. Otherwise, please investigate this with the system administrator.
 
Additional Information
----------------------------

For information about Persistent Connections, see the following resolution:
Connecting an Agent and Server with persistent connections when using a firewall: 000026617.

The following error messages in Control-M/Server's NS logs may be observed:

0609 11:27:22.095 NS: Agent <Agent_Name> set to unavailable by service SRVC_PE_TUNNEL (11026c050): receive error
0609 20:10:23.406 NS: Agent <Agent_Name> set to unavailable by service SRVC_PE_SUBMIT (1102d8f70): receive error
0609 20:34:25.841 NS: Agent  <Agent_Name> set to unavailable by service SRVC_PE_TRACE_ALL (11029e650): receive error

Suggestions to resolve are:
~Check the Protocol Version of the Agent and Control-M Server. Make sure this is correct for your Control-M environment.
~Perform telnet utility command between Control-M server and Control-M Agent environments
~Perform nslookup utility command between Control-M server and Control-M Agent environments