What are the recommended steps to implement SSL/TLS in zone 3 between Control-M/Server and Control-M/Agents when all Agents are currently in TCP mode?

Important Considerations

1. The solution below is only used when no Control-M/Agents are in SSL/TLS mode.
   For environments where SSL/TLS is currently in use see article 000442490.
2. Because the Control-M/Server keystore is the same for communication with Agents (zone 3) and the EM (zone 2), if using zone 2 SSL/TLS, the EM certificate will also need to be updated.
   • To determine whether SSL/TLS is in use in zone 2:
     1. In the Control-M Configuration Manager, right-click on the the relevant Control-M/Server.
     2. Select Properties.
     3. Open the Communication tab.
     4. Under Communication -> Protocol:
        TCP or SSL will be shown.
   • See the following documentation to change the zone 2 keystore on the EM:
     https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm
3. The solution below recommends Control-M Automation API (AAPI) for centralized updates.
   • PAKAI.9.0.21.306 or PAFZ4.9.0.22.004 are required to deploy certificates through AAPI.
     See the following technical bulletins for more information:
     • https://docs.bmc.com/xwiki/bin/view/Control-M-Orchestration/Control-M/ctm9021/Patches/Control-M-Agent-PAKAI-9-0-21-306/
     • https://docs.bmc.com/xwiki/bin/view/Control-M-Orchestration/Control-M/ctm9022/Patches/Control-M-Agent-PAKAI-9-0-22-004/
   • AAPI version 9.0.21.200 or higher is required on the EM for enabling SSL/TLS on Agents.
     • To verify this, on the EM host, check the Automation API version in the installed-versions.txt file.
     • Note: Automation API Monthly Releases are supported on EM versions 9.0.20 and higher.
       https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Installation.htm#ControlMRESTAPI
   • See the following documentation for the prerequisite step to configure the AAPI ctm cli utility.
     • https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Installation.htm#ControlMAutomationAPICLI

FAQ:

• What do I need to do when I need to add a new Agent after the Control-M/Server and existing Agents have had SSL/TLS applied?
  1. The new Agent will be in TCP mode.
     To put the new Agent in SSL mode, apply the steps relevant to the Agent from the solution below.

Solution:

1. Make sure to use your organization's own Certificate Authority (CA), provided by your organization's security team, and not a third-party CA.
   This CA should be dedicated to Control-M and not allow unauthorized issuance.
   Each certificate issued for a Control-M product should contain a unique email address (e.g. hostname_component@domain).

2. Verify no Control-M/Agents use SSL/TLS communication with Control-M/Server:

   1. Open the Control-M Configuration Manager (CCM).
   2. If the SSL column is not currently showing, right click on any column header and select "Column chooser."
   3. From the Column chooser dialog, drag the "SSL" column into the columns view.
   4. Select each Control-M/Server in the tree on the left to view the Control-M/Agents in the middle pane.
   5. Sort by the SSL column, and any Agents set to "Enabled" will need to be handled according to article 000442490
   6. Export this list to a file for future reference: choose File -> Exports List
3. For each Control-M Server, perform the below steps to create a new certificate and keystore.
   A video demonstrating these steps can be found at https://www.youtube.com/watch?v=3Jz-to-0e50
   1. Back up the existing SSL/TLS keystore, identified by the following commands:

      1. UNIX/Linux:
         directory: grep -i securitydir $HOME/ctm_server/data/SSL/cert/site.plc
         filename: grep -i keyfile $HOME/ctm_server/data/SSL/cert/site.plc

      2. Windows:
         directory: reg query "HKLM\SOFTWARE\BMC Software\Control-M/Server" /v securitydir /s
         filename: reg query "HKLM\SOFTWARE\BMC Software\Control-M/Server" /v keyfile /s

   2. Create a Certificate Signing Request (CSR) for the Control-M/Server.
      Follow the "Generating a Signed Certificate" section of the below documentation.
      It is recommended to use an organization-specific email address in the emailAddress field of the CSR.
      https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#GeneratingaSignedCertificate
      
      The CSR, which needs to be sent to your security team in the next step, is generated at:
      Linux/Unix: $HOME/ctm_server/data/SSL/certificate_requests/<hostname>_YYYYMMDD_HHMMSS.csr
      Windows: <Control-M/Server Install Path>\data\SSL\certificate_requests\<hostname>_YYYYMMDD_HHMMSS.csr
      
      The private key, which is used in step d to create a .p12 keystore, is generated at:
      Linux/Unix: $HOME/ctm_server/data/SSL/private_keys/<hostname>_YYYYMMDD_HHMMSS.pem
      Windows: <Control-M/Server Install Path>\data\SSL\private_keys\<hostname>_YYYYMMDD_HHMMSS.pem
   3. Have your organization's security team use the internal CA and the CSR to issue a certificate.
   4. Create a new keystore, with only the new certificate and its CA chain.
      Follow the "Generating the .p12 Keystore" section of the below documentation.
      Make sure to use a secure password for the keystore.
      https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#Generatingthep12Keystore

   5. Ensure the Security Level is set to 4 (mutual authentication) in the Control-M/Server Security Policy:
      https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Security_policies.htm

      • UNIX/Linux: Back up and edit the following files:
        $HOME/data/SSL/cert/site.plc
        $HOME/data/SSL/cert/ns.plc
        In the [client] and [server] sections, ensure security_level=4
      • Windows: Backup and modify the following registry keys:
        HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\client
        HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\server
        HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\client
        HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\server
        Ensure security_level=4
   6. Use ctmkeytool to configure each Control-M/Server to use the new keystore:
      https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#ConfiguringSSLinZone2and3
   7. When convenient, recycle each Control-M Server.
      1. Use the CCM to recycle.
         1. Right click on the Control-M/Server.
         2. Choose "Desired State"
         3. Click "Recycle"
         4. If prompted, click "Yes" to confirm.

4. Deploy certificates on each Agent.

   1. For each Agent currently in TCP mode with PAKAI.9.0.21.306 or PAKAI.9.0.22.004 installed, use the Automation API from a centralized location.

      1. Create a Certificate Signing Request (CSR) with the following command:

         ctm config server:agent:csr::create <server> <agent> -f <configuration.json> > <new_csr.pem>
         
         It is recommended to configure an Agent-specific and organization-specific email address in the emailAddress field of the CSR configuration.json.
         
         See the following documentation for more details on this command such as the syntax of the configuration file.

         https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_AutomatingSSLCertificates.htm#configserveragentcsrcreate
         
         Note: the configuration file must have a .json extension.
         
         There will be no screen output from the command, unless there is an error, in which case the error will be displayed and the CSR file will be empty.
         The created CSR file will used in the next step.

      2. Have your organization's security team use the internal CA and the CSR to issue a certificate.

      3. Deploy the signed certificate and CA chain to the Agent with the following command:

         ctm config server:agent:crt::deploy <server> <agent> <signed_cert.pem> <ca_chain.pem>
         
         Where "<signed_cert.pem>" is the certificate issued in the previous step in X.509 Base-64 encoded format, and "<ca_chain.pem>" Is the CA certificate chain which signed the certificate, also in X.509 Base-64 encoded format. See the following documentation for more details on this command.

         https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_AutomatingSSLCertificates.htm#configserveragentcrtdeploy

   2. If the Automation API cannot be used (e.g. when the patch cannot be installed on TCP Agents), certificates must be installed manually on each Agent.
      1. Back up any existing SSL/TLS keystore, identified by the below commands:

         1. UNIX/Linux:
            grep -i keyfile $CONTROLM/data/SSL/cert/site.plc

         2. Windows:
            reg query "HKLM\SOFTWARE\BMC Software\Control-M/Agent" /v keyfile /s

      2. Create a Certificate Signing Request (CSR) for the Control-M/Agent.
         Follow the "Generating a Signed Certificate" section of the below documentation.
         It is recommended to use an organization-specific email address in the emailAddress field of the CSR.
         https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#GeneratingaSignedCertificate
         
         

         The CSR, which needs to be sent to your security team in the next step, is generated at:
         Linux/Unix: $CONTROLM/data/SSL/certificate_requests/<hostname>_YYYYMMDD_HHMMSS.csr
         Windows: <Control-M/Agent Install Path>\data\SSL\certificate_requests\<hostname>_YYYYMMDD_HHMMSS.csr
         
         The private key, which is used in step iv to create a .p12 keystore, is generated at:
         Linux/Unix: $CONTROLM/data/SSL/private_keys/<hostname>_YYYYMMDD_HHMMSS.pem
         Windows: <Control-M/Agent Install Path>\data\SSL\private_keys\<hostname>_YYYYMMDD_HHMMSS.pem

      3. Have your organization's security team use the internal CA and the CSR to issue a certificate.
      4. Create a new keystore, with only the new certificate and its CA chain.
         Follow the "Generating the .p12 Keystore" section of the below documentation.
         Make sure to use a secure password for the keystore.
         https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#Generatingthep12Keystore
      5. Use ctmkeytool to apply the new keystore.
         Follow the "Configuring SSL in Zone 2 and 3" section of the below documentation:
         https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#ConfiguringSSLinZone2and3
5. The default security level for Agents is 4 (mutual authentication).
   For each Agent, if this has been changed from 4, it needs to be set back to 4.
   • Fully supported Control-M/Agent versions 9.0.21 and higher: security level is hard-coded to 4 and nothing further needs to be done on this step.
   • Version 9.0.20: Check and modify the site configuration to set the security level.
     • https://documents.bmc.com/supportu/9.0.20/help/Main_help/en-US/index.htm#4277.htm
     • https://documents.bmc.com/supportu/9.0.20/help/Main_help/en-US/index.htm#4243.htm
     • UNIX/Linux: Back up and edit the following files:
       $CONTROLM/data/SSL/cert/site.plc
       $CONTROLM/data/SSL/cert/ag.plc
       In the [server] section, ensure security_level=4
     • Windows: Backup and modify the following registry keys:
       HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Agent\SecurityPolicy\site\server
       HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Agent\SecurityPolicy\ag\server
       Ensure security_level=4
6. When convenient, use one of the following options to recycle each Agent.
   • Use API to recycle:
     ctm config item::recycle "CTMS:Agent:<ServerName>:<AgentName>"
     https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_Agent.htm#configitemrecycle
   • When unable to use the API, use the CCM to recycle.
     1. Right click on the Control-M/Agent.
     2. Choose "Desired State"
     3. Click "Recycle"
     4. If prompted, click "Yes" to confirm.
7. Once all affected Control-M Servers and Agents have been recycled to use the new keystore, use one of the following options to enable SSL/TLS on each Agent.
   • If Automation API version 9.0.21.200 or higher is installed on the Enterprise Manager, run the below Automation API command for each Agent to enable SSL/TLS:
     ctm config server:agent::update <ServerName> <AgentName> sslState Enabled
     https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_Agent.htm#configserveragentupdate
   • If the Automation API version is lower than 9.0.21.200 and cannot be updated at this time, use the following steps in the Control-M Configuration Manager (CCM) to enable SSL for each agent:
     1. Right click on Control-M/Agent
     2. Choose "Properties"
     3. Select the "Communication" tab
     4. Change "Secure Socket Layer" to "Enabled"
     5. Click "OK"
8. For Linux/Unix Control-M/Agents, verify the permissions on the security related files are set correctly.
   • For fully supported Control-M/Agent versions (9.0.21 and higher), run the following command as the Control-M/Agent user:
     $CONTROLM/toolbox/permission_check.sh --force
   • For other Control-M/Agent versions, see the alternate solution in article 000441965