What are the recommended steps to change the SSL/TLS Certificate Authority in zone 3 between Control-M/Server and Control-M/Agent?

BMC recommends applying any changes in a test environment prior to performing them in a production environment.

Important Considerations:

1. Failure to perform the below steps in the exact order provided will result in communication failure.
2. There are other methods to perform this procedure, but they will result in more Agent downtime.
3. If currently using zone 2 SSL/TLS between the Control-M/Server and Control-M/Enterprise Manager, the Control-M/Enterprise Manager certificate will also need to be updated.
   This is because the Control-M/Server keystore is the same for zone 2 and zone 3.
   • To determine whether SSL/TLS is in use in zone 2:
     1. In the Control-M Configuration Manager, right-click on the the relevant Control-M/Server.
     2. Select Properties.
     3. Open the Communication tab.
     4. Under Communication -> Protocol: TCP or SSL will be shown.
   • See the following documentation page for steps to change the zone 2 keystore on Control-M/Enterprise Manager:
     https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm

FAQ:

• What steps need to be performed when adding a new Control-M/Agent after the Control-M/Server and existing Control-M/Agents are already using SSL/TLS?
  1. The new Control-M/Agent will be in TCP mode, needing the steps related to the Agent in article 000442271.

Solution:

1. Make sure to use your organization's own Certificate Authority (CA), provided by your organization's security team, and not a third-party CA.
   This CA should be dedicated to Control-M and not allow unauthorized issuance.
   Each certificate issued for a Control-M product should contain a unique value in the emailAddress field (e.g. hostname_component@domain).
2. Determine which Control-M/Agents use SSL/TLS communication with Control-M/Server:
   1. Open the Control-M Configuration Manager (CCM).
   2. If the SSL column is not currently showing, right click on any column header and select "Column chooser."
   3. From the Column chooser dialog, drag the "SSL" column into the columns view.
   4. Select each Control-M/Server in the tree on the left to view the Control-M/Agents in the middle pane.
   5. Sort by the SSL column.
   6. As needed, export this list to a file: choose File -> Exports List
3. Check the Security Level in the Control-M/Server Security Policy.
   https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Security_policies.htm
   Look for security_level in the following locations:
   • UNIX/Linux: [client] and [server] sections of the following files:
     $HOME/data/SSL/cert/site.plc
     $HOME/data/SSL/cert/ns.plc
   • Windows: The following registry keys:
     HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\client
     HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\server
     HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\client
     HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\server
4. There are two options for the next steps in this solution:
   • Option 1: This is the recommended method when any of the Control-M/Server Security Levels are not 4, to avoid needing to schedule a Control-M-environment-wide downtime.
     1. Temporarily reconfigure Control-M/Agents to TCP mode to deploy the new keystore.
        • If Automation API version 9.0.21.200 or higher is installed on the Enterprise Manager, run the below Automation API command for each agent to disable SSL/TLS:
          ctm config server:agent::update <ServerName> <AgentName> sslState Disabled
          https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_Agent.htm#configserveragentupdate
        • Alternatively, use the following steps in the Control-M Configuration Manager (CCM) to manually disable SSL/TLS for each agent:
          1. Right click on the Control-M/Agent
          2. Choose "Properties"
          3. Select the "Communication" tab
          4. Change "Secure Socket Layer" to "Disabled"
          5. Click "OK"
     2. Perform the steps in article 000442271.
   • Option 2:
     1. On each Control-M/Server machine, perform the below steps to add the new Certificate Authority to the keystore.
        1. Back up the existing SSL/TLS keystore, identified by the following commands:

           • UNIX/Linux:
             directory: grep -i securitydir $HOME/ctm_server/data/SSL/cert/site.plc
             filename: grep -i keyfile $HOME/ctm_server/data/SSL/cert/site.plc

           • Windows:
             directory: reg query "HKLM\SOFTWARE\BMC Software\Control-M/Server" /v securitydir /s
             filename: reg query "HKLM\SOFTWARE\BMC Software\Control-M/Server" /v keyfile /s

        2. Use keytool to obtain the original CA chain from the Control-M/Server. We will use this later in this solution.
           keytool -printcert -rfc -sslserver <control-m-server-hostname>:<agent-to-server port> > original_ca_chain.pem
           • To determine the agent-to-server port, run "ag_diag_comm" and check the value for "Agent-to-Server Port Number"
        3. Obtain the new CA chain file, in X.509 base-64 encoded PEM format, for the new CA from your security team.
        4. Use keytool to merge the new CA chain into the existing keystore, with a new, temporary alias.
           keytool -importcert -alias <new alias> -file <new CA file> -keystore <existing keystore file>

        5. Run the following commands to recycle the Control-M/Server to apply the keystore change.
           shut_ctm
           start_ctm

     2. Once the Control-M/Server(s) have the new CA added and their recycles complete, deploy certificates on each Agent.
        1. Back up any existing SSL/TLS keystore, identified by the below commands:

           1. UNIX/Linux:
              grep -i keyfile $CONTROLM/data/SSL/cert/site.plc

           2. Windows:
              reg query "HKLM\SOFTWARE\BMC Software\Control-M/Agent" /v keyfile /s

        2. Issue a new certificate for this Agent from your organization's CA and create the new keystore.
           1. Create a Certificate Signing Request (CSR) for the Control-M/Agent.
              Follow the "Generating a Signed Certificate" section of the below documentation.
              It is recommended to use an organization-specific email address in the emailAddress field of the CSR.
              https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#GeneratingaSignedCertificate
              
              

              The CSR, which needs to be sent to your security team in the next step, is generated at:
              Linux/Unix: $CONTROLM/data/SSL/certificate_requests/<hostname>_YYYYMMDD_HHMMSS.csr
              Windows: <Control-M/Agent Install Path>\data\SSL\certificate_requests\<hostname>_YYYYMMDD_HHMMSS.csr
              
              The private key, which is used in step 3 to create a .p12 keystore, is generated at:
              Linux/Unix: $CONTROLM/data/SSL/private_keys/<hostname>_YYYYMMDD_HHMMSS.pem
              Windows: <Control-M/Agent Install Path>\data\SSL\private_keys\<hostname>_YYYYMMDD_HHMMSS.pem

           2. Have your organization's security team use the internal CA and the CSR to issue a certificate.
           3. Create a new keystore, with only the new certificate and its CA chain.
              Follow the "Generating the .p12 Keystore" section of the below documentation.
              Make sure to use a secure password for the keystore.
              https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#Generatingthep12Keystore
        3. Merge the original CA into the keystore, with a temporary alias.
           keytool -importcert -alias <temp alias> -file <original_ca_chain.pem> -keystore <new_keystore.p12>
           Make note of the alias for use later.
        4. Use ctmkeytool to apply the new keystore.
           Follow the "Configuring SSL in Zone 2 and 3" section of the below documentation:
           https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#ConfiguringSSLinZone2and3
        5. At your convenience, recycle each Agent.
           One of the following centralized methods can be used:
           • Use Automation API to recycle:
             ctm config item::recycle "CTMS:Agent:<ServerName>:<AgentName>"
             https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_Agent.htm#configitemrecycle
           • When unable to use the API, use the CCM to recycle.
             1. Right click on the Control-M/Agent.
             2. Choose "Desired State"
                1. Click "Recycle"
             3. If prompted, click "Yes" to confirm.
     3. Once all affected Control-M/Agents have been recycled and are using the new configuration:
        1. Create a new certificate and keystore for the Control-M/Server, using the same new CA used above.
           A video demonstrating these steps can be found at https://www.youtube.com/watch?v=3Jz-to-0e50
           1. Back up the existing SSL/TLS keystore again, using the steps mentioned above for backing up the Control-M/Server keystore.
           2. Create a Certificate Signing Request (CSR) for the Control-M/Server.
              Follow the "Generating a Signed Certificate" section of the below documentation.
              It is recommended to use an organization-specific email address in the emailAddress field of the CSR.
              https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#GeneratingaSignedCertificate
              
              

              The CSR, which needs to be sent to your security team in the next step, is generated at:
              Linux/Unix: $HOME/ctm_server/data/SSL/certificate_requests/<hostname>_YYYYMMDD_HHMMSS.csr
              Windows: <Control-M/Server Install Path>\data\SSL\certificate_requests\<hostname>_YYYYMMDD_HHMMSS.csr
              
              The private key, which is used in step 4 to create a .p12 keystore, is generated at:
              Linux/Unix: $HOME/ctm_server/data/SSL/private_keys/<hostname>_YYYYMMDD_HHMMSS.pem
              Windows: <Control-M/Server Install Path>\data\SSL\private_keys\<hostname>_YYYYMMDD_HHMMSS.pem

           3. Have your organization's security team use the internal CA and the CSR to issue a certificate.
           4. Create a new keystore, with only the new certificate and its CA chain.
              Follow the "Generating the .p12 Keystore" section of the below documentation.
              Make sure to use a secure password for the keystore.
              https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#Generatingthep12Keystore
           5. Use ctmkeytool to configure each Control-M/Server to use the new keystore:
              https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Zone_2_and_3_SSL_configuration.htm#ConfiguringSSLinZone2and3
           6. When convenient, recycle each Control-M Server for a second time.
     4. Once all Control-M/Servers have their new keystores and been recycled the second time, Configure the Agents to only use the new certificate and CA chain.
        • Option 1: Manually remove the old CA cert from the keystore
          Make a backup copy of the the current keystore file and then delete the temp alias.
          keytool -delete -keystore "<keyfile>" -alias <temp alias>
        • Option 2: Using Automation API generate a new CSR, and apply a new Certificate

          1. Create a Certificate Signing Request (CSR) with the following AAPI command:

             ctm config server:agent:csr::create <server> <agent> -f <configuration file> > <agent>_csr.pem
             
             The output will be the CSR used in the next step. See the following documentation for more details on this command such as the syntax of the configuration file.

             https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_AutomatingSSLCertificates.htm#configserveragentcsrcreate
             
             Note: It is recommended to use an Agent-specific and organization-specific email address in the emailAddress field of the CSR.

          2. Have your organization's security team use the new CA and the CSR to issue a certificate.
             Make sure the certificate and CA chain files returned from your security team are also in the x.509 Base-64 encoded PEM format.
          3. Deploy the signed certificate and the .pem containing only the new CA chain to the Agent:
             ctm config server:agent:crt::deploy <server> <agent> <new_signed_cert.pem> <new_ca_chain.pem>
             
             Where "<new_signed_cert.pem>" is the new certificate issued in the previous step in X.509 Base-64 encoded format, and "<new_ca_chain.pem>" Is the new CA certificate chain, also in X.509 Base-64 encoded format. See the following documentation for more details on this command.
             https://documents.bmc.com/supportu/API/Monthly/en-US/Documentation/API_Services_ConfigService_AutomatingSSLCertificates.htm#configserveragentcrtdeploy
        • When convenient, recycle each Agent again.
     5. For Linux/Unix Control-M/Agents, verify the permissions on the security related files are set correctly.
        • For fully supported Control-M/Agent versions (9.0.21 and higher), run the following command as the Control-M/Agent user:
          $CONTROLM/toolbox/permission_check.sh --force
        • For other Control-M/Agent versions, see the alternative solution in article 000441965.

     6. Ensure the Security Level is set to 4 mutual authentication) in the Control-M/Server Security Policy:
        https://documents.bmc.com/supportu/9.0.22/en-US/Documentation/Security_policies.htm

        • UNIX/Linux: Back up and edit the following files:
          $HOME/data/SSL/cert/site.plc
          $HOME/data/SSL/cert/site.plc
          In the [client] and [server] sections, ensure security_level=4
        • Windows: Backup and modify the following registry keys:
          HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\client
          HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\site\server
          HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\client
          HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Server\SecurityPolicy\ns\server
          Ensure security_level=4
        • If the security level was changed, recycle the Control-M/Server

     7. For each Agent the default security level is 4 (mutual authentication).
        If this has been changed from 4, it needs to be set back to 4.

        • Control-M/Agent versions 9.0.21 and higher: security level is hard-coded to 4 and nothing further needs to be done on this step.
        • 9.0.20 and lower: Check and modify the site configuration to set the security level.
          • https://documents.bmc.com/supportu/9.0.20/help/Main_help/en-US/index.htm#4277.htm
          • https://documents.bmc.com/supportu/9.0.20/help/Main_help/en-US/index.htm#4243.htm
          • UNIX/Linux: Back up and edit the following files
            $CONTROLM/data/SSL/cert/site.plc
            $CONTROLM/data/SSL/cert/ag.plc
            In the [server] section, ensure security_level=4
          • Windows: Backup and modify the following registry keys
            HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Agent\SecurityPolicy\site\server
            HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Control-M/Agent\SecurityPolicy\ag\server
            Ensure security_level=4
          • If the security level was changed, recycle the Control-M/Agent.