Why does a Job sometimes turn Yellow (Executing) and stay that way for a long time (longer than expected) and then all the sudden (~15 minutes) it's Status gets Updated and it turns Green or Red? (Track All Mechanism)
 

There are two Port numbers defined for the communication between the Control-M/Server and the CONTROL-M/Agent(s) - the Server-to-Agent Port and the Agent-to-Server Port.

a) The CONTROL-M/Agent listens on the "Server-to-Agent" Port, 7006 by default. This Port number is used for any type of communication that is initiated by the CONTROL-M/Server to the CONTROL-M/Agent (e.g. Job Submission request).

b) The CONTROL-M/Server listens on the "Agent-to-Server" Port, 7005 by default.  This Port number is used for any type of communication that is initiated by the CONTROL-M/Agent to the CONTROL-M/Server (e.g. Job started Executing notification).

Most communication between the CONTROL-M/Server and CONTROL-M/Agent flow in a one-way direction.  That is to say, from the CONTROL-M/Server to the CONTROL-M/Agent (i.e. Job Submission) or from the CONTROL-M/Agent to the CONTROL-M/Server (i.e. Job Complete).   This process is called unidirectional socket communication.

However, there is another mechanism used to ensure that Job Status updates are all collected.  It is a safety measure enacted to handle certain situations when a Job update might have been lost due to a Network communication problem.  This mechanism is called "Track All".

Understanding How Track All Works:

Once every 900 seconds (15 minutes), the CONTROL-M/Server will open a connection to each CONTROL-M/Agent on the Server-to-Agent Port.  The CONTROL-M/Server then holds the connection open and waits for the CONTROL-M/Agent to send back all it's Status updates before closing the connection.  This process is called bidirectional socket communication.  CONTROL-M/Server then continues to the next CONTROL-M/Agent with a Status of Available in the list.

If the Agent-to-Server port is blocked by a Firewall, the Job Statuses on the CONTROL-M/Agent will still update - it will just take 15 minutes (by default) for the Update to occur.

In a similar way, both Sysout Viewing and JCL Viewing requests are initiated by the CONTROL-M/Server (like "Track All").  In both cases the CONTROL-M/Agent responds using the same socket used to initiate the request - the Server-to-Agent Port.

The "Track All" parameter can be changed from 900 seconds to a fewer amount of time (for all connected CONTROL-M/Agents).  The parameter can be found in ctm_menu -> "Parameter Customization" -> "Default Parameters for Communicating with Agent Platforms" and is named "Polling Interval".
 

Additional information:

BMC Software strongly recommends that the Polling Interval not be lowered except in rare instances as directed by Customer Support.  Lowering the Polling Interval can cause a performance degradation as the CONTROL-M/Server TR and NS processes will spend too much time performing "Track All" requests.  (The TR process on the CONTROL-M/Server handles the Track All mechanism (among other things) and the NS process on the CONTROL-M/Server handles communication with CONTROL-M/Agents).

If a situation arises where the Agent-to-Server port cannot be opened through a Firewall, BMC Software recommends implementing Persistent Connections between the CONTROL-M/Agent and the CONTROL-M/Server. This will resolve the issue of the CONTROL-M/Agent needing to "call home" to the CONTROL-M/Server to pass Job Updates.

The following video demonstrates this solution: