How to improve such a significant slow performance issue when running a large volume of cyclic jobs (e.g., 500 cyclic jobs, 35000 jobs as a total per day) running every 1 minute for 24 hours in Control-M/Enterprise Manager (EM)  8.0.00/9.0.00 ?

When setting up such a large volume of cyclic jobs running every 1 minute for 24 hours, those jobs actually ended but it took some time to get updated at GUI although all the prior conditions were satisfied. 
When the usage rate of CPU and memory are less than 50%, Control-M performance got very slower. If those cyclic jobs were deleted and not executed, Control-M won't have any performance issue.

Given such a special circumstance, is there any way regarding how to improve such a Control-M slow performance issue with a large volume of cyclic jobs running every 1 minute for 24 hours?

[Control-M/Enterprise Manager (EM) side]
Since Control-M components are very dependent on the Operating System (OS) performance, database performance, network latency and etc., so it’s very complicated to say if we recommend this type of implementation.
 
First, the architecture can be explained in high level that for each job every status change is sent from the Control-M/Server CE process to the Gateway. The Gateway handles this updates to the database and send it to all the GUI Server(s) connected, The GUI Server handles it with few queues that related to the Gateway the job comes from, it will filter the information the user can see per the collections defined, the authorization and the filter the user defined in his view and will send the information about the job update to all the viewpoint opened in all the connected clients (only to the viewpoint that pass the filtering).
 
And for each job if it’s a simple job without too many status changes the Control-M/Server will send about 5 updates so we are talking about 2500 about every minute only for these 500 jobs without taking in consideration, the other jobs running in the environment.
 
If possible, share the load of the number of cyclic jobs between a few Control-M/Server(s) however  be aware that the GUI Server still handles all the job updates from all the Control-M/Server(s).
 
We can increase the number of worker threads the Gateway is working to update the database by adjusting the EM system parameter(gtw type) “GtwNumUpdateThreads”.
 
We can suggest to work with few collections and filters so every viewpoint will be related to small number of jobs and will decrease the traffic between the GUI Server to the clients.
 
We can also increase the number of threads used in the collection queue by setting the EM system parameter(gsr type)  “AllJobsCollNumProcThreads” to the number of Control-M/Servers; but we cannot increase the value to more than number of Control-M/Servers connected to EM. So if the environment has only one this setting won’t help.
 
[Control-M/Server side]
To understand where could be bottleneck and which configuration to rune for this situation, in addition to the number of jobs and their nature; we also need other environment information like database latency, database server location (local or remote), machine power(free CPU/memory) etc.
 
The following list of configuration parameters could be fined tune to see if the performance result could be better.
 
There is no need to recycle the Control-M/Server whenever these parameters are updated. This makes the tuning much easier.
There is a need to update the config.dat file and issue the below command:
 
ctmipc -dest all -msgid cfg
 
The following parameters values are based on a high scale performance test performed previously and might be a good starting point:
 
SEL_IO_THREADS 8
POST_PROCESS_THREAD_POOL_SIZE 6
DB_UPDATE_THREADS 3