In TrueSight Capacity Optimization (TSCO), ETL execution is failing with the following message in the ETL logs

ERROR FATAL - Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded]

For example, here is the error generating during a TSCO vCenter ETL execution:

01/26/2012 09:24:35 START [taskid=52]- Running task: 52 on scheduler 0
01/26/2012 09:24:35 INFO [taskid=52]- Starting process /opt/cpit/etl/jetlrun.sh start 52 207
01/26/2012 09:24:41 INFO ETL Engine starting on server [ut03262], PID=507
01/26/2012 09:24:41 OUTPUT [LOG] /opt/cpit/etl/log/M1Q92452.out;/opt/cpit/etl/log/M1Q92452.err
01/26/2012 09:24:41 REPORT Starting ETL in simulation mode
01/26/2012 09:24:41 INFO ETL Engine: EXTRACT step
01/26/2012 09:24:41 INFO DBconf: New DS_STATUS SOURCE 2
01/26/2012 09:24:41 INFO [Insert status] inserting new status for srcid = 2
01/26/2012 09:24:42 INFO Forcing timestamps to AS time zone: America/Chicago
01/26/2012 09:24:42 INFO Connecting to https://abc.com/sdk
01/26/2012 09:25:45 INFO Connected to abc.xyz.com - VMware vCenter Server 4.1.0 build-345043 UUID: 46C3168A-AC40-4EFE-9D38-0A744ED6036E - api 2.5. Supported sampling intervals: Past day [300] (interval: 5m - level: 1 - retention: 1d); Past week [1800] (interval: 30m - level: 1 - retention: 7d); Past month [7200] (interval: 2h - level: 1 - retention: 30d); Past year [86400] (interval: 1d - level: 1 - retention: 365d) diff:-21600000 - parallelism: 2
01/26/2012 09:25:45 INFO This VirtualCenter version has a known bug in Cluster Memory Utilization performance index. ETL will fix imported values.
01/26/2012 09:25:45 INFO Starting metric download. Forced minimum date: 25/01/2012 09:24:42. Sampling interval: Past week [1800] (interval: 30m - level: 1 - retention: 7d).
01/26/2012 09:39:53 ERROR FATAL - Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded] 
01/26/2012 09:39:53 INFO [Status type] setting status type to FAILED - ts = 2012-01-26 09:39:53
01/26/2012 09:39:53 INFO [DS update status] updating datasource status for srcid = 2
01/26/2012 09:39:56 INFO [taskid=52]- Process terminated correctly in 919.59[s]
01/26/2012 09:39:56 STOP [taskid=52]- Ended task: 52

For a service extractor ETL the message will generally be reported in the $CPITBASE/scheduler/log/scheduler.out or scheduler.err file.

The recommended workaround is to increase the Java Heap Size for this ETL Engine as each ETL runs in its own separate JVM launched by the TrueSight Capacity Optimization (TSCO) Scheduler.  Service Extractor ETLs such as the VMware Service Extractor run within the Scheduler as a separate thread.

NOTE: If the /TSCO Installation Directory]/customenvpre.sh file doesn't exist, then copy the customenvpre.sh.sample file to customenvpre.sh and then proceed to make the recommended changes to the customenvpre.sh described below.

- Increasing the ETL Heap Size for a Regular ETL
You can watch this video to see how to set the heap size for regular batch ETL:

The steps to increase the ETL JVM's Heap Size are:

(1) Edit the /TSCO Installation Directory]/customenvpre.sh file on the ETL Engine where the ETL is running:

(2) Change this:

#ETL_HEAP_SIZE="512m"
#export ETL_HEAP_SIZE

To this:

ETL_HEAP_SIZE="1024m"
export ETL_HEAP_SIZE


(3) Restart the Scheduler service on the machine:

cd /[BCO Installation Directory]
./cpit restart scheduler

(4) Check the scheduler status

cd /[BCO Installation Directory]
./cpit status scheduler

In general , Technical Support recommends increasing from the default of 512 MB to 1 GB and then if you continue to experience out of memory errors only then increasing beyond that to either 1536 MB or 2048 MB.  If need to increase beyond 2 GB we would generally recommend contacting Technical Support to discuss the environment and the problem you are seeing.  This is simply because 2 GB is a common java heap size memory configuration and we have less experience in Technical Support with environments having a java heap size larger than 3 GB.

Note that each running ETL runs within a separate JVM and thus when setting the Java Heap Size is it important to consider how many concurrent ETLs will be running in the environment in relation to the memory configuration of the computer.  The ETL_HEAP_SIZE setting corresponds to the Java command line 'Xmx' setting so this is the maximum heap size of the JVM not the starting heap size of the JVM.

Q: Is there a need to also modify the ETL_PERMGEN_SIZE setting?

Typically no.  If the ETL_PERMGEN_SIZE setting is to low the ETL would fail with the following error:

java.lang.OutOfMemoryError: PermGen space

For the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error that indicates a lack of Java Heap Space, not a lack of Java Perm Gen space.

Q: Is there need to restart the BCO Scheduler after changing the ETL_HEAP_SIZE?

No, regular ETLs run in a separate java process that reads the customenvpre.sh file on startup so any ETL started after the file has been updated with be executed with the increased Java Heap Size.

Q: Is there any way to validate that the new ETL_HEAP_SIZE parameter has been properly set and has been picked up by the ETL?

If the ETL_HEAP_SIZE has been correctly set when the ETL starts you'll be able to see the increased heap size on the 'java' process command line.

So, something like this:

> ps -ef | grep java | grep etl
cpit     25369 25365 15 13:40 ?        00:00:01 /data1/bmc/BCO/jre/bin/java -Dsrcid=121 -Dcpit.component=etl -Xmx512m -XX:MaxPermSize=128m -Djava.awt.headless=true -Djava.io.tmpdir=/data1/bmc/BCO/temp -Dfile.encoding=UTF-8 -DETLBASE=/data1/bmc/BCO/etl -DPID=25365 ...

The "-Xmx512m" means that the ETL_HEAP_SIZE in the custompenvpre.sh is set to 512 MB.

After changing it to 1024 run 'ps -ef | grep java | grep etl' shows the increased heap size:

So, updated customenvpre.sh:

#ETL

ETL_HEAP_SIZE="1024m"
export ETL_HEAP_SIZE

New 'ps -ef | grep java | grep etl' output:

cpit     25935 25931 96 13:42 ?        00:00:04 /data1/bmc/BCO/jre/bin/java -Dsrcid=121 -Dcpit.component=etl -Xmx1024m -XX:MaxPermSize=128m -Djava.awt.headless=true -Djava.io.tmpdir=/data1/bmc/BCO/temp -Dfile.encoding=UTF-8 -DETLBASE=/data1/bmc/BCO/etl -DPID=25931...


- Increasing the BCO Scheduler Heap Size for a Service Extractor

You can watch this video to see how to set Scheduler Heap size In Capacity Optimization for Service ETLs

For a Service Extractor, such as the VMware - vCenter Extractor Service:

(1) Edit the /TSCO Installation Directory]/customenvpre.sh file on the ETL Engine where the ETL is running.

(2) Change these lines from

#SCHEDULER_HEAP_SIZE="1024m"
#export SCHEDULER_HEAP_SIZE

To this:

SCHEDULER_HEAP_SIZE="2048m"
export SCHEDULER_HEAP_SIZE

(3)Remember to Issue a ./cpit restart scheduler to apply new heap settings.

Q:Is 2048 MB of Scheduler Heap Size sufficient for all Services Extractor ETLs?

No, 2048 MB is a good starting point but if the TSCO Scheduler continues to terminate it may be necessary to increase the Scheduler Heap Size setting to a larger value (such as 3096 MB, 4192 MB), or for a very large Service Extractor (or a machine running multiple Service Extractors) to an even higher value.

Some recommended guidelines:

• Reserve at least 1 GB  of memory for the OS and other tasks running on the ETL Engine machine
• Contact Technical Support if you plan to increase the Scheduler Heap Size to more than 1/2 of the total physical memory on the machine