Discovery: How to automate compaction? How to schedule compaction on the crontab?

Knowledge Article

Discovery: How to automate compaction? How to schedule compaction on the crontab?
How to automate compaction? How to schedule compaction on the crontab?
BMC Discovery
BMC Discovery
all versions
BMC Discovery
BMC Discovery
all versions
How to automate the compaction?
Do not use this article in these cases:
1- If multi-generational database is enabled. Use this blog post instead. See "Crontab for Datastore Compaction and Archive/backup"
2- If using Helix Discovery. The compaction is already automated in this case.

Step 1: If not already done, implement a backup plan
 
See this documentation page
Extract: "you must create a backup of the datastore before running [the compaction]".

    Step 2: Create a script that stops the services, compacts, and restarts the services

    Here is an example to do this (in an SSH session on the appliance/member as tideway user):
    mkdir /usr/tideway/myCustomFolder
    echo "/usr/tideway/bin/tw_service_control --stop" > /usr/tideway/myCustomFolder/myCustomCompactScript.sh
    echo "/usr/tideway/bin/tw_ds_compact --offline --smallest-first --force"  >> /usr/tideway/myCustomFolder/myCustomCompactScript.sh
    echo "/usr/tideway/bin/tw_service_control --start" >> /usr/tideway/myCustomFolder/myCustomCompactScript.sh
    chmod a+x /usr/tideway/myCustomFolder/myCustomCompactScript.sh

    Note:
    • This documentation page suggest to "run tw_ds_compact inside the screen utility" but this is not needed when using cron.
    • cron requires absolute paths
    • If the script is being used on a member of a non-fault tolerant cluster, the cluster will be down until the script restarts the member.
    • Ideally, before the compaction starts, the script should check that:
    • The latest backup is "recent enough" and it contains the expected files
    • The latest backup logs do not contain any errors or warnings

    Step 3: Schedule the execution of the script for the desired time
    See Discovery: How frequent should the datastore be compacted?

    For example: to compact the first day of each month
    echo "0 0 1 * * /usr/tideway/myCustomFolder/myCustomCompactScript.sh" > /usr/tideway/etc/cron/myCustomCronCompactJob.cron

    Alternatives settings: to compact ...
    - ... every 3 months on the first day of the month, replace "0 0 1 * *" with "0 0 1 */3 *"
    - ... every quarter at fixed dates (i.e. on the first day of January/April/July/Oct at 1:30am, replace "0 0 1 * *" with "1 30 1 1,4,7,10 *"

    In the case of a non-fault-tolerant cluster, it is recommended to cron the script in all the members at the same time to reduce the down time.

    In the case of a fault tolerant cluster, this is the opposite: to avoid any down time, sequentially cron the script. For example, compact the first member on the first day of the month, compact the second member on the second day of the month and so on.

    Step 4: Execute "tw_cron_update"
    For more details, see this documentation page.
    Extract: "Cron entry files in $TIDEWAY/etc/cron are applied to the live cron configuration by running $TIDEWAY/bin/tw_cron_update"

    Optional: verify that the script was actually added in the crontab with this command:
    crontab -l | grep myCustomCompactScript

    Step 5: Verify that it works
    These commands will provide a way to check that cron did execute the script as expected:
    su -
    grep myCustomCompactScript /var/log/cron

    This command will provide a way to check when the compaction was done and the result:
    grep -E "starting|complete|ERROR|not completed" /usr/tideway/log/*compact*

    Step 6: From time to time, re-execute step 4

    Step 7: If multi-generational database is later enabled, roll back this customization.
     
    BMC Discovery
    BMC Discovery
    all versions
    Attachment(s):