BMC Helix IT Operations Management Deployment: Create an Elasticsearch ILM Policy to Limit logstash-* or helix-* Indices in BMC Helix Logging

Knowledge Article

Article Number

000410548

Old Article Number

Article Type

FAQ/Procedural

Title

BMC Helix IT Operations Management Deployment: Create an Elasticsearch ILM Policy to Limit logstash-* or helix-* Indices in BMC Helix Logging

Summary

Describes how to create an Elasticsearch Index Lifecycle Management policy to control retention and limit the number of logstash-* or helix-* indices stored in the BMC Helix Logging system.

Product

BMC Helix Platform OnPrem Common Containers

Component

BMC Helix Platform OnPrem Common Containers

Applies to

All supported versions

Question

ow do I create an Elasticsearch Index Lifecycle Management (ILM) policy to limit the number of logstash-* or helix-* indices retained by the BMC Helix Logging system?

Excessive index retention can cause high disk usage and impact Elasticsearch cluster health. When disk usage reaches the high watermark threshold, Elasticsearch may relocate shards and generate warnings similar to the following:

[WARN ][o.e.c.r.a.DiskThresholdMonitor] [efk-elasticsearch-master-0]
High disk watermark [90%] exceeded on [efk-elasticsearch-data-1][/bitnami/elasticsearch/data].
Free space: 1.8 GB [9.4%].
Shards will be relocated away from this node.
Currently relocating shards totaling [0] bytes.

If this warning appears in an EFK-related BMC Helix Logging pod, it indicates disk pressure. To avoid further impact, configure an Index Lifecycle Management policy to automatically manage index retention and limit the number of logstash-* or helix-* indices stored by the system.

Answer

Follow these steps to create an Elasticsearch Index Lifecycle Management policy and apply it to logstash-* or helix-* indices.

1) Log in to the BMC Helix Logging Kibana user interface. Create the index data view as described in the Viewing Logs on Kibana

section of the product documentation.

2) Click on the three bars at the top left of the screen and scroll down to select the Dev Tools menu option.

3) Paste the JSON below into the Console changing the 7d value if you want to retain something other than 7 days of indices:

PUT _ilm/policy/BMC_logstash_cleanup
{
"policy": {
"phases": {
"hot": {
"actions": {
"set_priority": {
"priority": 100
}
},
"min_age": "0ms"
},
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}

4) Click on the first line and then the play icon that is displayed at the end of the line to create the policy:

5) Repeat the process using the JSON below to create an index template:

PUT _index_template/logstash_index_template
{
"index_patterns": ["logstash-*"],
"template": {
"settings": {
"index.lifecycle.name": "BMC_logstash_cleanup",
"index.lifecycle.rollover_alias": "logstash_ilm"
}
}
}

and again to apply the template to existing logstash-* indices:

PUT logstash-*/_settings
{
"index": {
"lifecycle": {
"name": "BMC_logstash_cleanup"
}
}
}

6) To verify that the policy is in place again click on the three bars and select Stack Management -> Index Management to list your current indices. Click on one of the logstash-* items and the index details, including the active lifecycle management policy. will be displayed:

Note: If your index uses a different naming pattern such as helix-*, replace the index patterns accordingly. For example, for indices named "logstash-*"):

PUT _ilm/policy/BMC_logstash_cleanup
{
"policy": {
"phases": {
"hot": {
"actions": {
"set_priority": {
"priority": 100
}
},
"min_age": "0ms"
},
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}

PUT _index_template/logstash_index_template
{
"index_patterns": ["helix-*"],
"template": {
"settings": {
"index.lifecycle.name": "BMC_logstash_cleanup",
"index.lifecycle.rollover_alias": "logstash_ilm"
}
}
}

PUT helix-*/_settings
{
"index": {
"lifecycle": {
"name": "BMC_logstash_cleanup"
}
}
}

Attachment(s):