Managing the index lifecycle | Elasticsearch Guide [7.0] (original) (raw)

Managing the index lifecycle

edit

The index lifecycle management (ILM) APIs enable you to automate how you want to manage your indices over time. Rather than simply performing management actions on your indices on a set schedule, you can base actions on other factors such as shard size and performance requirements.

You control how indices are handled as they age by attaching a lifecycle policy to the index template used to create them. You can update the policy to modify the lifecycle of both new and existing indices.

For time series indices, there are four stages in the index lifecycle:

• Hot—​the index is actively being updated and queried.
• Warm—​the index is no longer being updated, but is still being queried.
• Cold—​the index is no longer being updated and is seldom queried. The information still needs to be searchable, but it’s okay if those queries are slower.
• Delete—​the index is no longer needed and can safely be deleted.

The lifecycle policy governs how the index transitions through these stages and the actions that are performed on the index at each stage. The policy can specify:

• The maximum size or age at which you want to roll over to a new index.
• The point at which the index is no longer being updated and the number of primary shards can be reduced.
• When to force a merge to permanently delete documents marked for deletion.
• The point at which the index can be moved to less performant hardware.
• The point at which the availability is not as critical and the number of replicas can be reduced.
• When the index can be safely deleted.

For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says:

1. When the index reaches 50GB, roll over to a new index.
2. Move the old index into the warm stage, mark it read only, and shrink it down to a single shard.
3. After 7 days, move the index into the cold stage and move it to less expensive hardware.
4. Delete the index once the required 30 day retention period is reached.

index lifecycle management does not support mixed-version cluster usage. Although it may be possible to create such new policies against newer-versioned nodes, there is no guarantee they will work as intended. New policies using new actions that do not exist in the oldest versioned node will cause errors.