index-cleanup.py

🚀 Introduction

This script automates the expiration of Elasticsearch documents/indexes by cleaning up old data or removing expired indexes. It supports both timestamp-based cleanup (though not recommended by Elastic) and the removal of entire indices based on date information in the index name.

⚙️ Configuration

Configuration is stored in JSON format. See example in index-cleanup.json.example. All the indexes that are subject to cleanup must be described under "indexes" map:

"indexes": {
    "index01-": {
        "retain_days": 90,
        "timestamp_source": "suffix",
        "date_format": "%Y.%m.%d"
    },
    "index02-": {
        "retain_days": 183,
        "timestamp_source": "timestamp"
    }    
}

The script currenly supports two ways of expiring data:

1. Based on date stored in index name as a suffix. In this case the index name is expected to be <index_name><index_date> where

• index_name - is the name of the index in the "indexes" map :)
• index_date - the date that should be formatted according to "date_format" field in the "indexes" map Example: index01-2024.10.10

2. Based on timestamp field. The name of the timestamp field can be defined in timestamp_field field. Default value: "timestamp"

🔧 Usage

usage: index-cleanup [-h] [-c CONFIG] [-y] [--delete-max-docs DELETE_MAX_DOCS]

Automates expiry of old entries in ElasticSearch/OpenSearch indexes.

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        JSON config file
  -y, --no-confirmation
                        Disables confimation of indexes/documents deletion
  --delete-max-docs DELETE_MAX_DOCS
                        How many documents to expire in timestamp-based expiries

Have a good day!