Skip to content

[FEATURE] Add min_primary_shard_size transition condition #1496

@mergu

Description

@mergu

Is your feature request related to a problem?
We have a multi-tenant environment where most tenants have their own index. Our ingestion lambda handles creating daily indices for tenants as needed. Different tenants can have different primary counts based on volume. We are using an ISM policy to manage indices after they are created for the day.

From my understanding and what I've tested, we are unable to use the min_primary_shard_size option on the rollover action. This is because the action blocks the transitions from running, waiting on a primary to cross the threshold, which may never happen after we create the tenant's next daily index. Indices get stuck on the rollover action.

To get around this, we use a min_size transition and move the index into a "rollover" state that does the rollover without any conditions. This allows indices to move through all phases. But the min_size transition here is not ideal because different tenants have different numbers of primary shards. We would really like all of the primary shards to have the same size limit.

What solution would you like?
I would like a transition condition for min_primary_shard_size so we can move an index to our rollover phase when a single primary, not a total, crosses the threshold.

What alternatives have you considered?
We could have our lambda call the rollover api every now and then, offload the task from ISM. We could also split our ISM policy for the different tenants but this is not ideal.

Do you have any additional context?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions