[Failure store] Remove unused write indices from failure store

[gmarouli]

Description

Data stream failure store is using the lazy rollover to reduce its footprint. Let's see how:

1. Data stream gets created with 1 backing index and 0 failure indices. It also has the rollover_on_write set to true for the failure store.
2. When a failure is encountered, the data stream will execute the rollover of its failure store because of rollover_on_write, and it will write the failed document.
3. The data stream now has 1 backing index and 1 failure index and rollover_on_write set to false.
4. As time passes DLM will roll over the failure index and this will create another failure index.
5. The data stream now has 1 backing index and 2 failure index and rollover_on_write set to false
6. As time passes DLM will delete the initial failure index.

So far so good. However, if we encountered no more failure the last failure index created cannot be removed because DLM does not remove a write index, the only option is to roll them over.

Enhancement
We would like to be able to go back to 0 failure indices if we witness no failures for reasonable time period. This reduces the costs of having a failure store when we have no failures.

Draft idea

Introduce a lazy rollover with conditions, meaning that when this request is executed it will set the rollover_on_write only if the conditions are met.

• This can be used later by DLM to lazy rollover failure store when the conditions apply.
• The getWriteFailureIndex should only return the write index if the rollover_on_write is set to false.
• This means that the latest failure index can easily move through the DLM since it's not the write index anymore.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)