Automatically rollover all logsdb data streams when disabling logsdb

[martijnvg]

Logsdb is automatically enabled for all data streams in the logs-*-* pattern space. If for some reason automatic logsdb enabled needs to be disabled, then the cluster.logsdb.enabled can be configured to false. This is a dynamic cluster setting. All new backing indices of logs-*-* data streams will fallback to standard mode after rollover.

However it can take a while before all logs-*-* data streams have been rolled over. Depending on the ilm policy or dsl configuration. So typically you may want to do a manual rollover. But that is not trivial, because the rollover api doesn't accept index patterns, meaning that only one data stream can be rolled over at a time.

The feature request here, is to make rolling over all logs-*-* data streams easier after the cluster.logsdb.enabled setting is configured to false.. Either by having logic that would automatically do a rollover or via an admin level api that would do rollover for all logs-*-* data streams.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[salvatore-campagna]

In the "Obs & Es" Team Meeting on 6th November, we discussed the option of allowing wildcards, such as logs-*-*, if the data stream rollover happens lazily. See the lazy option for data stream rollover.

[dakrone]

Kostas and I met to discuss some requirements for something like this. Here are some notes:

• We'd like to have this (eventually) for both on-prem and Serverless.
• For on-prem, it's something a user may want to use.
• For Serverless, it's okay for it to be internal only to Elastic.
• Getting this into Serverless first would be good, and then on-prem at a later time, as logsdb is not going to be on-by-default until 9.0.
• It's not a requirement for Serverless GA, but sooner would be nicer there.

On the Data Management side, I think we have a couple of thoughts:

• If we want to do this quickly on the Serverless side, we can make the Transport action support wildcards for lazy rollover, and introduce an internal-only API that makes use of that (leaving the public REST API not supporting wildcards). This would allow us to avoid any breaking changes.
• We'll need to determine what kind of output we want for a "multi-rollover". We could either try to make the existing /_rollover API support this (it may be difficult with the existing output of that API), OR we can introduce a new /_multi_rollover API that has a new request/response format and allows us to have a good output that doesn't break the existing data stream API.

[felixbarny]

This may also relate to the first class data streams project where we've discussed that we'd want to be able to apply changes to an index/component template to all affected data streams. Either by changing dynamic settings/additive mappings on all backing indices or by triggering a lazy rollover for the affected data streams.

Something to consider for the long-term solution as I think we want to approach this particular case with a tactical solution that doesn't require changes in ES.

[kkrik-es]

@dakrone any news on the timeline for the Serverless-only fix? Thanks!

[dakrone]

@kkrik-es we haven't yet prioritized this, @bytebilly @mattc58 is this something we want to fit in somewhere?

[bytebilly]

Do we know the required effort?

[martijnvg]

@salvatore-campagna With the tooling that was added to serverless, this is now only a feature request for stateful Elasticsearch?