Skip to content

[Failure Store] Expose failure store lifecycle information via the GET data stream API #126668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

gmarouli
Copy link
Contributor

@gmarouli gmarouli commented Apr 11, 2025

To retrieve the effective configuration you need to use the GET data streams API, for example, if a data stream has empty data stream options, it might still have failure store enabled from a cluster setting. The failure store is managed by default with a lifecycle with infinite (for now) retention, so the response will look like this:

GET _data_stream/*
{
  "data_streams": [
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      .....
      "failure_store": {
        "enabled": true,
        "lifecycle": {
          "enabled": true
        },
        "rollover_on_write": false,
        "indices": [
           {
            "index_name": ".fs-my-data-stream-2099.03.08-000003",
            "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
            "managed_by": "Data stream lifecycle"
          }
        ]
      }
    },...
]

In case there is a failure indexed managed by ILM the failure index info will be displayed as follows.

      {
          "index_name": ".fs-my-data-stream-2099.03.08-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "my-lifecycle-policy",
          "managed_by": "Index Lifecycle Management"
        }

@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label v9.1.0 labels Apr 11, 2025
@gmarouli gmarouli added >non-issue :Data Management/Data streams Data streams and their lifecycles auto-backport Automatically create backport pull requests when merged v8.19.0 labels Apr 11, 2025
@elasticsearchmachine elasticsearchmachine added Team:Data Management Meta label for data/management team and removed needs:triage Requires assignment of a team area label labels Apr 11, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Apr 14, 2025
Copy link
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a couple small comments. I think also now that we've removed the feature flag some of the code can be simplified.

@@ -29,6 +29,7 @@
* supports the following configurations only explicitly enabling or disabling the failure store
*/
public record DataStreamFailureStore(Boolean enabled) implements SimpleDiffable<DataStreamFailureStore>, ToXContentObject {
public static final String FAILURES_LIFECYCLE_API_CAPABILITY = "failures_lifecycle";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: It looks like this is only used in the rest action. Does it make sense to move this there? Alternatively, it's describing something about the structure of the response, should it be moved to the response object/action class?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, name wise, should we make this something like failure_store.lifecycle to denote specifically what part of the request is new? I see we have other changes to the response as well, so this is a soft suggestion. If you think this should be more generic to describe those changes that's also fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged :Data Management/Data streams Data streams and their lifecycles >non-issue serverless-linked Added by automation, don't add manually Team:Data Management Meta label for data/management team v8.19.0 v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants