[TASK] Document how to configure retention time in the HC config section

[hisImminence]

Related to

While configuring retention time is getting easier with 8.8, there is no general guidance on it and documentation is missing on how to configure it via the helm charts.

We should create a section retention time in the config section of kuberenets with helm/configure with a config example (follow this template guide if possible).

Overview

Applications state this in the docs:

Limitation: ILM configuration updates are not applied automatically
If you update the value of ilmMinAgeForDeleteArchivedIndices in the application.yml after deployment, the change will not be applied to the existing ILM policy used by Operate. This is a known issue.

To change the ILM settings after installation, you must manually update the corresponding ILM policy in Elasticsearch. Be aware that if Camunda 8 is installed without ILM configured initially, applying ILM later may lead to issues unless handled carefully and manually.

Zeebe does correctly apply ILM updates via configuration, but Operate does not.

For reliable ILM behavior, Camunda recommends configuring ilmMinAgeForDeleteArchivedIndices during initial installation and verifying the applied policy in Elasticsearch.

This needs to be extended so that instead of only warning, the docs also:

• Show how to verify if an ILM policy exists in Elasticsearch/OpenSearch.
• Show how to create the policy manually if it does not exist (must be done manually if no fresh install).

Clarify version differences:

• In all versions, Zeebe, Operate and Tasklist create policies automatically on initial install.
• Tested in Zeebe 8.5.6 / Operate 8.5.5 / Tasklist 8.5.4 with Opensearch:
  • Zeebe and Tasklist create their policies automatically when configured after initial install (once the component exports a new record to Opensearch).
  • Operate does not create its policy automatically when configured after initial install.
  • Zeebe will update its policy automatically (new retention.enabled or retention.minimumAge) when configured after initial install (once Zeebe exports a new record to Opensearch).
  • The other components do not update their policies.
• I know the behavior is still the same and Operate 8.7. I did not test Tasklist 8.7 to see if it behaves any better.

Many information, queries and settings can be found in the support tickets linked in this thread

Actions

• have a retention guide for 8.8 in the helm charts
• lower prio: have a retention guide for 8.7 in the helm charts
• @daniel-ewing has reviewed the PR and approved
• Link this guide in the upgrade guide from 87-88

Reference

• Data layer team issue

PR: camunda/camunda-docs#6998