[ENHANCEMENT] Add data retention for Optimize

[bendzsak-dpc]

Describe the use case:

We would like to enable ILM based elasticsearch retention policy to prevent the elasticsearch cluster overloading.

Describe the enhancement/feature:

The official camunda documentation give a very good option to enable and configure this:
https://docs.camunda.io/docs/self-managed/operate-deployment/data-retention/

Unfortunately this feature cannot be configured from the helm chart:
https://github.com/camunda/camunda-platform-helm/blob/main/charts/camunda-platform/charts/operate/templates/configmap.yaml#L9

Desired outcome and acceptance tests:

ILM archive function can be enabled and configured in the helm chart.

[ManuelDittmar]

Hi @bendzsak-dpc,

I faced an similar issue and was able to handle it via environment variables:

optimize:
  enabled: true
# Possible starting from 3.11.0-alpha4
  env:
  - name: CAMUNDA_OPTIMIZE_HISTORY_CLEANUP_CRON_TRIGGER
    value: "0 1 * * *"
  - name: CAMUNDA_OPTIMIZE_HISTORY_CLEANUP_TTL
    value: "P2Y"
  - name: CAMUNDA_OPTIMIZE_HISTORY_CLEANUP_PROCESS_DATA_CLEANUP_ENABLED
    value: "true"
  - name: CAMUNDA_OPTIMIZE_HISTORY_CLEANUP_PROCESS_DATA_CLEANUP_CLEANUP_MODE
    value: "all"
  - name: CAMUNDA_OPTIMIZE_HISTORY_CLEANUP_PROCESS_DATA_CLEANUP_BATCH_SIZE
    value: "10000"
# prior to 3.11.0-alpha4
  extraVolumeMounts:
        - name: config-volume
          mountPath: /optimize/config/environment-config.yaml
          subPath: environment-config.yaml
  extraVolumes:
    - name: config-volume
      configMap:
        name: optimize-config

operate:
  enabled: true
  env:
# Retention
  - name: CAMUNDA_OPERATE_ARCHIVER_ILMENABLED
    value: "true"
  - name: CAMUNDA_OPERATE_ARCHIVER_ILMMINAGEFORDELETEARCHIVEDINDICES
    value: 31d 
# Archive https://docs.camunda.io/docs/self-managed/operate-deployment/importer-and-archiver/#archive-period 
  - name: CAMUNDA_OPERATE_ARCHIVER_WAITPERIODBEFOREARCHIVING
    value: 5s

tasklist:
  enabled: true
  env:
# Retention
  - name: CAMUNDA_TASKLIST_ARCHIVER_ILMENABLED
    value: "true"
  - name: CAMUNDA_TASKLIST_ARCHIVER_ILMMINAGEFORDELETEARCHIVEDINDICES
    value: 31d
# Archive NOT Documented
  - name: CAMUNDA_TASKLIST_ARCHIVER_WAITPERIODBEFOREARCHIVING
    value: 5s

# Reduce for Zeebe and Gateway the configured replicas and with that the required resources
# to get it running locally
zeebe:
  affinity:
    podAntiAffinity: null
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 3Gi
  resources:
    requests:
      cpu: "100m"
      memory: "1Gi"
    limits:
      cpu: "512m"
      memory: "3Gi"
  env:
# Retention
  - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_ENABLED
    value: "true"
  - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_MINIMUMAGE
    value: 31d
  - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_POLICYNAME
    value: zeebe-records-retention-policy

Configmap for Optimize:

apiVersion: v1
data:
  environment-config.yaml: |
    ---
    ###############################################################################
    #                                                                             #
    #                     Optimize Environment Configuration                      #
    #                                                                             #
    ###############################################################################

    # Note: for all possible configuration options please have a look
    #       at the Optimize documentation:
    #       https://docs.camunda.org/optimize/${docs.version}/technical-guide/setup/configuration/
    historyCleanup:
      # cron expression for when the cleanup should run
      cronTrigger: '*/2 * * * *'
      # default time to live (ttl) for data, when reached the corresponding process/decision/event instances will get cleaned up
      # Format is ISO_8601 duration  https://en.wikipedia.org/wiki/ISO_8601#Durations
      ttl: 'P1D'
      processDataCleanup:
        # switch for the Camunda process data cleanup, defaults to false
        enabled: true
        # type of process data cleanup to perform, possible values:
        # 'all' - delete everything related to the process instance
        # 'variables' - only delete associated variables of a process instance
        cleanupMode: 'all'
        # process definition specific configuration parameters that will overwrite the general parameters (ttl, processDataCleanupMode)
        # for the specific processDefinition key
        perProcessDefinitionConfig:
        # 'myProcessDefinitionKey':
        #   ttl: 'P2M'
        #   cleanupMode: 'variables'
      decisionDataCleanup:
        # switch for the Camunda decision data cleanup, defaults to false
        enabled: false
        # decision definition specific configuration parameters that will overwrite the general parameters (ttl)
        # for the specific decisionDefinition key
        perDecisionDefinitionConfig:
        # 'myDecisionDefinitionKey':
        #   ttl: 'P2M'
      ingestedEventCleanup:
        # switch for the ingested event data cleanup, defaults to false
        enabled: false
kind: ConfigMap
metadata:
  name: optimize-config
  namespace: default

[aabouzaid]

The feature has been added in v8.3.0 for Operate, Tasklist, and Zeebe.

For Optimize, as I know it doesn't use the ES ILM but its own mechanism and it will be added soon.

[hisImminence]

Thanks for the update @aabouzaid, moving to the backlog until Optimize supports ES ILM.

[jessesimpson36]

@hisImminence @theburi , in this thread:
https://camunda.slack.com/archives/C02UMKN3DTL/p1705006824487639

@Zelldon and @akeller brought up some concerns about SM ILM policies differing from SaaS, and whether there should be a default ILM policy in SM. Perhaps we should clarify as part of this ticket (or on a different ticket) what we believe the default ILM policy should be. As in, should it mirror SaaS? Should it be disabled by default? That step could be done separately from actually implementing this ticket.

[aravindhrs]

#827 (comment)

Hi @ManuelDittmar, How can we setup ES ILM for camunda Self Managed 8.2.0? I was unable to find the application.yaml file for operate/tasklist/zeebe.

Is the above ES ILM data retention settings for version 8.3 will also work for 8.2?

operate:8.2.0
tasklist:8.2.0
zeebe:8.2.0
optimize:3.10.0
elasticsearch:7.17.1