How to increase retention_lease period

[koonwen]

First of all, thank you for this project. It's a nice piece of technology.

I note that in the docs for CCR,

You can’t resume replication after it’s been paused for more than 12 hours. You must stop replication, delete the follower index, and restart replication of the leader.

FWIU, this limit is related to the index.soft_deletes.retention_lease.period to ensure that details of deleted docs are retained so that the follower cluster can replay these from the translog (ref). However, I also note that there exist the retention_lease_max_failure_duration, with 1h by default but it's max value is 12h. I'm wondering which of these two settings is responsible for the reason why replication can't be paused for more than 12 hours and whether this limit can be increased. My teams indices are 100Tb so restarting CCR from scratch is really not ideal.

Separately, we have also encountered cases when we reconfigured and restarted nodes and replication failed without explanation and cannot be resumed replication even though from the time the replication failure occurred to the time we tried to resume replication was within the 12h window

_plugins/_replication/<index>/_status?pretty
{
  "status" : "FAILED",
  "reason" : "",
  "leader_alias" : "repl_conn",
  "leader_index" : "<index>",
  "follower_index" : "<index>"
}

We suspect that maybe it was because some _state directories were lost but we're not sure whether CCR relies on this. Could I get any indication of what this issue might be related to? Some of the logs we found mentioned

[2025-05-13T20:09:24,763][WARN ][o.o.p.PersistentTasksClusterService] [master-us2-1] trying to update state on task replication:[<index>][540] with unexpected allocation id 11421
[2025-05-13T20:09:25,386][WARN ][o.o.p.PersistentTasksClusterService] [master-us2-1] trying to update state on task replication:[<index>][913] with unexpected allocation id 11426

[andrross]

Catch All Triage - 1 2

[mohit10011999]

hi @koonwen , i don't think the max. limit for ndex.soft_deletes.retention_lease.period is 12 h, default is 12h but it can be increased arbitrarily to 7d, 30d and so on.

Trade-off:
Higher values mean more disk space usage and larger index segments, as soft-deleted operations are retained longer

Recommendation:-
Keep it just long enough for follower clusters or recovering nodes to catch up — typically 12h to 7d based on your replication lag and recovery frequency.

[mohit10011999]

PUT /<index-name>/_settings
{
  "index.soft_deletes.retention_lease.period": "7d"
}

[koonwen]

@mohit10011999
Indeed we can increase the index.soft_deletes.retention_lease.period but for CCR itself, it has the following setting that has a hardcoded limit of 12h that it will try apply to the retention_lease

PUT _cluster/settings
{
  "persistent": {
    "plugins.replication.follower.retention_lease_max_failure_duration": "20h"
  }
}

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "failed to parse value [20h] for setting [plugins.replication.follower.retention_lease_max_failure_duration], must be <= [12h]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "failed to parse value [20h] for setting [plugins.replication.follower.retention_lease_max_failure_duration], must be <= [12h]"
  },
  "status": 400