Consider preventing repo location update when it is used by searchable snapshots

[ywangd]

When deleting a repository, we check for its searchable snapshots usage and reject the deletion if any usage is found. However, the same is not done when updating the repo's location. This can potentially lead to red cluster due to missing snapshots. We should consider adding such check to prevent it from happening.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

The check that blocks unregistration of an in-use repository is too strict (and should have a manual override) - see #122714. I suspect if we addressed that then we wouldn't have encountered the case that led to this issue in the first place.

Blocking updates to the repository location is tricky because different repository implementations have different notions of "location", and it's arguable which bits of the repository metadata can or cannot be updated under this rule. For instance, with a S3 repository, would we permit switching to a different client? What if the new client had a different S3 endpoint or proxy settings?

It's also not possible for us to prevent all changes to the underlying data anyway because this is ultimately out of Elasticsearch's control. For instance, with a FS repository you can mount a different filesystem at the same location, or use symlinks, to change the underlying data too. Conversely I can imagine situations where folks may want to mount the data at a different path and then update the Elasticsearch config to point to the new path.

Thus if we did impose a restriction on changing the repository location we must recognise it's only a best-effort thing and make it something that can be overridden if needed.

[ywangd]

Thanks for the thoughtful reply, David. You are right that there are many nuances in updating repo location beyond the shared-file-system repo. I (incorrectly) phrased this issue as just updating the repo location, but the "fix" I had in mind is to prevent any update that is not "update-in-place", i.e. insert a searchable snapshot check here

The reason is that such an update is effectively deletion plush creation and I think the deletion part should go through the same check as directly deleting the repo. Does that make sense?

[DaveCTurner]

No I think that'd be far too strict too. canUpdateInPlace only checks whether it's only updating dynamic settings, which in practice means just the max_restore_bytes_per_sec and max_snapshot_bytes_per_sec which we want to be able to change on a running repository instance so that we can rethrottle ongoing restores. Yet searchable snapshots look up the repository instance afresh on each operation (see #116918) so we can (and indeed do sometimes) swap that out for a new instance with different config easily enough.