Unregister repository API has overly strict checks on searchable-snapshot use

[DaveCTurner]

In #73714 we introduced a check to make sure we didn't unregister a repository if any of its snapshots are mounted as searchable snapshots. As currently implemented this doesn't work well if the same repository is registered multiple times, because all those registrations will have the same repository UUID which makes them all appear to be in use and therefore none of them can be unregistered.

In fact, we should be able to relax these checks a little:

• As long as one of the copies of a repository remains registered there should be no problem with unregistering the others. This ought to be particularly true since Split searchable snapshot into multiple repo operations #116918 which made it so that we look up the repository instance afresh, by UUID, each time we need to access it.

• If all the shards of searchable snapshot indices over snapshots in a repository are unassigned then there seems little value in preventing the user from unregistering it. In this case, the repository may be broken in some manner which can only be addressed by unregistering and re-registering it, and it is unnecessarily painful to have to unmount and remount all the searchable snapshots as part of this process.

• We could also let users manually override the check with an API parameter, forcing the unregistration of an in-use repository and unassigning all the affected shards in a single step. This would be appropriate in cases where the repository is found to be broken in some manner that can only be addressed by unregistering and re-registering it, but not so broken as to cause all the shards to become unhealthy. After doing this, the cluster would of course be in red health, but it will recover those shards to health once a copy of the repository is registered again.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

In addition to the proposed changes, I wonder whether we should (if possible) prevent a repository being created with duplicated UUID in the first place?

[DaveCTurner]

I don't think it'd be possible to make such a check watertight, because we do not guarantee to read the contents of a repository during registration (e.g. if ?verify=false) and the repository contents can technically change underneath us after registration anyway (e.g. restoring a different backup, or mucking around with symlinks). But yeah we could consider adding some best-effort protection of that nature.