Make it possible to shrink only a subset of quorum queues by providing a naming pattern

[Ayanda-D]

Proposed Changes

Hello Rabbit Team! 👋

These changes extend the QQ force shrink operations to use regx for queue names on shrink (globally or per vhost). There cases where in the event of Quorum Queues being in a bad state (e.g no leader or bad multiple partitions), only a set need be shrunk on emergency disaster recovery operations, to prevent and rescue total message loss before queue is either re-grown or permanently deleted and re-declared. The extension (to the existing operations) allows the following:

ctl eval 'rabbit_quorum_queue:force_all_queues_shrink_member_to_current_member( QQMatchSpec ).'

ctl eval 'rabbit_quorum_queue:force_vhost_queues_shrink_member_to_current_member( VHost, QQMatchSpec ).'

Default will match all queues as existing behaviour. This change will help rescue only a specific set of queues without necessarily shrinking everything (which can take a longtime on heavily loaded brokers). We are really keen on having this available for recovery procedures to help prevent message loss. Please take a look, thanks!

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
This is simply a reminder of what we are going to look for before merging your code.

• Mandatory: I (or my employer/client) have have signed the CA (see https://github.com/rabbitmq/cla)
• I have read the CONTRIBUTING.md document
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

[michaelklishin]

Some members of the core team cannot be convinced that "mass shrinking" of replicated queues is a good idea.

This PR like can be reimplemented as a custom plugin, both the CLI part and the QQ-related functions that run on RabbitMQ nodes.

[Ayanda-D]

@michaelklishin hello! 👋 i'm not sure if the purpose of these changes has been understood? Our current problem is the "mass shrinking" which we cant avoid when using these default helper shrink functions provided by rabbitmq.

Some members of the core team cannot be convinced that "mass shrinking" of replicated queues is a good idea.

This change is actually helping reduce "mass shrinking", by adding extra clauses to reduce the number of queues shrunk (matching a smaller set, instead of just shrinking "everything"). Without these changes, the shrink all will shrink everything across the broker/cluster with no option to reduce the QQ blast radius for the user.

This PR like can be reimplemented as a custom plugin

We're happy to implement things via plugins. But our current problem is the "mass shrinking" which we cant avoid from the broker. Here, we are providing a means to reduce "mass shrinking" of everything on the broker (current shrink all ends up shrinking even queues which dont need to be shrunk when recovering from disaster).

[michaelklishin]

One final note: together #13873 #15166 with #grow these public API extensions allow a custom plugin to

1. Shrink a subset of queues (maybe even just one queue) down to 1 replica
2. Trigger a leader election for those queues (that are virtually guaranteed to succeed with one replica)
3. Re-grow the number of replicas to 3 (or 5, or another desired odd number) in a single operation

Exposing these new options to rabbitmq-queues shrink is also possible if needed, and reaching consensus around a CLI extension should be lightning fast compared to how long this PR and #13873 #15166 have taken.