INT-175 RabbitMQ Message Splitting#7
INT-175 RabbitMQ Message Splitting#7DmitryDodzin wants to merge 10 commits intometalbear-co:mainfrom
Conversation
| ## Guide-level explanation | ||
| [guide-level-explanation]: #guide-level-explanation |
There was a problem hiding this comment.
Should these be Reference-level?
There was a problem hiding this comment.
Because on line 18 there is already a guide level subtitle
t4lz
requested review from
a team and
vladrbg
and removed request for
a team
Co-authored-by: Yuan Lyu <lyuyuan92@gmail.com>
| name: cluster-definition-name-with-credenitals | ||
| spec: | ||
| clientProperties: {} | ||
| connectionString: amqp://guest:guest@rabbit:5672/%2F |
There was a problem hiding this comment.
Maybe will be better to store all the sensitive data in secrets instead of CR? Also if this CR is only for config purposes, maybe the non-sensitive data can be stored in configmap also and we won't need this CR at all?
There was a problem hiding this comment.
I agree we probably need a solution that does not require keeping auth secrets directly in a CRD.
| One main impact on the cluster itself will be a performance one, since we are moving messages through the external api we are subject to multiple "transactions" we create where we want to guarantee delivery, and the replication that can happen if the queues are defined as highly available. | ||
|
|
||
| ## Rationale and alternatives | ||
| [rationale-and-alternatives]: #rationale-and-alternatives |
There was a problem hiding this comment.
I remember we considered rabbitMQ plugin? why did it fall? I think it'd be worth mentioning.
Did we consider re-configuring the exchange and use the filtering nature of it?
There was a problem hiding this comment.
The main problem with custom plugins is the involvement required in setting it up, I think the easiest way would be to provide custom images with the plugin already installed (and have several variants and versions like the real one)
There was a problem hiding this comment.
I agree but that's what this section in the RFC for - to specify what we thought and said ney to
Co-authored-by: vladrbg <vladr@metalbear.com>
| name: cluster-definition-name-with-credenitals | ||
| spec: | ||
| clientProperties: {} | ||
| connectionString: amqp://guest:guest@rabbit:5672/%2F |
There was a problem hiding this comment.
I agree we probably need a solution that does not require keeping auth secrets directly in a CRD.
|
|
||
| In this graphic the idea is that some Publisher is aware of the existence of the Input Exchange only where as the Target knows of the Input Exchange and the Target Queue, as rabbit by default works on "get or create" method where when one of the Target Pods starts it may create the Target Queue and then also bind it to the Input Exchange (or maybe also create it if it's not done with Publisher beforehand) | ||
|
|
||
| There is more to it some patterns are more like RPC requests where the Publisher embeds the reply_to argument (it can be done in many ways just the reply_to is the rabbit provided way) and then is expected by the target pod to send the reply to provided queue. |
There was a problem hiding this comment.
...and then is expected by the target pod to send the reply to provided queue
Queue, not exchange?
There was a problem hiding this comment.
I'm not sure but it just worked when I tested it
| └────────────────┘ └────────────────┘ | ||
| ``` | ||
|
|
||
| ### "Shovel" |
There was a problem hiding this comment.
Why a new name? We're using "forwarder" for both SQS and Kafka
| status: | ||
| activeRmqSplits: | ||
| queueNames: | ||
| first-queue: | ||
| exchangeName: my-prefix2-dummy-exchange | ||
| originalName: target-queue | ||
| outputName: my-prefix-mirrord-sink-my-suffix |
There was a problem hiding this comment.
Queue registry status should no longer be used. It's a public CRD, and can be modified/deleted by the admin
|
|
||
| ### MirrordRabbitMQSession | ||
|
|
||
| The main purpose of the rabbitmq session is to create and handle the lifespan of created queues and the state of their creation so the operator can assertively define the state. I want to store all needed data for the split in the session since I want to avoid management api and only use the declarative AMQP api only. |
There was a problem hiding this comment.
We have a new thingy for tracking external resources - MirrordClusterExternalResource.
The lifetime of the tmp resources should be bound to:
- Lifetime of the session, if the queue is for the user. This ensures that the queue is not deleted before the session is closed
- Lifetime of the
MirrordClusterWorkloadPatchRequestresource, if the queue is for the patched target. This ensures that the queue is not deleted while there are patched pods
There was a problem hiding this comment.
Reminder to rename the file to next available RFC number before merging
Co-authored-by: t4lz <t4lz.git@gmail.com>
6 participants
Queue splitting is pretty neat so it would be nice to do so on RabbitMQ too