Conversation
... by using `keep_while` conditions on bindings and auto-delete exchanges. [Why] The `delete_queue` transaction's anonymous function has to be be extracted by Horus, like any Khepri transaction. This is an expensive operation, but Horus uses caching to avoid most work after the first extraction. The problem is when there are many concurrent executions of the same transaction, before it has been executed once: the cache is not hot and Horus has to extract the same transaction many times in parallel currently. An example of this situation is when there are massive disconnections from RabbitMQ clients that trigger massive queue deletions. This can put a lot of load on RabbitMQ. [How] This patch removes the entire transaction. Instead, it uses `keep_while` conditions on bindings and auto-delete exchanges to let Khepri handle the deletion of semantically related tree nodes. RabbitMQ just has to make a simle "delete this queue" command.
dumbbell
force-pushed
the
remove-delete-queue-transaction
branch
from
082e24d to
3a75b6c
Compare
|
I was looking into eliminating the transaction for queue deletion a while back but I didn't get the changes over the finish line. I just pushed up the branch I was working on: https://github.com/rabbitmq/rabbitmq-server/tree/md/khepri-q-delete-command. It's quite old but maybe it has some interesting leads for this branch. I also looked into using stored procedures where we use transactions now - wouldn't help us with the BEAM file changes in OTP 28 but it avoided some performance impacts. The code-server work for transactions is expensive but also the size of transaction commands can be quite large (the compiled module binary + env) and we could do a lot less work with commands + keep-while conditions. But, if I am remembering correctly, we would need some sort of migration for existing Khepri DBs to transition from transactions to keep-while and regular commands so that they could add the keep-while conditions. |
Yes, exactly. This patch is incomplete on purpose (the needed feature flag is a simple hard-coded Thank you very much for sharing your work! I will take a look at it :-) |
2 participants
(to be written; CI doesn’t run on a commit without an explicit pull request…)