Please test the job runner in galaxyproject/galaxy#21528
In this PR are some instructions how to test it locally. I can also build a Docker Galaxy container from this branch if this is easier. If you think this is ok, I would like to request a review from some backend devs.
I decided to create a complete new runner, this should make our migration later hopefully easier.
If you think it is working, we can deploy this patch and configure some destinations that take this runner and test it per user.