New configuration options for message queue consumers aren't flexible enough

[hostep]

Summary (*)

Magento 2.4.1 shipped with a bunch of new options for message queue consumers based upon an architectural proposition.

But after playing around with it, I'm seeing some inconsistencies and things which don't work like how I originally envisioned this. I'm basing my findings on what I found by playing around, maybe I missed something, maybe not everything is documented properly yet, so there might be mistakes here.
Currently you can configure a bunch of settings in the queue_consumer.xml file, only some things in the env.php file and you have very little flexibility to override a particular setting if you aren't happy with the global default or default setting from the queue_consumer.xml file.

So here is a proposal about how these options can be configured in such a way that they are flexible enough for most cases.

This is about these configuration options btw:

• maxMessages
• maxIdleTime
• sleep
• onlySpawnWhenMessageAvailable
• consumers_wait_for_messages

	Global default	Consumer default	Global override	Consumer override
Summary	Some global default for these configuration options should always exist	The queue_consumer.xml file in a module should be allowed to override these global defaults	A devops engineer might want to override the global defaults in the app/etc/env.php file. This override should only apply when the queue_consumer.xml doesn't have a specific value set.	A devops engineer might want to override the config values of a particular consumer using the app/etc/env.php file.
Example	maxMessages: 10000 (defined somewhere in code)	See this for example, it overrides it to 5000	Should be possible to define this in the env.php file: 'queue' => [ 'max_messages' => 10, ],	Should be possible to define this in the env.php file: 'queue' => [ 'config' => [ 'consumers' => [ 'product_action_attribute.update' => [ 'max_messages' => 5, ], ], ], ],
Feature exists in Magento (y/n)	This already exists for most/all options I think?	This functionality exists and is being used in some core Magento modules	As far as I can tell, this is currently not possible	As far as I can tell, this is currently not possible

In the example above, the end result should be that the product_action_attribute.update consumer runs with a setting of maxMessages set to 5

Please ignore for a bit that you can override the maxMessages option in the env.php file in the cron_consumers_runner section, I think we should deprecate that somehow or figure out a way to make it backwards compatible with the new proposal from above as it's more flexible.

Here's an example of a bunch of cases with how the outcome should be:

Global default	Consumer default	Global override	Consumer override	Outcome
1000	-	-	-	1000
1000	500	-	-	500
1000	-	300	-	300
1000	-	-	200	200
1000	500	300	-	500
1000	500	-	200	200
1000	500	300	200	200
1000	-	300	200	200

I hope this makes sense somehow.

The goal for me would be to be able to very specifically tailor per consumer how it behaves in a certain project on a certain environment (dev/staging/production). Currently we have very little flexibility and have to work with a bunch of defaults Magento set and no easy way to override them.

I'm open for discussion about this, but I'm leaving in a big holiday in a few days. I'll do my best to start answering somewhere from halfway December or January...

Examples (*)

See above

Proposed solution

See above

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users with no workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @hostep. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

[hostep]

This is still relevant, would be nice if somebody could verify.

[nuzil]

Hey @hostep
Got some time to review your proposal.

So as result:

Global default

yes, correct and logisc is already built in a way that there are default values exists in code.

Consumer default

Also correct, already implemented.

One exclusion I found:
"consumers_wait_for_messages" - is only global one

Global override

Here I'm not sure you are right.
You can overwrite globals in env.php:

'queue' => [
        'amqp' => [
            'host' => '127.0.0.1',
            'port' => '5672',
            'user' => 'guest',
            'password' => 'guest'
        ],
        'consumers_wait_for_messages' => 1,
        'only_spawn_when_message_available' => 0,
        'max_idle_time' => 15
    ],

OR 

'cron_consumers_runner' => [
        'cron_run' => true,
        'max_messages' => 100,
        'consumers' => [
            'async.operations.all'
        ]
    ],

One exclusion:
"max_idle_time" is only defined per consumer, which is strange for me. Looks like a missing implementation.

Consumer override

This one is true, and only way to overwrite settings, are define your own module and redefine consumer in *.xml file with required options.

Conclusion

We definitely need one standard related to configurations processing:

• "max_idle_time" - have to be aligned and added to global scope also
• Consumer override - will be great to implement
• consumers_wait_for_messages - have to be re-checked. Cause its strange that you cannot define it per consumer

Table you provided totally make sense for me!

Thanks a lot!

[m2-assistant]

Hi @nuzil. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[hostep]

Awesome, thanks for the review @nuzil !

You are correct about the "Global override", but indeed max_idle_time is not implemented here yet.
The current way to configure consumers is a bit inconsistent or not flexible enough for various scenario's, so let's try to improve this! 🙂

I'm currently swamped with work unfortunately, so if somebody else could try to pick this up, that would be awesome!