Description
Preconditions and environment
- Any version
Steps to reproduce
I'm not exactly sure how to address this, if it's a bug or not, but it stems from my initial observation of a memory leak in #34851 and #37639
In my task of creating an xml feed for 3 million products I observed #34851 and #37639 so I thought of an idea to break the operation into chunks of batches that can be processed asynchronously via queue messages.
I decided to follow this approach
Step 1. Gather all product IDs
Step 2. Publish a message with let's say 3m IDs
Step 3. In the handler process 1000 of them, then publish a message with (3m - 1000) product IDs until the queue is exhausted.
I thought that since it's an asynchronous operation, the handler will be invoked for every single message and terminate so the batch would be processed then terminate as php is an interpreter language so when one php process terminates it will release all memory used, thus give precise control of memory limits.
However in reality I observed this is not the case and it suffers from the same problems a foreach loop suffers.
To replicate this I created a mini module which in the handler, sets an array and variable inside the array.
Find attached the module which has:
- a cli command
php bin/magento ioweb:replicatememorymessagewhich will set 100 messages in the queue - a handler that will allocate some memory in an array
- a definition for a queue
Try to now execute
php bin/magento ioweb:replicatememorymessage
and then
php bin/magento que:co:st replicatememoryissue
You will notice that it outputs an ever increasing memory usage
Memory usage: 130.70127105713 MB
Memory usage: 140.70547485352 MB
Memory usage: 150.70964813232 MB
Memory usage: 160.71382141113 MB
Memory usage: 170.71799468994 MB
Memory usage: 180.72216796875 MB
Memory usage: 190.72634124756 MB
Memory usage: 200.73051452637 MB
Memory usage: 210.73499298096 MB
Memory usage: 220.73916625977 MB
Memory usage: 230.74333953857 MB
Memory usage: 240.74751281738 MB
Memory usage: 250.75168609619 MB
Memory usage: 260.755859375 MB
Memory usage: 270.76003265381 MB
Memory usage: 280.76420593262 MB
Memory usage: 290.76898956299 MB
Memory usage: 300.7731628418 MB
Memory usage: 310.77733612061 MB
Memory usage: 320.78150939941 MB
Memory usage: 330.78568267822 MB
Memory usage: 340.78985595703 MB
Memory usage: 350.79402923584 MB
Memory usage: 360.79820251465 MB
Memory usage: 370.80237579346 MB
Memory usage: 380.80654907227 MB
Memory usage: 390.81072235107 MB
Memory usage: 400.81489562988 MB
Memory usage: 410.81906890869 MB
Memory usage: 420.8232421875 MB
Memory usage: 430.82741546631 MB
Memory usage: 440.83158874512 MB
Memory usage: 450.83698272705 MB
Memory usage: 460.84115600586 MB
Memory usage: 470.84532928467 MB
Memory usage: 480.84950256348 MB
Memory usage: 490.85367584229 MB
Memory usage: 500.85784912109 MB
Memory usage: 510.8620223999 MB
Memory usage: 520.86619567871 MB
Memory usage: 530.87036895752 MB
Memory usage: 540.87454223633 MB
Memory usage: 550.87871551514 MB
Memory usage: 560.88288879395 MB
Memory usage: 570.88706207275 MB
Memory usage: 580.89123535156 MB
Memory usage: 590.89540863037 MB
Memory usage: 600.89958190918 MB
Memory usage: 610.90375518799 MB
Memory usage: 620.9079284668 MB
Memory usage: 630.91210174561 MB
Memory usage: 640.91627502441 MB
Memory usage: 650.92044830322 MB
Memory usage: 660.92462158203 MB
Memory usage: 670.92879486084 MB
Memory usage: 680.93296813965 MB
Memory usage: 690.93714141846 MB
Memory usage: 700.94131469727 MB
Memory usage: 710.94548797607 MB
Memory usage: 720.94966125488 MB
Memory usage: 730.95383453369 MB
Memory usage: 740.9580078125 MB
Memory usage: 750.96218109131 MB
Memory usage: 760.96635437012 MB
Memory usage: 770.97589874268 MB
Memory usage: 780.98007202148 MB
Memory usage: 790.98424530029 MB
Memory usage: 800.9884185791 MB
Memory usage: 810.99259185791 MB
Memory usage: 820.99676513672 MB
Memory usage: 831.00093841553 MB
Memory usage: 841.00511169434 MB
Memory usage: 851.00928497314 MB
Memory usage: 861.01345825195 MB
Memory usage: 871.01763153076 MB
Memory usage: 881.02180480957 MB
Memory usage: 891.02597808838 MB
Memory usage: 901.03015136719 MB
Memory usage: 911.034324646 MB
Memory usage: 921.0384979248 MB
Memory usage: 931.04267120361 MB
Memory usage: 941.04684448242 MB
Memory usage: 951.05101776123 MB
Memory usage: 961.05519104004 MB
Memory usage: 971.05936431885 MB
Memory usage: 981.06353759766 MB
Memory usage: 991.06771087646 MB
Memory usage: 1001.0718841553 MB
Memory usage: 1011.0760574341 MB
Memory usage: 1021.0802307129 MB
Memory usage: 1031.0844039917 MB
Memory usage: 1041.0885772705 MB
Memory usage: 1051.0927505493 MB
Memory usage: 1061.0969238281 MB
Memory usage: 1071.1010971069 MB
Memory usage: 1081.1052703857 MB
Memory usage: 1091.1094436646 MB
Memory usage: 1101.1136169434 MB
Memory usage: 1111.1177902222 MB
Memory usage: 1121.121963501 MB
The problem here is that even with queues we cannot guarantee that if any part of the framework is consuming and not cleaning up memory, will release the memory when a batch is processed thus leaving us to the mercy of all memory leaks in Magento.
Expected result
Memory usage is not increasing per message as the allocation is consistent
Actual result
Memory usage keeps increasing
Additional information
No response
Release note
No response
Triage and priority
- Severity: S0 - Affects critical data or functionality and leaves users without workaround.
- Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
- Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
- Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
- Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.
Activity