Is there a way to implement FIFO stack / job re-prioritization?

[dmeziere]

Hello,

I use Gearman to drive an ETL farm. I think it is the perfect solution, and you did a really great job, but i've got one need not covered.

• 3 gearman-job-server/jammy,now 1.1.19.1+ds-2build1 amd64
• 3 php 8.0 Gearman clients
• 35 php 8.0 Gearman workers

We are lacking processing power, and therefore workers, so there is frequent trafic jam. Each hour, our client declares a thousand jobs, but the queue is not always terminated. Gearman seems to work as a LIFO stack. So we always have a few jobs that are delayed again and again, by newer jobs being declared. And that numbers grows from hour to hour until low trafic hour or a crash (not on Gearman side, it is rock-solid).

Is there a way to use Gearman as a FIFO stack, or to repriorize existing jobs before adding new ones ? Here is what i mean :

• By FIFO stack i mean that newly added jobs will only be handled after existing ones

• By repriorizing existing jobs i mean that each hour, when our client starts, it could first make all existing jobs as high priority, then add new jobs as normal priority, emulating a FIFO stack

• or anything else that could solve my problem

[esabol]

Jobs are assigned to workers in the order they are given to the server (FIFO). However, the task system in libgearman as used by PHP clients is an abstraction above jobs, and it sends these "tasks" as jobs. It sends them all at one time, and it happens to send them LIFO. Refer to the discussion in issue #319.

Basically, if you change how you submit the tasks/jobs in your clients (hint: use doBackground), you should get FIFO.

Alternatively, you are welcome to contribute a PR which changes the order that tasks are added in libgearman to be FIFO.

I also think you need to add more workers until the rate of jobs you can complete exceeds the rate of jobs that you add. Try doubling or tripling the number of workers you have.

[dmeziere]

Thank you for this track to explore. Adding more workers, in my case, means adding more physical servers (i already pushed the number of process per machine to a confortable ratio), and the costs will explode. That said, if i can prevent a not-yet-ended import to be pushed again and again by incoming ones, it will be a major upgrade !

When i say "35 PHP workers", i was meaning 7 physical servers each hosting 5 VM using each 3 worker processes.

[SpamapS]

You may also be running into the round robin problem. If you have multiple functions per worker, the default behavior is to assign all the jobs in one function before sending another function. Try passing --round-robin to gearmand.

…

On Tue, Jul 9, 2024, 7:29 AM David Mézière ***@***.***> wrote: Thank you for this track to explore. Adding more workers, in my case, means adding more physical servers (i already pushed the number of process per machine to a confortable ratio), and the costs will explode. That said, if i can prevent a not-yet-ended import to be pushed again and again by incoming ones, it will be a major upgrade ! — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADS6YGDW7447V2627GGBUTZLPXVLAVCNFSM6AAAAABKSOLRSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJXHA4DSNJTGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dmeziere]

Does doBackground() have other behavioural differences with addTask() / runTasks() ? I mean the jobs are executed, but gearadmin can't see them, and it looks like the callbacks are not executed. I use them a lot to generate a Gantt diagram, showing all the jobs in realtime. There nothing works at the monitoring level.

[edit] I now can see the job with gearadmin. The documentation (that is a bit light to my taste) states that all the callbacks handling only works with runTasks(). I really need this behaviour, it is a problem to me.

[esabol]

Just to be clear, the PHP extension is a separate project, and we are not responsible for it (except that it uses libgearman.so under the hood and we are responsible for that). If doBackground does not fit your needs, you are welcome to submit a PR which changes the behavior of libgearman, as mentioned previously.

[SpamapS]

Can you provide some partial sample client code? If you're already using GearmanClient::do, and not addTask/runTasks, then this is not the LIFO task problem, and I'm very suspicious it's the round-robin problem.

…

On Wed, Jul 10, 2024 at 8:48 AM Ed Sabol ***@***.***> wrote: Just to be clear, the PHP extension is a separate project, and we are not responsible for it (except that it uses libgearman.so under the hood and we are responsible for that). If doBackground does not fit your needs, you are welcome to submit a PR which changes the behavior of libgearman, as mentioned previously. — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADS6YHJ4UT4R426V22ZGULZLVJWJAVCNFSM6AAAAABKSOLRSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQHA4DKMZZGU> . You are receiving this because you commented.Message ID: ***@***.***>

[dmeziere]

@esabol I am not blaming anyone or anything. I love Gearman ! I am just trying to understand and locate where my problem is, and to find the cheapest solution to it. Believe me, if i could provide any quality code in C / Boost, i would be proud to contribute, if it was nessessary. The only thing i said is that the Gearman documentation on the PHP website (that i understand is not gearmand related) could be enhanced.

[dmeziere]

@SpamapS I am not using GearmanClient::do. I experienced it this week thanks to your help on this issue, but i did not go very far because i use a lot the callbacks and communication provided by GeamanClient tasks to manage my jobs. I achieved running my jobs with GearmanClient::do, but without any feedback of course.
I have a second method, also, but it is nominative (one method per "workshop" (a group of workers handled by a PHP master process in my project) used to warmup an import, before running the real jobs, that have the same function name for all the farm.

[SpamapS]

The callback stuff isn't hard to do with GermanClient::do .. doBackground was a problem because you never get results from it.

…

On Fri, Jul 12, 2024 at 3:02 AM David Mézière ***@***.***> wrote: @SpamapS <https://github.com/SpamapS> I am not using GearmanClient::do. I experienced it this week thanks to your help on this issue, but i did not go very far because i use a lot the callbacks and communication provided by GeamanClient tasks to manage my jobs. I achieved running my jobs with GearmanClient::do, but without any feedback of course. I have a second method, also, but it is nominative (one method per "workshop" (a group of workers handled by a PHP master process in my project) used to warmup an import, before running the real jobs, that have the same function name for all the farm. — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADS6YFXIG7SDNAG4K5XYVLZL6STZAVCNFSM6AAAAABKSOLRSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGI2DENJWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[esabol]

I'm kind of wondering if this is actually a problem with the PHP extension after all. The implementation for the addTask method in the PHP extension has a comment that says "prepend task to list of tasks on client obj", which would seem to imply that it's the one that's setting the order to LIFO instead of FIFO.

gearman_client_add_task_handler (https://www.php.net/manual/en/gearmanclient.addtask.php):
https://github.com/php/pecl-networking-gearman/blob/a52052cdd712a95091ce926be3bcdca41c730696/php_gearman_client.c#L736

[SpamapS]

No, that's a bit of a ruse, that's just how it's managing its own data structures. It happens here:

https://github.com/gearman/gearmand/blob/master/libgearman/packet.cc#L190-L199

Tasks are stored in the universal here until run_tasks is run. For whatever reason, they decided to prepend rather than append. As we've said before, if you want to use tasks FIFO, then you have to add them in reverse order.

The docs don't define this order, but I don't think we could change it without most likely breaking some folks.

We could probably add a new universal option to reverse the order, and if nothing else, maybe we should document that they are LIFO.

[esabol]

I think undocumented behavior is subject to change, personally, and I really doubt anyone wants LIFO. Just my two cents.