[feature] Queue jobs from a project to workers on another project

[olaurendeau]

Hello !

We use your awesome bundle in many of our projects to handle gearman tasks an jobs and really appreciate it.

Lately we needed to queue a job from a symfony project and have the work done by a worker located on another symfony project.

We discovered that there is a check in GearmanClient:L193 that throw a JobNotFoundException when we try to enqueue the job.

So we made a work-around on our project, overloading GearmanClient with our own one.

We could find some time to implement this feature, so if you think it could be useful, could you point us which solution should be the most appropriated.

• Simply remove use of getJob() on enqueue
• Have a parameter in config.yml allowing to use workers from others project
• Setup workers from other project in config.yml
• ...

Thanks !

[mmoreram]

HI there!

Thanks for your comment. Is very useful for me to see that this Bundle is being used by other companies :)

Nowadays, Jobs are configured by annotations. When I developed the first version, I figured it was a great idea, but right now, I think it's not. Maybe the best approach for your problem ( and for sure, most people problem ) is to allow workers definition by yml / xml / php as well.

With this new feature, you could define more workers with a simple format, allowing to create workers from other servers.

Is this a good workaround for U?

Maybe we could create a new Milestone to work on it :)

Looking forward for your feedback :) Thanks.

[olaurendeau]

If so workers configuration will not be related anymore to worker code ?

For example :

I have a project "Awesome frontend"
With an UI allowing users to send emails.
When an user want to send an email "Awesome frontend" use his GearmanClient to enqueue an "AwesomeBackendBundle~sendEmail" job.

And a project "Awesome backend"
With a worker sending emails able to treat "AwesomeBackendBundle~sendEmail" jobs

In this case worker configuration is in "Awesome backend" project, and I don't see a good reason for "Awesome frontend" to be aware of this configuration.
He should juste be able to enqueue jobs using his callable name.

So being able to chose annotation, xml, yml or php to configure workers could be nice, but I'm not sure that really solve our problem.

[mmoreram]

When you talk about projects, maybe could be more useful to talk about Bundles.

If you have many projects using same queues, maybe you should focus on "How to share your code". If you Isolate your workers in a simple Bundle, you can require this Bundle in as many projects as you need, and as far as workers configuration are defined in the worker itself, this should work.

A worker is just a class that takes data from a "queue" from a server, so I think isolating the workers should be the best solution.

Am I right?

[olaurendeau]

Well for sure it could be a possibility. But we don't want to overload our projects with useless stuff.

If I a take back my example, my "Awesome frontend" do not need to be aware of email logic. He just need to be able to ask for a "AwesomeBackendBundle~sendEmail" to be executed . If we were using a shared bundle for each or various workers, it will mean to add theirs dependencies in each project also, so in this case to add Email logic in our "Awesome frontend" who doesn't need them.

At small scale it's probably a solution but I think it quickly will have performance and maintainability issues.

[olaurendeau]

up

[mmoreram]

hmmm, I understand...

let me think about this. If you have any idea, please let me know.

[olaurendeau]

Thinking more about that, I've found an even better example.
Gearman is language agnostic, you can have a Symfony project queuing jobs executed by a java project. So you will not be able to share workers definition between project.

So I bring back my suggestions :

• Simply remove use of getJob() on enqueue method
• Have a parameter in config.yml allowing to use workers from others project
  Something like that :

gearman:
    external_workers_allowed: true

• Setup "extra jobs" in config.yml
  Then use a hasJob method here instead of getJob where we could check existence of this job in defined Workers or in configuration

gearman:
    extra_jobs:
        - AwesomeBackendBundleWorkerMessageWorker~sendEmail
        - doAnUselessJavaTask
        - doSomethingInRuby

Then we should be able to call AwesomeBackendBundleWorkerMessageWorker~sendEmail from a symfony project and have the job done in another one.

[olaurendeau]

up ?

[mmoreram]

I did not see your answer. Sorry, no time enough :(

One of new features for new version is to define new queues using configuration ( yml, php, xml ), so this could fix your problem. We could define new local workers, checking their integrity ( valid classes, etc ) and new foreign workers, just defining queue name.

Agree?

Your format sounds good to me :)

[olaurendeau]

Look nice, is this feature logged somewhere ?
Could we help for implementation ?

[mmoreram]

@olaurendeau #114

Once the code is done, we could all implement/document it and maybe write GearmanBundle's first cookbook :) How it sounds? ^^

Anyone wants to help? :)

Thanks for your feedback !

[bmeynell]

Wait. Can you just use the same name in different projects? Instead of AwesomeBackendBundleWorkerMessageWorker~sendEmail you would call it MessageWorker~sendEmail on both projects.

[mmoreram]

yes, you can name a job with specific name, so if you define it, you can name it with same string :)