Consider migrating from Jobber to Cloudprober

[rdimaio]

Follow-up from rucio/containers#369

Jobber is no longer actively maintained and the last commit dates to 4 years ago: https://github.com/dshearer/jobber

Other than preventing the arm64 build of the probes image, this is not causing major issues at the moment. However, it might be worth moving to a more actively maintained project, like https://github.com/cloudprober/cloudprober.

[ericvaandering]

I've been working on this. Cloudprober wasn't ideal either. First, it's a tool for a different job and expects that the STDOUT of a script it ran would be a prometheus metric. Of course, we could not bother to scrape them and then it is a cron-like thing.

But the bigger problem with any solution other than jobber was the lack of jobber's H option. This allows us to run a probe every minute (for example) but at a consistent and random time within the minute. I.e. probe foo will always run at 36 seconds past the minute and bar will always run at 19 seconds past.

What I decided to do (and have working in a test instance) is write a simple python program that uses the text of the command to run to pick a consistent but "random" delay, sleep that amount, then run the code. Then we can just use regular cron.

As a first step, I'd propose building containers and setting up helm charts that allow either jobber or cron and at some point we deprecate the jobber portion. Thoughts, @rdimaio ?

[rdimaio]

I've been working on this. Cloudprober wasn't ideal either. First, it's a tool for a different job and expects that the STDOUT of a script it ran would be a prometheus metric. Of course, we could not bother to scrape them and then it is a cron-like thing.

But the bigger problem with any solution other than jobber was the lack of jobber's H option. This allows us to run a probe every minute (for example) but at a consistent and random time within the minute. I.e. probe foo will always run at 36 seconds past the minute and bar will always run at 19 seconds past.

What I decided to do (and have working in a test instance) is write a simple python program that uses the text of the command to run to pick a consistent but "random" delay, sleep that amount, then run the code. Then we can just use regular cron.

As a first step, I'd propose building containers and setting up helm charts that allow either jobber or cron and at some point we deprecate the jobber portion. Thoughts, @rdimaio ?

That seems reasonable, but I don't really work on probes, so I don't really have a strong opinion here. I think it would be best to check what the other probe developers think

[dchristidis]

Despite its shortcomings, ATLAS intends to continue using Nagios for the medium-term future. Hence, we are unaffected by any changes to the probes container.