[FEATURE] - Monitoring added for scheduler

[uded]

I am quite happy with the Monitor [#317] implementation for jobs, really simple to implement Prometheus metrics.

But...

I still need a method to monitor the scheduler. It would be nice to get information about the number of jobs and queued jobs in the scheduler. Currently, such an implementation would require a rather nasty hook to the metrics endpoint or something similar.

IMHO, the best solution would be to have some event listeners or SchedulerMonitor.

[JohnRoesler]

hey @uded

What sort of monitoring would you like to see from the scheduler? I agree, we could have another simple interface for SchedulerMonitor

[uded]

Honestly, I've wrapped up my implementation, added some magic to monitor it, and forgot about it. I would have to resurrect that code to remember where I had issues. Been a while since I last worked on it, like 4 months or so...

[iyashjayesh]

@JohnRoesler sharing here what we discussed. I'll pick this up.

I think we can add the following metrics at scheduler level, I’ve broken them down below:

• Lifecycle: Scheduler (Started/Stopped/Shutdown)
• Job Management: Job (Registered/Unregistered/Active(maybe)/Runnning(maybe))
• Performance: CurrentJobQueue, SchedulingLag or ConcurrencyLimitCheck
• Stats: ErrorRate, AverageExecutionTime (optional - not very confident whether it will be useful or not)

These would provide a high-level view of scheduler health and resource utilization.

We’ll start with these for now and can add more metrics later if needed.

[iyashjayesh]

@JohnRoesler

I've started working on the scheduler monitoring feature and have implemented the first part - scheduler lifecycle monitoring (SchedulerStarted/SchedulerStopped).

I've raised a draft PR with the initial implementation #889 It would be great if you could review it when you get a chance, just to make sure we're aligned before I continue with the remaining methods.

Thanks!