feat: add scheduler lifecycle monitoring <> #785

[iyashjayesh]

Summary

Implements comprehensive scheduler and job lifecycle monitoring as proposed in #785. This PR adds the complete SchedulerMonitor interface with all planned metrics for tracking scheduler health, job execution, and performance.

Changes

• Add SchedulerMonitor interface with 12 event methods
• Integrate monitor callbacks throughout scheduler lifecycle (scheduler.go)
• Add execution time and scheduling delay tracking (executor.go)
• Add concurrency limit notifications for singleton and limit modes
• Comprehensive test coverage for all events
• Update README with detailed documentation and examples

Implemented Metrics

Lifecycle:

• SchedulerStarted - Called when Start() is invoked
• SchedulerStopped - Called when scheduler stops (before cleanup)
• SchedulerShutdown - Called when Shutdown() completes

Job Management:

• JobRegistered - Track jobs added to scheduler
• JobUnregistered - Track jobs removed from scheduler
• JobStarted - Job begins execution
• JobRunning - Job is actively running

Performance:

• JobExecutionTime - Enables AverageExecutionTime calculations
• JobSchedulingDelay - Enables SchedulingLag detection
• ConcurrencyLimitReached - Detects singleton/limit mode constraints

Job Execution:

• JobCompleted - Track successful completions
• JobFailed - Track failures (enables ErrorRate calculation)

Derived Metrics (calculable by monitor implementations):

• Error Rate: JobFailed / (JobCompleted + JobFailed)
• Average Execution Time: from JobExecutionTime events
• Active Jobs: JobRegistered - JobUnregistered
• Current Queue Depth: JobStarted - (JobCompleted + JobFailed)

Use Cases

• Prometheus/Grafana: Export metrics for visualization and alerting
• Custom Dashboards: Build internal monitoring dashboards
• Alerting Systems: Trigger alerts on high error rates or scheduling lag
• Performance Analysis: Identify slow jobs and bottlenecks
• Capacity Planning: Track queue depths and concurrency limits

@JohnRoesler Ready for review! All metrics from the original plan are now implemented and tested.

[iyashjayesh]

@JohnRoesler Apart from the scope we discussed, can you review the PR once?

If needed, we can add any missing metrics. So far, I’ve tried to cover all the ones we discussed.