Add comprehensive metrics instrumentation for scheduler and executor

[phillipleblanc]

Rationale for this change

This PR adds comprehensive metrics instrumentation to track scheduler and executor performance, enabling better observability into task scheduling, shuffle operations, and resource utilization.

What changes are included in this PR?

This PR includes 6 commits that progressively build out the metrics infrastructure:

1. Add shuffle read metrics extraction and QueryStageExecutor::plan() method

• Add public getter methods to PartitionStats (num_rows, num_batches, num_bytes)
• Extend QueryStageExecutor trait with plan() method to access underlying ExecutionPlan
• Add extract_shuffle_read_metrics() to walk plan tree and sum ShuffleReaderExec partition stats
• Record shuffle read metrics (bytes, rows, duration) after successful task execution in executor

2. Add shuffle locality metrics to ExecutorMetricsCollector, SchedulerMetricsCollector, and ShuffleReaderExec

• Add record_shuffle_read_local/remote methods to ExecutorMetricsCollector trait
• Add record_task_shuffle_affinity_hit/miss methods to SchedulerMetricsCollector trait
• Add ShuffleReadMetricsCallback trait in ballista-core for tracking local vs remote reads
• Instrument shuffle_reader.rs to call metrics callback during partition fetches
• Add SessionConfigExt methods to pass metrics callback via session config

3. Add metrics collector to SchedulerState and instrument executor and planning metrics

• Add metrics_collector field to SchedulerState struct
• Instrument record_planning_duration in submit_job
• Instrument record_executor_registered/deregistered and set_active_executor_count
• Update all SchedulerState constructors and call sites

4. Add stage and task lifecycle metrics instrumentation to update_task_status flow

• Instrument stage and task lifecycle metrics in the task status update flow

5. Add shuffle affinity metrics to scheduler task binding

• Track shuffle affinity hits/misses when binding tasks to executors

6. Add actual task scheduling latency tracking

• Add schedulable_time_millis field to TaskDescription to track when a task became schedulable (when its stage transitioned to running state)
• Update all TaskDescription creation sites to pass RunningStage.stage_running_time
• Calculate actual scheduling latency by computing the difference between current time and schedulable_time_millis
• Enables accurate scheduler_task_scheduling_latency_ms metrics instead of the previous placeholder value of 0

Are there any user-facing changes?

Yes, this PR introduces new metrics that can be collected via the ExecutorMetricsCollector and SchedulerMetricsCollector traits:

Executor Metrics:

• Shuffle read bytes, rows, and duration
• Local vs remote shuffle reads

Scheduler Metrics:

• Planning duration
• Executor registration/deregistration events
• Active executor count
• Stage and task lifecycle events
• Shuffle affinity hits/misses
• Task scheduling latency

These metrics are exposed through the existing Prometheus metrics endpoint when using the PrometheusMetricsCollector implementation.