Add PollNow interrupt for Ballista executors to reduce task scheduling latency

[phillipleblanc]

This feature allows the scheduler to send a PollNow message to connected executors via the bidirectional control stream, which will interrupt the 100ms idle sleep in the Ballista poll_loop and cause immediate polling for new work.

Currently, executors poll the scheduler every 100ms when idle. This means there can be up to 100ms latency between when work becomes available and when an executor picks it up. By sending a PollNow signal, we can reduce this latency to near-zero for interactive workloads.

Architecture

┌─────────────────┐                    ┌─────────────────┐
│    Scheduler    │                    │    Executor     │
│                 │                    │                 │
│  ┌───────────┐  │   ControlStream    │  ┌───────────┐  │
│  │ ClusterSvc│──┼────────────────────┼──│CtrlStream │  │
│  │   Impl    │  │   PollNow msg      │  │  Client   │  │
│  └─────┬─────┘  │   ─────────────►   │  └─────┬─────┘  │
│        │        │                    │        │        │
│        │        │                    │        ▼        │
│        │        │                    │  ┌───────────┐  │
│        │        │                    │  │  Notify   │  │
│        │        │                    │  │(per-sched)│  │
│        │        │                    │  └─────┬─────┘  │
│        │        │                    │        │        │
│        │        │                    │        ▼        │
│  ┌───────────┐  │                    │  ┌───────────┐  │
│  │  poll_work│◄─┼────────────────────┼──│ poll_loop │  │
│  └───────────┘  │   gRPC request     │  └───────────┘  │
└─────────────────┘                    └─────────────────┘

• Depends on Add poll_now_notify parameter to poll_loop and on_work_available callback datafusion-ballista#12
• Add PollNowCommand proto message and update SchedulerControlMessage
• Add ExecutorStreamRegistry to track connected executor control streams
• Implement broadcast_poll_now() to send PollNow to all executors
• Update ControlStreamManager to provide shared Notify handle for poll wake-up
• Wire poll_now_notify through spawn_scheduler_poll_loop and update_scheduler_pollers
• Add executor_stream_registry field to DataFusion struct for scheduler access
• Update ClusterServiceImpl to use shared ExecutorStreamRegistry
• Update Cargo.toml to reference ballista fork commit e0ab27eeef9f

[github-actions]

✅ Pull with Spice Passed

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ No banned labels detected
• ✅ Has a label from required category kind/
• ✅ Has a label from required category area/
• ✅ Has at least one assignee: phillipleblanc

[Copilot]

Pull request overview

This pull request implements a "PollNow interrupt" feature for Ballista executors to reduce task scheduling latency from up to 100ms (the idle poll interval) to near-zero for interactive workloads. The scheduler can now broadcast a PollNow command to all connected executors via bidirectional control streams, causing them to immediately poll for new work rather than waiting for the next poll interval.

Changes:

• Added PollNow protobuf message for scheduler-to-executor control stream communication
• Implemented ExecutorStreamRegistry to track connected executor control streams and broadcast PollNow commands
• Wired poll_now_notify through executor control stream client, poll loops, and scheduler initialization
• Updated Ballista fork dependency to e0ab27eeef9f which includes poll_loop modifications to accept Notify handle

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
plans/INTERRUPT_POLL.md	Comprehensive design document explaining architecture, implementation details, and testing strategy
crates/runtime-proto/proto/spice.proto	Added PollNowCommand message and updated SchedulerControlMessage with oneof field
crates/runtime/src/cluster/service.rs	Implemented ExecutorStreamRegistry with broadcast_poll_now, register/unregister methods; updated ClusterServiceImpl to use registry
crates/runtime/src/cluster/control_stream_client.rs	Updated to handle PollNow messages and provide shared Notify handle for poll loop wake-up
crates/runtime/src/cluster/mod.rs	Wired poll_now_notify through spawn_scheduler_poll_loop and update_scheduler_pollers; created on_work_available callback
crates/runtime/src/cluster/servers.rs	Updated to use shared ExecutorStreamRegistry when available for scheduler mode
crates/runtime/src/datafusion/mod.rs	Added executor_stream_registry field, bind_executor_stream_registry and getter methods
crates/runtime/src/datafusion/builder.rs	Initialized executor_stream_registry field in DataFusion builder
Cargo.toml	Updated ballista-* dependencies to new fork commit e0ab27eeef9f with PollNow interrupt support
Cargo.lock	Updated ballista dependencies and transitive dependency versions

[Copilot]

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.