feat: add a shared timer scheduler for high-rate timers (Tokio Executor 4/4)

[azerupi]

Note: This is based on #655 , I was hoping I could target that PR as base branch but that seems to only work if the branches are in the ros2_rust repo and not in my fork.

---

This is part of a set of PRs that attempt to add an event-driven Tokio executor to rclrs. The goal is to have something that:

1. Works in a similar way than the events executor in rclcpp
2. Interoperates nicely with the tokio ecosystem

I tried to split the work up into multiple PRs that build on top of each other to make the reviewing easier.

1. feat: add push-callback registration for rcl primitives (Tokio Executor 1/4) #653
2. feat: add an event-driven multi-threaded Tokio executor (Tokio Executor 2/4) #654
3. feat: support actions on the Tokio executor (Tokio Executor 3/4) #655
4. feat: add a shared timer scheduler for high-rate timers (Tokio Executor 4/4) #657

To make the CI work reliably for the Tokio event executor changes we need the following PRs to be merged:

1. fix: action server clock use-after-free #651
2. fix: make the test log handler coexist with rosout #656

Disclaimer
This work was heavily helped by the use of AI. That helped me clear a lot of ground quickly, but I want to be honest about it.

I'm submitting this set of PRs as a draft to get eyes on it and have other people help test, find limitations and flaws and provide feedback on how to improve it in order to reach the quality to be able to merge this.

a shared timer scheduler for high-rate timers

In #654, timers are driven one task each: a tokio::time sleep to the next deadline, then a wait for the worker to run the callback before scheduling the next fire. That capped high-rate timers, a 1 kHz timer fired at roughly 600 Hz, as called out in #654's known limitations. This PR removes that cap.

Why it was capped

Profiling the old driver found three costs per fire, only the first of which is fundamental:

1. Tokio's timer wheel rounds sleeps to about 1 ms on Linux.
2. The driver waited for the worker to finish the callback before computing the next deadline, a full round-trip per tick.
3. A readiness poll slept up to another 1 ms re-checking is_ready.

Costs 2 and 3 are what a per-timer, execution-coupled design pays. Ceiling controls (a bare loop counting ticks, no ROS) confirmed that both a dedicated thread on a Condvar and Tokio's sleep_until hold 100% of target through 20 kHz on this host, so the wait mechanism itself was never the limit, the coupling was.

How it works

The per-timer drivers are replaced by one shared TimerScheduler per executor, modeled on the TimersManager in rclcpp's EventsExecutor. It keeps a min-heap of timers keyed by next deadline. A single dedicated OS thread sleeps until the earliest deadline, enqueues a tick on the owning worker's mailbox, and immediately advances that timer's deadline, without waiting for the callback to run. Decoupling pacing from execution removes costs 2 and 3.

Pacing reuses the same coalescing path as every other entity (see #654): each tick is delivered through the timer's EntityDispatch, so a tick only enqueues a mailbox message if one is not already pending. When the worker is slower than the timer, the late tick is dropped rather than allowed to build a backlog. Deadlines advance by period from the previous deadline, not from "now", so a timer that keeps up does not drift.

Why a dedicated thread and not Tokio

The waiter is a portable std::thread parked on Condvar::wait_timeout, not a Tokio task.

I tried to make this work with Tokio using tokio::select! over sleep_until and a watch/Notify because that would have made the design significantly simpler. However the performance was significantly worse. I measured two Tokio waiters against the condvar thread and got the following results (single timer, % of target achieved):

Target Hz	Condvar thread	Tokio (shared runtime)	Tokio (dedicated runtime)
1 000	100%	~79%	~57%
5 000	100%	~18%	~17%
10 000	100%	~9%	~9%
16 000	~100%	—	~6%

The first Tokio variant runs the waiter on the executor's shared multi-threaded runtime, and my first guess was that it degraded because it contends with the very worker tasks it is pacing. So I built a second variant on its own dedicated single-thread runtime to remove that contention and it was worse, capping at ~900 fires/s regardless of target.

The cost of the Condvar design is that there is a lost-wakeup window between "deciding to sleep" and "actually sleeping" that has to be managed. The waiter does that with the "generation counter" that acts as a version stamp. It snapshots under the state lock and re-checks under the condvar's own lock right before sleeping, so a registration or reset that lands in that gap is not missed.

Lifecycle

Timers notify the scheduler when their state changes, through a small handle stored on the timer. reset() re-seeds the deadline from rcl and wakes the waiter. cancel() drops the timer from the heap but keeps a tombstone entry so a later reset() can re-arm it. Dropping the timer removes the entry entirely, ordered before rcl_timer_fini so the scheduler thread can no longer touch it.

Performance

Measured on a Linux machine, 5 s windows, empty callbacks unless noted. A single timer now tracks its target up to the worker round-trip limit (the old driver delivered ~600 Hz at a 1 kHz target):

Target Hz	Achieved	% of target
1 000	1 000	100
5 000	5 000	100
10 000	10 000	100
16 000	15 977	~100
18 000	16 333	91
20 000	10 023	50

A single timer holds ~100% through ~16 kHz, then cliffs near 20 kHz to about half the target: the scheduler still paces at the requested rate, but the worker sustains only ~10 kHz of round-trips and the overload policy drops every other tick. That is far above the sub-kHz range timers are normally used at; ≤5 kHz is a comfortable single-timer recommendation.

Many timers scale to a high aggregate because the heap is cheap and the cost is total callback CPU on the worker, not scheduling. 64 timers at 10 kHz each delivered ~640k fires/s at 100%. Callbacks that do real work serialize on the worker, so the practical limit is rate × work_us approaching one CPU-second per wall second.

As with the executor itself, I'd encourage people to run their own timer benchmarks so we can find cases this doesn't handle well.

Known limitations

Cross-platform

I've only tested this on Linux.

Simulation time

Like #654, this PR does not properly support simulation time. Sim-time timers run on the scheduler but are paced against the wall clock, so their rate does not track a paused or scaled /clock. Proper support should be a follow-up and probably depends on jump-callback infrastructure (#629).