Merge hook UX rework + re-measured demo + subscription-cache wiring

Owner-directed fixes: module/CLI renamed autoaccel->hook (python -m
rclcppyy.hook); ONE env var (RCLCPPYY_ENABLE_HOOK=1 on, unset/0 off;
DISABLE flag deleted); README table headers rclpy/rclcppyy; dev-history
paragraph deleted; "Honest boundary" retitled factually; redundant section
cut. rmem_max verdict: the sysctl was necessary and sufficient — 3 MB
BEST_EFFORT now works at CycloneDDS defaults. Showcase re-measured big+fast:
delivered rates match wherever the publisher can feed; CPU differs 2.6-3.1x
(rclpy crosses a full core at 200-250 Hz @3 MB vs ~1/3 core). node.py now
uses rclcpp_kit's per-type subscription cache (per-UUID cppdef gone): warm
CLI start 7.0->4.2 s (supervisor-reproduced 4.22 s; stock 1.72 s; residual =
uncached bringup discovery/adapters, identified as the next lever).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: awesomebytes

README.md

@@ -238,116 +238,101 @@ the stock `ros2 topic hz` (and other rclpy-based verbs) runs on the C++ backend
 **zero code changes**:
 
 ```bash
-python -m rclcppyy.autoaccel install     # once per environment (uninstall/status too)
+python -m rclcppyy.hook install          # once per environment (uninstall/status too)
 
-RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic  # accelerated: rclcpp executor under the hood
+RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic  # runs on the rclcpp backend
 ros2 topic hz /some_topic                   # env var unset -> ordinary rclpy, untouched
 ```
 
-Two environment variables gate it, side by side:
-
-| Variable | Effect |
-|---|---|
-| `RCLCPPYY_ENABLE_HOOK=1` | turn the hook on for this process (unset = off) |
-| `RCLCPPYY_DISABLE_HOOK=1` | force it off even if `RCLCPPYY_ENABLE_HOOK=1` — the opt-out wins |
-
-Mechanically, `install` writes a site-packages `.pth` (modelled on cppyy_kit's
-auto-PCH bootstrap: guarded, never-raising, silent and near-zero-cost when the env
-var is unset, `RCLCPPYY_DISABLE_HOOK=1` opt-out, fully uninstallable). At interpreter
-start the `.pth` registers a post-import hook on `rclpy`; the first `import rclpy`
-then triggers `enable_cpp_acceleration()` — after rclpy is importable but before the
-tool builds its node.
-
-Making `ros2 topic hz` work took one addition to the accelerator: it drives its own
-loop with `rclpy.spin_once` (both in ros2cli's `DirectNode` discovery loop and the
-hz loop), which the stock rclpy executor cannot use to service rclcpp-backed
-entities — so `enable_cpp_acceleration()` now also patches `rclpy.spin_once` to
-drive an rclcpp executor. Without that the CLI hangs at node bring-up.
-
-**What acceleration buys `ros2 topic hz`, measured** (stock `ros2 topic hz` vs the
-same binary under `RCLCPPYY_ENABLE_HOOK=1`, against a C++ publisher of a 256×256 `Image`
-≈ 0.19 MB, RELIABLE, `--window 100`; `pixi run -e heavydemo demo-topic-hz-cli`;
-absolute numbers vary by machine/run):
-
-| target Hz | publisher Hz | reported Hz (both) | stock CPU % | RCLCPPYY_ENABLE_HOOK CPU % | CPU ratio |
+`RCLCPPYY_ENABLE_HOOK` is the single control: `RCLCPPYY_ENABLE_HOOK=1` turns the
+hook on for a process; unset, `0`, or any other value leaves it off. `install`
+writes a `.pth` into the environment's site-packages that runs at every interpreter
+start; when the hook is off it is a near-zero-cost no-op, and `python -m
+rclcppyy.hook uninstall` removes it.
+
+At interpreter start the `.pth` registers a post-import hook on `rclpy`; the first
+`import rclpy` triggers `enable_cpp_acceleration()`, after rclpy is importable but
+before the tool builds its node. `ros2 topic hz` drives its loop with
+`rclpy.spin_once` (in ros2cli's `DirectNode` discovery loop and in the hz loop), so
+`enable_cpp_acceleration()` also patches `rclpy.spin_once` to run an rclcpp executor;
+otherwise the CLI would stall at node bring-up.
+
+**Measured** (stock `ros2 topic hz` vs the same binary under
+`RCLCPPYY_ENABLE_HOOK=1`, against a C++ publisher of a `sensor_msgs/Image`,
+BEST_EFFORT, `--window 100`, with `net.core.rmem_max` raised (see below);
+`pixi run -e heavydemo demo-topic-hz-cli`; numbers vary by machine and run):
+
+| payload | target Hz | rclpy Hz | rclcppyy Hz | rclpy CPU % | rclcppyy CPU % |
 |---|---|---|---|---|---|
-| 50  | 50.0  | 50.0  | 1.6  | 1.4  | 1.1× |
-| 100 | 100.0 | 100.0 | 3.8  | 2.1  | 1.8× |
-| 200 | 200.0 | 200.0 | 6.3  | 2.7  | 2.3× |
-| 500 | 500.0 | 500.0 | 11.7 | 6.3  | 1.9× |
-| 700 | 700.0 | ~690  | 22.0 | 16.6 | 1.3× |
-
-Both report the true rate; the win is **CPU**: acceleration runs the same hz on
-roughly **half to a third** less CPU at moderate rates. The win is narrower than for
-a message-crunching node, and for an honest reason: `ros2 topic hz` (without
-`--filter`) subscribes with `raw=True`, so **stock rclpy already skips
-deserialisation** — the saving here is the C++ executor and message handling
-replacing the Python executor, not avoiding a per-message copy. At very high rates
-the gap narrows further because the accelerated path deserialises into a C++ proxy
-while the stock raw path deserialises nothing.
-
-**Startup: the acceleration is a one-time cost.** Time-to-first-hz-line
+| 3.0 MB  | 100  | 99.0   | 97.0   | 48.7  | 18.8 |
+| 3.0 MB  | 200  | 195.1  | 197.0  | 92.2  | 34.5 |
+| 3.0 MB  | 300  | 291.4  | 296.7  | 102.7 | 42.6 |
+| 0.05 MB | 1000 | 998.2  | 999.7  | 19.3  | 7.3  |
+| 0.05 MB | 3000 | 2967.6 | 2985.0 | 46.6  | 17.2 |
+
+Both backends deliver essentially the publisher's rate wherever the publisher can
+sustain it; the difference is CPU — the accelerated backend uses about a third to a
+half as much. `ros2 topic hz` (without `--filter`) subscribes with `raw=True`, so
+stock rclpy does not deserialise the payload; the saving is the C++ executor and
+message handling replacing the Python executor, not avoiding a per-message copy.
+Delivered rates stay close because the raw stock reader is cheap enough to keep up
+where the publisher can feed it — but note where the CPU lands: on the 3 MB stream
+stock rclpy crosses one full core near 200–250 Hz (92–107 %), while the accelerated
+backend stays around a third of a core, so on a busier machine or at a higher rate
+the stock reader saturates first.
+
+**Startup: a one-time cost.** Time to first hz line
 (`pixi run -e heavydemo demo-topic-hz-startup`):
 
 | `ros2 topic hz` first hz line | time |
 |---|---|
 | stock (no acceleration) | ~1.7 s |
-| `RCLCPPYY_ENABLE_HOOK=1`, cold PCH cache | ~8.0 s |
-| `RCLCPPYY_ENABLE_HOOK=1`, warm PCH cache | ~7.0 s |
-
-The suite's zero-config Cling PCH shaves ~1.1 s (the `rclcpp` header parse) off the
-first line, cold→warm; the remaining ~5 s over stock is one-time rclcpp bring-up
-plus first-use JIT of the message type and subscription glue, which the header PCH
-does not cover. So acceleration adds a multi-second startup and **pays off for
-long-running `hz` sessions**, not short invocations. (Needs the newer auto-PCH; see
-the dev-bridge note below.)
-
-**On "why not 100 Hz?"** With the real tool the publisher hits its target exactly —
-`50.0 … 700.0` published Hz in the table, confirmed by the publisher's own
-`PUB achieved_hz=` line and by the stock hz reading. An earlier custom subscriber
-that reported ~93 at a 100 Hz target was measuring its own per-second window with a
-shallow RELIABLE queue that dropped a few messages, not a publisher shortfall.
-
-**Honest boundary — heavy (3 MB) topics + BEST_EFFORT.** `ros2 topic hz` fixes its
-reader to `qos_profile_sensor_data` (BEST_EFFORT). A 3 MB image fragments into
-~2000 UDP datagrams, and this machine's `net.core.rmem_max` is 208 KB, so the
-socket buffer overflows and whole messages are lost — the stock hz reader often gets
-**nothing** on a 3 MB topic, and acceleration does not rescue it (same BEST_EFFORT
-reader). This is a transport/kernel limit, not an rclpy-vs-rclcpp one. To measure a
-genuinely heavy topic, raise the buffer first (owner action, needs root):
+| `RCLCPPYY_ENABLE_HOOK=1`, cold cache | ~8.3 s |
+| `RCLCPPYY_ENABLE_HOOK=1`, warm cache | ~4.1 s |
+
+Two caches cut the warm start: the suite's Cling PCH removes the `rclcpp` header
+parse, and a per-message-type subscription cache removes the ~2.8 s one-time JIT of
+`create_subscription<MsgT>` (both built on the first run and loaded on later ones,
+cold → warm above). The warm figure is still above stock because the rest of the
+rclcpp bring-up — cppyy symbol discovery and the rclpy-compatibility adapters — is
+not yet cached. Acceleration therefore remains a one-time startup cost that pays off
+over a long-running `hz` session rather than a single short call. (Needs the newer
+suite; see the dev-bridge note below.)
+
+### Large messages under BEST_EFFORT QoS
+
+`ros2 topic hz` subscribes with `qos_profile_sensor_data` (BEST_EFFORT). A 3 MB
+image fragments into roughly two thousand UDP datagrams, and if the OS socket
+receive buffer is smaller than one message a single dropped fragment loses the whole
+message. On a stock Linux install `net.core.rmem_max` defaults to about 200 KB, so a
+BEST_EFFORT reader receives little or nothing on a 3 MB topic — for stock rclpy and
+for the accelerated backend alike, since both use the same reader QoS. Raising the
+kernel limit resolves it (this needs root, and applies to any DDS user, not only
+rclcppyy):
 
 ```bash
-sudo sysctl -w net.core.rmem_max=2147483647   # then re-run demo-topic-hz-cli --width 1024 --height 1024
+sudo sysctl -w net.core.rmem_max=2147483647
 ```
 
-The tests cover the feature hermetically (`test/test_autoaccel.py`: install /
-uninstall, `RCLCPPYY_ENABLE_HOOK=1` accelerates a fresh `import rclpy`, unset and
-`RCLCPPYY_DISABLE_HOOK=1` leave stock rclpy untouched).
-
-### Controlled comparison: a deserialising subscriber (dev bridge)
+With that in place, CycloneDDS's default socket buffer is large enough that a 3 MB
+BEST_EFFORT topic is received normally; no `CYCLONEDDS_URI` tuning is required. The
+3 MB figures above were measured with this setting.
 
-Where `ros2 topic hz` uses raw subscriptions, a *normal* node that reads message
-fields pays full per-message Python deserialisation on stock rclpy — and that is
-where acceleration wins biggest. `scripts/heavy_hz_demo/run_heavy_hz.py` is a
-controlled harness for that case (a C++ publisher + an hz-style subscriber that
-reads the header, same script both ways via one `enable_cpp_acceleration()` line).
-On a 3 MB image, RELIABLE, plain rclpy tops out ~210 Hz saturating ~1.6 cores while
-the accelerated subscriber climbs past 330 Hz at half the CPU and a third of the
-latency.
-
-```bash
-pixi run -e heavydemo demo-heavy-hz          # rclpy vs rclcppyy deserialising-subscriber table
-pixi run -e heavydemo demo-heavy-startup     # cold vs warm rclcppyy bring-up
-```
+The startup hook is covered by `test/test_hook.py` (install / uninstall / status;
+`RCLCPPYY_ENABLE_HOOK=1` accelerates a fresh `import rclpy`; unset and `=0` leave
+stock rclpy untouched). For the acceleration on a node that *reads* message fields —
+where stock rclpy pays a per-message Python deserialisation that `ros2 topic hz`
+(raw subscription) does not — `scripts/heavy_hz_demo/run_heavy_hz.py` is a controlled
+publisher/subscriber harness (`pixi run -e heavydemo demo-heavy-hz`).
 
-> **Dev bridge.** The cold-vs-warm startup story and the fastest bring-up need the
+> **Dev bridge.** The startup measurement and the fastest bring-up use the
 > zero-config auto-PCH, which postdates the published suite 0.1.0. Until the next
 > suite release the `heavydemo` pixi env bridges the newer `rclcpp_kit`/`cppyy_kit`
-> from a sibling `cppyy_kit` **source checkout** (default `../cppyy_kit`, override
-> `CPPYY_KIT_SRC`) and isolates its PCH cache under `.heavy_demo_cache/`. It is a
-> **dev/demo-only** env: shares the default solve (no extra conda deps) and leaves
-> the default env's published-channel dependency untouched
-> (`workspace_activation.sh`, gated on the `heavydemo` env).
+> from a sibling `cppyy_kit` source checkout (default `../cppyy_kit`, override
+> `CPPYY_KIT_SRC`) and isolates its PCH cache under `.heavy_demo_cache/`. It shares
+> the default solve (no extra conda deps) and leaves the default env's
+> published-channel dependency untouched (`workspace_activation.sh`, gated on the
+> `heavydemo` env).
 
 ### Extra demos (optional env)
 

pixi.toml

@@ -109,9 +109,10 @@ demo-heavy-hz-rclpy = "python scripts/heavy_hz_demo/run_heavy_hz.py --variant rc
 demo-heavy-hz-rclcppyy = "python scripts/heavy_hz_demo/run_heavy_hz.py --variant rclcppyy"
 # Cold (fresh PCH cache) vs warm rclcppyy bringup: parse cost + time-to-first-msg.
 demo-heavy-startup = "python scripts/heavy_hz_demo/run_heavy_hz.py --startup-story"
-# Accelerate the REAL `ros2 topic hz` binary via RCLCPPYY_ENABLE_HOOK: plain vs auto CPU/hz
-# across a rate sweep (installs the opt-in .pth first; idempotent).
-demo-topic-hz-cli = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --sweep"
+# Accelerate the REAL `ros2 topic hz` binary via RCLCPPYY_ENABLE_HOOK: rclpy vs
+# rclcppyy CPU/hz on a 3 MB image (installs the opt-in .pth first; idempotent).
+# Vary payload/rate with flags, e.g. --width 128 --height 128 --rates 1000,3000.
+demo-topic-hz-cli = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --sweep --rates 100,200,300 --width 1024 --height 1024 --reliability best_effort"
 # Time-to-first-hz-line of the real `ros2 topic hz`: stock vs RCLCPPYY_ENABLE_HOOK, cold vs warm.
 demo-topic-hz-startup = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --startup-story"
 

rclcppyy/_autoaccel_boot.py rclcppyy/_hook_boot.pyrclcppyy/_autoaccel_boot.py renamed to rclcppyy/_hook_boot.py

@@ -1,15 +1,15 @@
-"""Standalone bootstrap for rclcppyy's opt-in auto-acceleration (RCLCPPYY_ENABLE_HOOK).
+"""Standalone bootstrap for rclcppyy's startup hook (RCLCPPYY_ENABLE_HOOK).
 
 A copy of this file is installed into the active environment's site-packages (as
-``_rclcppyy_autoaccel.py``) next to a ``.pth`` whose single line calls
-``activate()`` at **every interpreter start, before any user import**. That is what
-lets a process which never mentions rclcppyy -- the stock ``ros2`` CLI, for example
--- pick up the C++ backend: with ``RCLCPPYY_ENABLE_HOOK=1`` set, ``activate()`` arranges
-for ``rclcppyy.enable_cpp_acceleration()`` to run the moment ``rclpy`` is imported,
-so ``rclpy.create_node`` / ``spin`` / ``spin_once`` are already patched before the
-tool builds its node.
-
-The SAME module is imported by ``rclcppyy.autoaccel`` as the single source of the
+``_rclcppyy_hook.py``) next to a ``.pth`` whose single line calls ``activate()`` at
+**every interpreter start, before any user import**. That is what lets a process
+which never mentions rclcppyy -- the stock ``ros2`` CLI, for example -- pick up the
+C++ backend: with ``RCLCPPYY_ENABLE_HOOK=1`` set, ``activate()`` arranges for
+``rclcppyy.enable_cpp_acceleration()`` to run the moment ``rclpy`` is imported, so
+``rclpy.create_node`` / ``spin`` / ``spin_once`` are already patched before the tool
+builds its node.
+
+The SAME module is imported by ``rclcppyy.hook`` as the single source of the
 install-path and activation logic. Two hard constraints follow (mirroring
 cppyy_kit's auto-PCH bootstrap):
 
@@ -19,25 +19,23 @@
   * **never raises** -- a bootstrap that threw would print a traceback on every
     ``python`` invocation. ``activate()`` swallows everything.
 
-When ``RCLCPPYY_ENABLE_HOOK`` is unset (the default) ``activate()`` returns immediately,
-before importing anything, so the cost for unrelated processes is negligible.
-``RCLCPPYY_DISABLE_HOOK=1`` is an explicit opt-out that wins even if ``RCLCPPYY_ENABLE_HOOK=1``.
+The single control is ``RCLCPPYY_ENABLE_HOOK``: exactly ``"1"`` turns the hook on;
+unset, ``"0"``, or any other value leaves it off, so ``activate()`` returns
+immediately (before importing anything) and the cost for unrelated processes is
+negligible.
 """
 import os
 import sys
 
 ENABLE_ENV = "RCLCPPYY_ENABLE_HOOK"
-DISABLE_ENV = "RCLCPPYY_DISABLE_HOOK"
 
 # Per-process guard. A module global (not an env var) so it never leaks into child
 # processes -- each interpreter decides for itself whether to accelerate.
 _activated = False
 
 
 def _enabled():
-    """Whether auto-acceleration is requested for this process."""
-    if os.environ.get(DISABLE_ENV) == "1":
-        return False
+    """Whether the hook is requested for this process (only ``"1"`` enables)."""
     return os.environ.get(ENABLE_ENV) == "1"
 
 

rclcppyy/autoaccel.py rclcppyy/hook.pyrclcppyy/autoaccel.py renamed to rclcppyy/hook.py

@@ -4,37 +4,37 @@
 accelerate a process you cannot edit -- notably the stock ``ros2`` CLI verbs
 (``ros2 topic hz`` and friends) -- rclcppyy can install a startup ``.pth`` into the
 environment's site-packages. The ``.pth`` runs at every interpreter start and calls
-``_autoaccel_boot.activate()``, which does nothing unless ``RCLCPPYY_ENABLE_HOOK=1`` is
+``_hook_boot.activate()``, which does nothing unless ``RCLCPPYY_ENABLE_HOOK=1`` is
 set; when it is, it arranges for ``enable_cpp_acceleration()`` to run the moment
 ``rclpy`` is imported. So the workflow is:
 
-    python -m rclcppyy.autoaccel install        # once per environment
+    python -m rclcppyy.hook install             # once per environment
     RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic    # accelerated, zero code changes
     ros2 topic hz /some_topic                     # unchanged: env var unset -> stock
 
-    python -m rclcppyy.autoaccel status
-    python -m rclcppyy.autoaccel uninstall
+    python -m rclcppyy.hook status
+    python -m rclcppyy.hook uninstall
 
-This mirrors cppyy_kit's auto-PCH bootstrap: the ``.pth`` is opt-IN (gated on the
-env var), silent and near-zero-cost when unset, honours the ``RCLCPPYY_DISABLE_HOOK=1``
-opt-out, and is fully uninstallable.
+The single control is ``RCLCPPYY_ENABLE_HOOK``: exactly ``"1"`` turns the hook on
+for a process; unset, ``"0"``, or any other value leaves it off. The ``.pth`` itself
+is silent and near-zero-cost when the hook is off, and is fully uninstallable.
 """
 import argparse
 import os
 import sys
 import sysconfig
 
-from rclcppyy import _autoaccel_boot as _boot
+from rclcppyy import _hook_boot as _boot
 
-# The installed boot copy is a top-level module (``_rclcppyy_autoaccel``) so it needs
-# no package on sys.path at interpreter start. The .pth's one line imports it and
-# calls activate(), fully guarded so a broken/absent copy can never crash or spam a
+# The installed boot copy is a top-level module (``_rclcppyy_hook``) so it needs no
+# package on sys.path at interpreter start. The .pth's one line imports it and calls
+# activate(), fully guarded so a broken/absent copy can never crash or spam a
 # traceback into an interpreter start.
-_PTH_NAME = "rclcppyy_autoaccel.pth"
-_BOOT_INSTALLED_NAME = "_rclcppyy_autoaccel.py"
+_PTH_NAME = "rclcppyy_hook.pth"
+_BOOT_INSTALLED_NAME = "_rclcppyy_hook.py"
 _PTH_LINE = (
     'import sys; exec('
-    '"try:\\n import _rclcppyy_autoaccel as _m; _m.activate()\\n'
+    '"try:\\n import _rclcppyy_hook as _m; _m.activate()\\n'
     'except Exception: pass")\n'
 )
 
@@ -111,27 +111,26 @@ def status(site_dir=None):
         "site_dir": site,
         "installed": is_installed(site),
         "RCLCPPYY_ENABLE_HOOK": os.environ.get(_boot.ENABLE_ENV),
-        "RCLCPPYY_DISABLE_HOOK": os.environ.get(_boot.DISABLE_ENV),
     }
 
 
 def main(argv=None):
     parser = argparse.ArgumentParser(
-        prog="python -m rclcppyy.autoaccel",
+        prog="python -m rclcppyy.hook",
         description="Install the opt-in RCLCPPYY_ENABLE_HOOK startup hook so stock ros2 CLI "
                     "tools accelerate when RCLCPPYY_ENABLE_HOOK=1 is set.")
     sub = parser.add_subparsers(dest="cmd", required=True)
     sub.add_parser("install", help="install the .pth + boot module into site-packages")
     sub.add_parser("uninstall", help="remove the .pth + boot module")
-    sub.add_parser("status", help="show whether the hook is installed and the env vars")
+    sub.add_parser("status", help="show whether the hook is installed and the env var")
     args = parser.parse_args(argv)
 
     if args.cmd == "install":
         paths = install()
-        print("Installed rclcppyy auto-acceleration hook:")
+        print("Installed rclcppyy startup hook:")
         print("  %s" % paths["pth"])
         print("  %s" % paths["boot"])
-        print("Enable per process with RCLCPPYY_ENABLE_HOOK=1 (opt out with RCLCPPYY_DISABLE_HOOK=1).")
+        print("Enable per process with RCLCPPYY_ENABLE_HOOK=1.")
     elif args.cmd == "uninstall":
         removed = uninstall()
         if removed:
@@ -142,10 +141,9 @@ def main(argv=None):
             print("Nothing to remove (hook not installed).")
     elif args.cmd == "status":
         st = status()
-        print("site-packages : %s" % st["site_dir"])
-        print("installed     : %s" % st["installed"])
+        print("site-packages       : %s" % st["site_dir"])
+        print("installed           : %s" % st["installed"])
         print("RCLCPPYY_ENABLE_HOOK : %s" % (st["RCLCPPYY_ENABLE_HOOK"] or "<unset>"))
-        print("RCLCPPYY_DISABLE_HOOK: %s" % (st["RCLCPPYY_DISABLE_HOOK"] or "<unset>"))
     return 0