Rework RCLCPPYY_ENABLE_HOOK UX + demo; re-measure with raised rmem_max

Owner-driven revisions to the merged hook feature:

- Naming coherence: rename the module/CLI from `autoaccel` to `hook` --
  rclcppyy/hook.py + rclcppyy/_hook_boot.py, `python -m rclcppyy.hook
  install|uninstall|status`; the installed files are _rclcppyy_hook.py /
  rclcppyy_hook.pth. test/test_autoaccel.py -> test/test_hook.py.
- One env var: RCLCPPYY_ENABLE_HOOK=1 turns the hook on; unset, 0, or any other
  value leaves it off. Deleted RCLCPPYY_DISABLE_HOOK entirely (code, tests, docs).
- README reworked for a first-time reader: CPU columns are rclpy / rclcppyy;
  removed the development-history paragraph; the boundary section is retitled
  "Large messages under BEST_EFFORT QoS" and rewritten; the deserialising-
  subscriber note is reduced to one sentence pointing at run_heavy_hz.py.

Re-measured with net.core.rmem_max=2147483647 (stock ros2 topic hz vs the same
binary under RCLCPPYY_ENABLE_HOOK=1, C++ publisher, BEST_EFFORT, --window 100):

  payload  target  rclpy Hz  rclcppyy Hz  rclpy CPU%  rclcppyy CPU%
  3.0 MB   100     99.0      97.0         48.7        18.8
  3.0 MB   200     195.1     197.0        92.2        34.5
  3.0 MB   300     291.4     296.7        102.7       42.6
  0.05 MB  1000    998.2     999.7        19.3        7.3
  0.05 MB  3000    2967.6    2985.0       46.6        17.2

Both backends deliver the publisher's rate wherever it can be sustained; the
difference is CPU (~2.6-3.1x). ros2 topic hz uses a raw subscription, so stock
rclpy already skips deserialisation and stays cheap enough to keep up; the saving
is the C++ executor. On the 3 MB stream stock rclpy crosses one full core near
200-250 Hz while the accelerated backend stays ~1/3 core.

rmem truth chain: on stock rmem_max (~208 KB) a 3 MB BEST_EFFORT topic overflows
the socket buffer and the hz reader (stock or accelerated) gets little/nothing --
a transport/kernel limit, not rclpy-vs-rclcpp. `sudo sysctl -w
net.core.rmem_max=2147483647` was necessary and sufficient; CycloneDDS's default
socket buffer then suffices, no CYCLONEDDS_URI tuning needed.

demo-topic-hz-cli now defaults to the 3 MB BEST_EFFORT config shown in the README.
pixi run test: 32 passed. pixi run lint: clean. pixi run build: clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: awesomebytes

README.md

@@ -238,53 +238,48 @@ the stock `ros2 topic hz` (and other rclpy-based verbs) runs on the C++ backend
 **zero code changes**:
 
 ```bash
-python -m rclcppyy.autoaccel install     # once per environment (uninstall/status too)
+python -m rclcppyy.hook install          # once per environment (uninstall/status too)
 
-RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic  # accelerated: rclcpp executor under the hood
+RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic  # runs on the rclcpp backend
 ros2 topic hz /some_topic                   # env var unset -> ordinary rclpy, untouched
 ```
 
-Two environment variables gate it, side by side:
-
-| Variable | Effect |
-|---|---|
-| `RCLCPPYY_ENABLE_HOOK=1` | turn the hook on for this process (unset = off) |
-| `RCLCPPYY_DISABLE_HOOK=1` | force it off even if `RCLCPPYY_ENABLE_HOOK=1` — the opt-out wins |
-
-Mechanically, `install` writes a site-packages `.pth` (modelled on cppyy_kit's
-auto-PCH bootstrap: guarded, never-raising, silent and near-zero-cost when the env
-var is unset, `RCLCPPYY_DISABLE_HOOK=1` opt-out, fully uninstallable). At interpreter
-start the `.pth` registers a post-import hook on `rclpy`; the first `import rclpy`
-then triggers `enable_cpp_acceleration()` — after rclpy is importable but before the
-tool builds its node.
-
-Making `ros2 topic hz` work took one addition to the accelerator: it drives its own
-loop with `rclpy.spin_once` (both in ros2cli's `DirectNode` discovery loop and the
-hz loop), which the stock rclpy executor cannot use to service rclcpp-backed
-entities — so `enable_cpp_acceleration()` now also patches `rclpy.spin_once` to
-drive an rclcpp executor. Without that the CLI hangs at node bring-up.
-
-**What acceleration buys `ros2 topic hz`, measured** (stock `ros2 topic hz` vs the
-same binary under `RCLCPPYY_ENABLE_HOOK=1`, against a C++ publisher of a 256×256 `Image`
-≈ 0.19 MB, RELIABLE, `--window 100`; `pixi run -e heavydemo demo-topic-hz-cli`;
-absolute numbers vary by machine/run):
-
-| target Hz | publisher Hz | reported Hz (both) | stock CPU % | RCLCPPYY_ENABLE_HOOK CPU % | CPU ratio |
+`RCLCPPYY_ENABLE_HOOK` is the single control: `RCLCPPYY_ENABLE_HOOK=1` turns the
+hook on for a process; unset, `0`, or any other value leaves it off. `install`
+writes a `.pth` into the environment's site-packages that runs at every interpreter
+start; when the hook is off it is a near-zero-cost no-op, and `python -m
+rclcppyy.hook uninstall` removes it.
+
+At interpreter start the `.pth` registers a post-import hook on `rclpy`; the first
+`import rclpy` triggers `enable_cpp_acceleration()`, after rclpy is importable but
+before the tool builds its node. `ros2 topic hz` drives its loop with
+`rclpy.spin_once` (in ros2cli's `DirectNode` discovery loop and in the hz loop), so
+`enable_cpp_acceleration()` also patches `rclpy.spin_once` to run an rclcpp executor;
+otherwise the CLI would stall at node bring-up.
+
+**Measured** (stock `ros2 topic hz` vs the same binary under
+`RCLCPPYY_ENABLE_HOOK=1`, against a C++ publisher of a `sensor_msgs/Image`,
+BEST_EFFORT, `--window 100`, with `net.core.rmem_max` raised (see below);
+`pixi run -e heavydemo demo-topic-hz-cli`; numbers vary by machine and run):
+
+| payload | target Hz | rclpy Hz | rclcppyy Hz | rclpy CPU % | rclcppyy CPU % |
 |---|---|---|---|---|---|
-| 50  | 50.0  | 50.0  | 1.6  | 1.4  | 1.1× |
-| 100 | 100.0 | 100.0 | 3.8  | 2.1  | 1.8× |
-| 200 | 200.0 | 200.0 | 6.3  | 2.7  | 2.3× |
-| 500 | 500.0 | 500.0 | 11.7 | 6.3  | 1.9× |
-| 700 | 700.0 | ~690  | 22.0 | 16.6 | 1.3× |
-
-Both report the true rate; the win is **CPU**: acceleration runs the same hz on
-roughly **half to a third** less CPU at moderate rates. The win is narrower than for
-a message-crunching node, and for an honest reason: `ros2 topic hz` (without
-`--filter`) subscribes with `raw=True`, so **stock rclpy already skips
-deserialisation** — the saving here is the C++ executor and message handling
-replacing the Python executor, not avoiding a per-message copy. At very high rates
-the gap narrows further because the accelerated path deserialises into a C++ proxy
-while the stock raw path deserialises nothing.
+| 3.0 MB  | 100  | 99.0   | 97.0   | 48.7  | 18.8 |
+| 3.0 MB  | 200  | 195.1  | 197.0  | 92.2  | 34.5 |
+| 3.0 MB  | 300  | 291.4  | 296.7  | 102.7 | 42.6 |
+| 0.05 MB | 1000 | 998.2  | 999.7  | 19.3  | 7.3  |
+| 0.05 MB | 3000 | 2967.6 | 2985.0 | 46.6  | 17.2 |
+
+Both backends deliver essentially the publisher's rate wherever the publisher can
+sustain it; the difference is CPU — the accelerated backend uses about a third to a
+half as much. `ros2 topic hz` (without `--filter`) subscribes with `raw=True`, so
+stock rclpy does not deserialise the payload; the saving is the C++ executor and
+message handling replacing the Python executor, not avoiding a per-message copy.
+Delivered rates stay close because the raw stock reader is cheap enough to keep up
+where the publisher can feed it — but note where the CPU lands: on the 3 MB stream
+stock rclpy crosses one full core near 200–250 Hz (92–107 %), while the accelerated
+backend stays around a third of a core, so on a busier machine or at a higher rate
+the stock reader saturates first.
 
 **Startup: the acceleration is a one-time cost.** Time-to-first-hz-line
 (`pixi run -e heavydemo demo-topic-hz-startup`):
@@ -302,52 +297,40 @@ does not cover. So acceleration adds a multi-second startup and **pays off for
 long-running `hz` sessions**, not short invocations. (Needs the newer auto-PCH; see
 the dev-bridge note below.)
 
-**On "why not 100 Hz?"** With the real tool the publisher hits its target exactly —
-`50.0 … 700.0` published Hz in the table, confirmed by the publisher's own
-`PUB achieved_hz=` line and by the stock hz reading. An earlier custom subscriber
-that reported ~93 at a 100 Hz target was measuring its own per-second window with a
-shallow RELIABLE queue that dropped a few messages, not a publisher shortfall.
+### Large messages under BEST_EFFORT QoS
 
-**Honest boundary — heavy (3 MB) topics + BEST_EFFORT.** `ros2 topic hz` fixes its
-reader to `qos_profile_sensor_data` (BEST_EFFORT). A 3 MB image fragments into
-~2000 UDP datagrams, and this machine's `net.core.rmem_max` is 208 KB, so the
-socket buffer overflows and whole messages are lost — the stock hz reader often gets
-**nothing** on a 3 MB topic, and acceleration does not rescue it (same BEST_EFFORT
-reader). This is a transport/kernel limit, not an rclpy-vs-rclcpp one. To measure a
-genuinely heavy topic, raise the buffer first (owner action, needs root):
+`ros2 topic hz` subscribes with `qos_profile_sensor_data` (BEST_EFFORT). A 3 MB
+image fragments into roughly two thousand UDP datagrams, and if the OS socket
+receive buffer is smaller than one message a single dropped fragment loses the whole
+message. On a stock Linux install `net.core.rmem_max` defaults to about 200 KB, so a
+BEST_EFFORT reader receives little or nothing on a 3 MB topic — for stock rclpy and
+for the accelerated backend alike, since both use the same reader QoS. Raising the
+kernel limit resolves it (this needs root, and applies to any DDS user, not only
+rclcppyy):
 
 ```bash
-sudo sysctl -w net.core.rmem_max=2147483647   # then re-run demo-topic-hz-cli --width 1024 --height 1024
+sudo sysctl -w net.core.rmem_max=2147483647
 ```
 
-The tests cover the feature hermetically (`test/test_autoaccel.py`: install /
-uninstall, `RCLCPPYY_ENABLE_HOOK=1` accelerates a fresh `import rclpy`, unset and
-`RCLCPPYY_DISABLE_HOOK=1` leave stock rclpy untouched).
-
-### Controlled comparison: a deserialising subscriber (dev bridge)
+With that in place, CycloneDDS's default socket buffer is large enough that a 3 MB
+BEST_EFFORT topic is received normally; no `CYCLONEDDS_URI` tuning is required. The
+3 MB figures above were measured with this setting.
 
-Where `ros2 topic hz` uses raw subscriptions, a *normal* node that reads message
-fields pays full per-message Python deserialisation on stock rclpy — and that is
-where acceleration wins biggest. `scripts/heavy_hz_demo/run_heavy_hz.py` is a
-controlled harness for that case (a C++ publisher + an hz-style subscriber that
-reads the header, same script both ways via one `enable_cpp_acceleration()` line).
-On a 3 MB image, RELIABLE, plain rclpy tops out ~210 Hz saturating ~1.6 cores while
-the accelerated subscriber climbs past 330 Hz at half the CPU and a third of the
-latency.
-
-```bash
-pixi run -e heavydemo demo-heavy-hz          # rclpy vs rclcppyy deserialising-subscriber table
-pixi run -e heavydemo demo-heavy-startup     # cold vs warm rclcppyy bring-up
-```
+The startup hook is covered by `test/test_hook.py` (install / uninstall / status;
+`RCLCPPYY_ENABLE_HOOK=1` accelerates a fresh `import rclpy`; unset and `=0` leave
+stock rclpy untouched). For the acceleration on a node that *reads* message fields —
+where stock rclpy pays a per-message Python deserialisation that `ros2 topic hz`
+(raw subscription) does not — `scripts/heavy_hz_demo/run_heavy_hz.py` is a controlled
+publisher/subscriber harness (`pixi run -e heavydemo demo-heavy-hz`).
 
-> **Dev bridge.** The cold-vs-warm startup story and the fastest bring-up need the
+> **Dev bridge.** The startup measurement and the fastest bring-up use the
 > zero-config auto-PCH, which postdates the published suite 0.1.0. Until the next
 > suite release the `heavydemo` pixi env bridges the newer `rclcpp_kit`/`cppyy_kit`
-> from a sibling `cppyy_kit` **source checkout** (default `../cppyy_kit`, override
-> `CPPYY_KIT_SRC`) and isolates its PCH cache under `.heavy_demo_cache/`. It is a
-> **dev/demo-only** env: shares the default solve (no extra conda deps) and leaves
-> the default env's published-channel dependency untouched
-> (`workspace_activation.sh`, gated on the `heavydemo` env).
+> from a sibling `cppyy_kit` source checkout (default `../cppyy_kit`, override
+> `CPPYY_KIT_SRC`) and isolates its PCH cache under `.heavy_demo_cache/`. It shares
+> the default solve (no extra conda deps) and leaves the default env's
+> published-channel dependency untouched (`workspace_activation.sh`, gated on the
+> `heavydemo` env).
 
 ### Extra demos (optional env)
 

pixi.toml

@@ -109,9 +109,10 @@ demo-heavy-hz-rclpy = "python scripts/heavy_hz_demo/run_heavy_hz.py --variant rc
 demo-heavy-hz-rclcppyy = "python scripts/heavy_hz_demo/run_heavy_hz.py --variant rclcppyy"
 # Cold (fresh PCH cache) vs warm rclcppyy bringup: parse cost + time-to-first-msg.
 demo-heavy-startup = "python scripts/heavy_hz_demo/run_heavy_hz.py --startup-story"
-# Accelerate the REAL `ros2 topic hz` binary via RCLCPPYY_ENABLE_HOOK: plain vs auto CPU/hz
-# across a rate sweep (installs the opt-in .pth first; idempotent).
-demo-topic-hz-cli = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --sweep"
+# Accelerate the REAL `ros2 topic hz` binary via RCLCPPYY_ENABLE_HOOK: rclpy vs
+# rclcppyy CPU/hz on a 3 MB image (installs the opt-in .pth first; idempotent).
+# Vary payload/rate with flags, e.g. --width 128 --height 128 --rates 1000,3000.
+demo-topic-hz-cli = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --sweep --rates 100,200,300 --width 1024 --height 1024 --reliability best_effort"
 # Time-to-first-hz-line of the real `ros2 topic hz`: stock vs RCLCPPYY_ENABLE_HOOK, cold vs warm.
 demo-topic-hz-startup = "python scripts/heavy_hz_demo/run_topic_hz_cli.py --startup-story"
 

rclcppyy/_autoaccel_boot.py rclcppyy/_hook_boot.pyrclcppyy/_autoaccel_boot.py renamed to rclcppyy/_hook_boot.py

@@ -1,15 +1,15 @@
-"""Standalone bootstrap for rclcppyy's opt-in auto-acceleration (RCLCPPYY_ENABLE_HOOK).
+"""Standalone bootstrap for rclcppyy's startup hook (RCLCPPYY_ENABLE_HOOK).
 
 A copy of this file is installed into the active environment's site-packages (as
-``_rclcppyy_autoaccel.py``) next to a ``.pth`` whose single line calls
-``activate()`` at **every interpreter start, before any user import**. That is what
-lets a process which never mentions rclcppyy -- the stock ``ros2`` CLI, for example
--- pick up the C++ backend: with ``RCLCPPYY_ENABLE_HOOK=1`` set, ``activate()`` arranges
-for ``rclcppyy.enable_cpp_acceleration()`` to run the moment ``rclpy`` is imported,
-so ``rclpy.create_node`` / ``spin`` / ``spin_once`` are already patched before the
-tool builds its node.
-
-The SAME module is imported by ``rclcppyy.autoaccel`` as the single source of the
+``_rclcppyy_hook.py``) next to a ``.pth`` whose single line calls ``activate()`` at
+**every interpreter start, before any user import**. That is what lets a process
+which never mentions rclcppyy -- the stock ``ros2`` CLI, for example -- pick up the
+C++ backend: with ``RCLCPPYY_ENABLE_HOOK=1`` set, ``activate()`` arranges for
+``rclcppyy.enable_cpp_acceleration()`` to run the moment ``rclpy`` is imported, so
+``rclpy.create_node`` / ``spin`` / ``spin_once`` are already patched before the tool
+builds its node.
+
+The SAME module is imported by ``rclcppyy.hook`` as the single source of the
 install-path and activation logic. Two hard constraints follow (mirroring
 cppyy_kit's auto-PCH bootstrap):
 
@@ -19,25 +19,23 @@
   * **never raises** -- a bootstrap that threw would print a traceback on every
     ``python`` invocation. ``activate()`` swallows everything.
 
-When ``RCLCPPYY_ENABLE_HOOK`` is unset (the default) ``activate()`` returns immediately,
-before importing anything, so the cost for unrelated processes is negligible.
-``RCLCPPYY_DISABLE_HOOK=1`` is an explicit opt-out that wins even if ``RCLCPPYY_ENABLE_HOOK=1``.
+The single control is ``RCLCPPYY_ENABLE_HOOK``: exactly ``"1"`` turns the hook on;
+unset, ``"0"``, or any other value leaves it off, so ``activate()`` returns
+immediately (before importing anything) and the cost for unrelated processes is
+negligible.
 """
 import os
 import sys
 
 ENABLE_ENV = "RCLCPPYY_ENABLE_HOOK"
-DISABLE_ENV = "RCLCPPYY_DISABLE_HOOK"
 
 # Per-process guard. A module global (not an env var) so it never leaks into child
 # processes -- each interpreter decides for itself whether to accelerate.
 _activated = False
 
 
 def _enabled():
-    """Whether auto-acceleration is requested for this process."""
-    if os.environ.get(DISABLE_ENV) == "1":
-        return False
+    """Whether the hook is requested for this process (only ``"1"`` enables)."""
     return os.environ.get(ENABLE_ENV) == "1"
 
 

rclcppyy/autoaccel.py rclcppyy/hook.pyrclcppyy/autoaccel.py renamed to rclcppyy/hook.py

@@ -4,37 +4,37 @@
 accelerate a process you cannot edit -- notably the stock ``ros2`` CLI verbs
 (``ros2 topic hz`` and friends) -- rclcppyy can install a startup ``.pth`` into the
 environment's site-packages. The ``.pth`` runs at every interpreter start and calls
-``_autoaccel_boot.activate()``, which does nothing unless ``RCLCPPYY_ENABLE_HOOK=1`` is
+``_hook_boot.activate()``, which does nothing unless ``RCLCPPYY_ENABLE_HOOK=1`` is
 set; when it is, it arranges for ``enable_cpp_acceleration()`` to run the moment
 ``rclpy`` is imported. So the workflow is:
 
-    python -m rclcppyy.autoaccel install        # once per environment
+    python -m rclcppyy.hook install             # once per environment
     RCLCPPYY_ENABLE_HOOK=1 ros2 topic hz /some_topic    # accelerated, zero code changes
     ros2 topic hz /some_topic                     # unchanged: env var unset -> stock
 
-    python -m rclcppyy.autoaccel status
-    python -m rclcppyy.autoaccel uninstall
+    python -m rclcppyy.hook status
+    python -m rclcppyy.hook uninstall
 
-This mirrors cppyy_kit's auto-PCH bootstrap: the ``.pth`` is opt-IN (gated on the
-env var), silent and near-zero-cost when unset, honours the ``RCLCPPYY_DISABLE_HOOK=1``
-opt-out, and is fully uninstallable.
+The single control is ``RCLCPPYY_ENABLE_HOOK``: exactly ``"1"`` turns the hook on
+for a process; unset, ``"0"``, or any other value leaves it off. The ``.pth`` itself
+is silent and near-zero-cost when the hook is off, and is fully uninstallable.
 """
 import argparse
 import os
 import sys
 import sysconfig
 
-from rclcppyy import _autoaccel_boot as _boot
+from rclcppyy import _hook_boot as _boot
 
-# The installed boot copy is a top-level module (``_rclcppyy_autoaccel``) so it needs
-# no package on sys.path at interpreter start. The .pth's one line imports it and
-# calls activate(), fully guarded so a broken/absent copy can never crash or spam a
+# The installed boot copy is a top-level module (``_rclcppyy_hook``) so it needs no
+# package on sys.path at interpreter start. The .pth's one line imports it and calls
+# activate(), fully guarded so a broken/absent copy can never crash or spam a
 # traceback into an interpreter start.
-_PTH_NAME = "rclcppyy_autoaccel.pth"
-_BOOT_INSTALLED_NAME = "_rclcppyy_autoaccel.py"
+_PTH_NAME = "rclcppyy_hook.pth"
+_BOOT_INSTALLED_NAME = "_rclcppyy_hook.py"
 _PTH_LINE = (
     'import sys; exec('
-    '"try:\\n import _rclcppyy_autoaccel as _m; _m.activate()\\n'
+    '"try:\\n import _rclcppyy_hook as _m; _m.activate()\\n'
     'except Exception: pass")\n'
 )
 
@@ -111,27 +111,26 @@ def status(site_dir=None):
         "site_dir": site,
         "installed": is_installed(site),
         "RCLCPPYY_ENABLE_HOOK": os.environ.get(_boot.ENABLE_ENV),
-        "RCLCPPYY_DISABLE_HOOK": os.environ.get(_boot.DISABLE_ENV),
     }
 
 
 def main(argv=None):
     parser = argparse.ArgumentParser(
-        prog="python -m rclcppyy.autoaccel",
+        prog="python -m rclcppyy.hook",
         description="Install the opt-in RCLCPPYY_ENABLE_HOOK startup hook so stock ros2 CLI "
                     "tools accelerate when RCLCPPYY_ENABLE_HOOK=1 is set.")
     sub = parser.add_subparsers(dest="cmd", required=True)
     sub.add_parser("install", help="install the .pth + boot module into site-packages")
     sub.add_parser("uninstall", help="remove the .pth + boot module")
-    sub.add_parser("status", help="show whether the hook is installed and the env vars")
+    sub.add_parser("status", help="show whether the hook is installed and the env var")
     args = parser.parse_args(argv)
 
     if args.cmd == "install":
         paths = install()
-        print("Installed rclcppyy auto-acceleration hook:")
+        print("Installed rclcppyy startup hook:")
         print("  %s" % paths["pth"])
         print("  %s" % paths["boot"])
-        print("Enable per process with RCLCPPYY_ENABLE_HOOK=1 (opt out with RCLCPPYY_DISABLE_HOOK=1).")
+        print("Enable per process with RCLCPPYY_ENABLE_HOOK=1.")
     elif args.cmd == "uninstall":
         removed = uninstall()
         if removed:
@@ -142,10 +141,9 @@ def main(argv=None):
             print("Nothing to remove (hook not installed).")
     elif args.cmd == "status":
         st = status()
-        print("site-packages : %s" % st["site_dir"])
-        print("installed     : %s" % st["installed"])
+        print("site-packages       : %s" % st["site_dir"])
+        print("installed           : %s" % st["installed"])
         print("RCLCPPYY_ENABLE_HOOK : %s" % (st["RCLCPPYY_ENABLE_HOOK"] or "<unset>"))
-        print("RCLCPPYY_DISABLE_HOOK: %s" % (st["RCLCPPYY_DISABLE_HOOK"] or "<unset>"))
     return 0