feat(tools/sim): measure raw-mode collection volume against the simulated robot

doc/src/dc/raw_topics.md's "How much data is this?" figures came from a
hand-written topic emulation. tools/sim/scripts/measure_raw_volume.sh boots
dc_simulation's warehouse world headless, drives the Waffle, runs dc_bridge in
raw mode against its live topics in three configurations and reports per-Tag
Records, bytes, bytes/Record, publish rate and a projected GB/day. Local and
informational — not wired into ci.yaml.

The world runs at ~4% of real time without a GPU, and raw.max_rate_hz limits
Records per *wall* second, so measuring the shipped defaults there measures the
box. The profiles therefore collect uncapped, the window is counted in simulated
seconds sampled off /clock, and the report multiplies exact bytes-per-Record by
min(published_hz, max_rate_hz) — the rate a robot running at real time would
ship at.

check_raw()'s volume accounting moves to tools/e2e/scripts/raw_volume.py so the
verifier and the benchmark report the same thing; ten fixture tests pin it and a
standalone raw-volume CI job runs them.

Closes #326

Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RsF8rCLWfP8u7CvhUcJ3LK

Author: Minipada

.github/workflows/ci.yaml

@@ -16,6 +16,7 @@
 #   build-workspace (build + colcon test + coverage) ─┬─▶ sim               (pull dc-workspace, simulation smoke check)
 #                                                      └─▶ build-e2e-image ─▶ e2e   (pull dc-e2e, zero-loss harness)
 #   kpi-views (standalone: SQL + a throwaway Postgres, no workspace image needed)
+#   raw-volume (standalone: stdlib Python fixtures over the shared volume accounting)
 #
 # e2e calls the same tools/e2e/scripts/run.sh a developer runs locally, driving the
 # harness with plain podman (no compose) so nothing extra is installed on the runner;
@@ -312,6 +313,23 @@ jobs:
           done
           exit 1
 
+  # tools/e2e/scripts/raw_volume.py is stdlib Python shared by the zero-loss verifier and
+  # the sim-backed volume benchmark (#326) — no workspace image, no simulator, no database.
+  # The benchmark itself is deliberately not a CI job: it boots gz-sim for a number nobody
+  # asserts a threshold on.
+  raw-volume:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/0.12.5/install.sh | sh
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Fixture tests for the raw-volume accounting
+        run: uv run --frozen pytest tools/e2e/test -q
+
   # The KPI definitions are SQL, not workspace code: this needs a database and nothing else.
   kpi-views:
     runs-on: ubuntu-latest

.gitignore

@@ -280,6 +280,7 @@ cython_debug/
 # scratch state, not source.
 .ci-ws/
 tools/e2e/.run/
+tools/sim/.run/
 
 # dc_bridge local core test build dir (fast-iteration cmake, not colcon)
 dc_bridge/test/local/build/

dc_simulation/README.md

@@ -33,6 +33,18 @@ Without it gz-sensors stamps messages with the scoped sensor name, which is not
 frame, and every tf2 `MessageFilter` downstream — AMCL, both Nav2 costmaps — silently
 drops the message.
 
+## What the robot's topics cost to collect
+
+```bash
+./tools/sim/scripts/measure_raw_volume.sh     # per-Tag bytes, rates and a GB/day
+```
+
+Boots this world headless, drives the robot, runs `dc_bridge` in raw mode against its
+topics and reports what each one is worth in stored bytes — the source of the figures in
+[Raw topic collection](../doc/src/dc/raw_topics.md#how-much-data-is-this). Local and
+informational, never a CI gate; see the script's header for the profiles and why its rates
+are counted in simulated seconds.
+
 ## Running headless, without a GPU
 
 The rendering sensors (`gpu_lidar` and both `rgbd_camera`s) **do** work with no GPU and

doc/src/dc/raw_topics.md

@@ -147,9 +147,9 @@ The values above are the defaults, and they matter:
   everything twice, under two different Tags.
 - **High-rate sensor types are excluded**, by type rather than by name (a camera topic is
   not reliably called anything in particular). One 640×480 `sensor_msgs/msg/Image` is
-  roughly 900 kB of JSON numbers; at 30 Hz nothing downstream is sized for it. Override
-  the list deliberately if you want them — DC will not stop you, but read the next
-  section first.
+  900 kB on the wire and roughly 3.7 MB once every pixel byte is a JSON number; at 30 Hz
+  nothing downstream is sized for it. Override the list deliberately if you want them —
+  DC will not stop you, but read the next section first.
 
 To replace a list, write the replacement; note that an *empty* YAML list (`[]`) cannot be
 loaded by rclcpp (it has no inferable element type), so use a pattern that matches
@@ -198,38 +198,68 @@ acknowledged (see [Destinations](./destinations.md#delivery-guarantees)).
 
 ### How much data is this?
 
-Measured against a TurtleBot3-Waffle-shaped workload — the topic mix
-`dc_simulation`'s warehouse world bridges out, at the rates those sensors actually run
-(IMU 100 Hz, odom/tf/joint_states 50 Hz, camera 30 Hz, cmd_vel 20 Hz, lidar 10 Hz,
-battery 1 Hz) — collected into a `file` Destination:
+These figures come off a **simulated robot**, not a spreadsheet:
+`tools/sim/scripts/measure_raw_volume.sh` boots `dc_simulation`'s warehouse world, drives
+the TurtleBot3-Waffle in a slow circle, points a Bridge in raw mode at its live topics,
+and reports what a `file` Destination stored, per Tag. Everything below is one run of it —
+re-run it after anything that changes what a Record costs. It is a local tool, not a CI
+gate; its header explains the three configurations and why its rates are counted in
+*simulated* seconds.
 
-| Configuration | Stored |
+Rates are that world's Waffle's: IMU 200 Hz, odometry and TF 30 Hz (the `DiffDrive`
+plugin), lidar and both RGBD cameras 5 Hz. `/cmd_vel` and `/clock` are left out — the
+first is the benchmark's own driving, the second exists only because the robot is
+simulated.
+
+| Configuration | Shipped |
 |---|---|
-| **Defaults** (10 Hz cap, sensor types excluded) | 19 kB/s — **67 MB/hour, 1.6 GB/day** |
-| …plus `scan` and `tf` re-enabled | ≈ 4.3 GB/day |
-| …plus the 640×480 camera re-enabled | ≈ **1.6 TB/day** |
+| **Defaults** (10 Hz cap, sensor types excluded) | 24 kB/s — **86 MB/hour, 2.1 GB/day** |
+| …plus `scan` and `tf` re-enabled | 69 kB/s — **5.9 GB/day** |
+| …plus both 1280×720 cameras re-enabled | ≈ **9.5 TB/day**, if anything could carry it |
 
 Per-Record cost, which is what to multiply by your own topics' rates:
 
-| Message | Bytes per Record |
-|---|---|
-| `geometry_msgs/msg/Twist` | 202 |
-| `sensor_msgs/msg/JointState` (2 joints) | 326 |
-| `sensor_msgs/msg/Imu` | 488 |
-| `nav_msgs/msg/Odometry` | 605 |
-| `tf2_msgs/msg/TFMessage` (2 transforms) | 547 |
-| `sensor_msgs/msg/LaserScan` (360 ranges + intensities) | 2 585 |
-| `sensor_msgs/msg/Image` (640×480 `rgb8`) | **1 843 521** |
-
-The last row is the whole argument for the default type exclusions: one VGA frame costs
-as much as ~3 000 odometry Records, and JSON roughly **doubles** a byte array (every
-pixel byte becomes `"0,"`).
-
-**The size cap will not save you from a camera** — and it should not be asked to. That
-same 640×480 frame is 921 600 bytes on the wire, *under* the default
-`max_message_size_bytes` of 1 MiB, so it passes the size gate and lands as 1.8 MB of
-JSON. It is `exclude_types` that keeps images out, and removing that list removes the
-protection entirely.
+| Topic (message) | Publishes at | Bytes per Record |
+|---|---|---|
+| `/joint_states` (`JointState`, 2 joints) | 1 000 Hz † | 355 |
+| `/tf` (`TFMessage`, 1 transform) | 30 Hz | 406 |
+| `…/camera_info` (`CameraInfo`) | 5 Hz | 581 |
+| `/odom` (`Odometry`) | 30 Hz | 669 |
+| `/imu` (`Imu`) | 200 Hz | 767 |
+| `/scan` (`LaserScan`, 360 ranges + intensities) | 5 Hz | 7 878 |
+| `…/image_raw` (`Image`, 1280×720 `rgb8`) | 5 Hz | **10 981 032** |
+
+† the simulator's `JointStatePublisher` runs every physics step; a real driver is far
+slower. It makes no difference to the total, which is the point of the next paragraph.
+
+**The rate cap is what makes the first row affordable, not the topic list.** A topic
+publishing faster than `max_rate_hz` contributes `bytes_per_record × 10` per second no
+matter how fast it actually runs, so `joint_states` at 1 000 Hz and `imu` at 200 Hz cost
+3.6 kB/s and 7.7 kB/s respectively. Only the topics *below* the cap — the 5 Hz sensors —
+bill at their real rate, which is why re-enabling one 5 Hz lidar (7.9 kB per Record, 40
+kB/s, 3.5 GB/day) nearly triples the total on its own.
+
+The last row is the whole argument for the default type exclusions: one 1280×720 frame
+costs as much as ~16 000 odometry Records. JSON does not merely double a byte array —
+`10 981 032` bytes for a 2 764 800-byte frame is **four times** the wire size, because
+most pixel values print as three digits and a comma.
+
+Two cameras at 5 Hz is 110 MB/s of Records, and **nothing in the pipeline carries that**,
+which is why that row says "if anything could carry it": in the run it comes from, the
+Shipper refused 475 of the 894 Records offered to it (`dropped … 475 shipper`, 53 %). The
+collapse was not confined to the images either — `imu` arrived at 4 Hz instead of 200 and
+`odom` at 0.7 Hz instead of 30, because the Bridge spent the window serializing frames.
+Collecting a camera raw does not cost you a camera's worth of storage; it costs you the
+rest of your collection.
+
+**The size cap will not save you from a camera** — and it should not be asked to. A
+640×480 `rgb8` frame is 921 600 bytes on the wire, *under* the default
+`max_message_size_bytes` of 1 MiB, so it passes the size gate and lands as megabytes of
+JSON. (This world's 1280×720 frames are three times that and the default cap does drop
+them whole — which is why the benchmark's camera profile has to lift it to measure
+anything at all. Relying on that is a collection policy that silently switches on the day
+someone fits a smaller sensor.) It is `exclude_types` that keeps images out, and removing
+that list removes the protection entirely.
 
 The tempting fix — lower the size cap until frames stop fitting — is a bad trade, because
 **the two limits fail differently**:

progress.txt

@@ -7243,3 +7243,62 @@ eyeballed. `prek run --all-files --skip build-doc` is green.
 and `unknown` otherwise — and routed to the `pgsql` Destination, so the utilisation panels
 populate on the existing demo. The intervention and fault panels stay empty until #362/#365,
 for the same reason a battery panel would.
+
+## #326 - Sim-backed benchmark for raw-mode collection volume
+
+`raw_topics.md`'s "How much data is this?" numbers came from a hand-written topic emulation:
+plausible rates typed into a generator, not a robot. Now they come off the simulated Waffle.
+`tools/sim/scripts/measure_raw_volume.sh` boots `dc_simulation`'s warehouse world headless,
+drives the robot in a slow circle, runs `dc_bridge` in raw mode against its live topics in
+three configurations (`tools/sim/params/raw_{defaults,sensors,camera}.yaml`) and reports
+per-Tag Records, bytes, bytes/Record, publish rate and a projected GB/day. Local and
+informational, in the spirit of `measure_resources.sh` — deliberately not in `ci.yaml`, which
+would pay minutes of gz-sim boot per PR for a number nobody asserts a threshold on.
+
+**The measurement's one real problem: the simulator does not run at real time.** This world
+sits at RTF ~0.04 on 8 CPUs with software rendering, so every wall-clock rate it produces is
+the renderer's number, not the robot's — and worse, `raw.max_rate_hz` limits Records per
+*wall* second, so at that factor the 10 Hz default fires against topics publishing at 250 Hz
+of simulated time (the first trial: 14 859 forwarded, 3 799 dropped rate). Measuring "the
+defaults" that way measures the box. The resolution splits the two halves of the number:
+the profiles collect with the limiter **off**, so every published message becomes a Record;
+the window is counted in *simulated* seconds sampled off `/clock` either side of it; and the
+report multiplies exact bytes-per-Record by `min(published_hz, 10)` — the rate raw mode
+would ship at on a robot where the limiter binds on real time. Bytes per Record are exact
+regardless of how slowly the world runs, and that is what the projection rests on.
+
+Reused rather than reimplemented, per the issue: `check_raw()`'s volume accounting moved to
+`tools/e2e/scripts/raw_volume.py` (`Volume`, `summarize_by_tag()`, `project()`), which the
+E2E verifier now calls for the same `stored_bytes`/`bytes_per_record`/`projected_mb_per_day`
+keys it published before (verified identical against a fixture). One wrinkle the sim exposed:
+with `time_key: "date"`, a Record's sink timestamp is the *message's* header stamp — sim time
+under a simulator, and the Bridge's wall clock for headerless messages, so spans across the
+two are meaningless. Hence `arrival_rates=False` for the sim reporter, which takes its window
+from `/clock` instead. Ten fixture tests (`tools/e2e/test/test_raw_volume.py`) pin the
+accounting; a standalone `raw-volume` CI job runs them (stdlib only, no image, no simulator).
+
+**What the run says.** Defaults: 24 kB/s, 2.1 GB/day. Plus `scan` and `tf`: 5.9 GB/day — the
+lidar alone is 3.5 GB/day at 7 878 bytes a Record and 5 Hz, because it publishes *below* the
+cap and so bills at its real rate, while `imu` at 200 Hz and `joint_states` at 1 000 Hz are
+decimated to 10. Per-Record: JointState 355, TFMessage 406, CameraInfo 581, Odometry 669,
+Imu 767, LaserScan (360 ranges + intensities) 7 878, and a 1280×720 `rgb8` Image **10 981 032**
+— four times its wire size, not the doubling the old text claimed, because most pixel bytes
+print as three digits and a comma.
+
+The camera profile's finding is the one worth having: it is not that images are expensive, it
+is that **the pipeline collapses**. Two cameras at 5 Hz is 110 MB/s of Records; the Shipper
+refused 475 of the 894 offered (53 %), and the rest of the collection went down with them —
+`imu` arrived at 4 Hz instead of 200, `odom` at 0.7 instead of 30, because the Bridge spent
+the window serializing frames. The old doc's "≈1.6 TB/day" reads as a storage bill; the
+measured behaviour is that you lose the robot's other telemetry first. That is now in the
+doc, along with the fact that this world's 2.7 MB frames are *over* the 1 MiB
+`max_message_size_bytes` default (the camera profile has to lift it to measure anything),
+which makes the existing "the size cap will not save you from a camera" argument concrete
+rather than hypothetical.
+
+Verified by running it: three profiles, each a fresh container from the workspace image, with
+the Bridge's own drop counters extracted per run as the fidelity check (`dropped 0 rate /
+0 oversize / 0 shipper / 0 undecodable` for `sensors`; the camera profile's 53 % is reported
+as a warning and folded into the doc rather than hidden). `prek run --all-files --skip
+build-doc` is green, `mdbook build` clean, and `tools/sim/.run/` — which `run.sh` has been
+writing to since #279 without ever being ignored — is now in `.gitignore`.

tools/e2e/scripts/raw_volume.py

@@ -0,0 +1,143 @@
+# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
+# SPDX-License-Identifier: MPL-2.0
+
+"""Stored-volume accounting for raw-mode (#227) Records.
+
+Both users of it — verify_zero_loss.py's `check_raw()` (one Tag, the E2E harness) and
+tools/sim/scripts/measure_raw_volume.sh's reporter (every Tag, the simulated robot) —
+have to answer the same question the same way, or the doc's numbers and the harness's
+numbers stop being comparable. What a Record *costs* is the bytes the Destination
+actually wrote: the JSON line plus the newline the sink terminates it with.
+"""
+
+import datetime
+import json
+
+
+class Volume:
+    """Bytes, Records and arrival span over one raw Destination's NDJSON output."""
+
+    def __init__(self) -> None:
+        self.records = 0
+        self.stored_bytes = 0
+        self.first_seen: datetime.datetime | None = None
+        self.last_seen: datetime.datetime | None = None
+
+    def add_line(self, line: str) -> None:
+        """Count one stored line, newline included — what the sink wrote to disk."""
+        self.records += 1
+        self.stored_bytes += len(line) + 1
+
+    def add_time(self, stored_at) -> None:
+        """Extend the arrival span with a Record's sink timestamp, if it parses."""
+        if not isinstance(stored_at, str):
+            return
+        try:
+            when = datetime.datetime.fromisoformat(stored_at.replace("Z", "+00:00"))
+        except ValueError:
+            return
+        if self.first_seen is None or when < self.first_seen:
+            self.first_seen = when
+        if self.last_seen is None or when > self.last_seen:
+            self.last_seen = when
+
+    @property
+    def span_seconds(self) -> float | None:
+        """Wall-clock seconds the Records arrived over, or None if unmeasurable.
+
+        Under a second of arrivals is not a rate — dividing by it turns sampling noise
+        into a throughput claim.
+        """
+        if self.first_seen is None or self.last_seen is None:
+            return None
+        span = (self.last_seen - self.first_seen).total_seconds()
+        return span if span >= 1.0 else None
+
+    def summary(self) -> dict:
+        """Volume as the report keys both callers publish."""
+        out = {
+            "stored_bytes": self.stored_bytes,
+            "bytes_per_record": self.stored_bytes // self.records if self.records else None,
+        }
+        span = self.span_seconds
+        if span:
+            out["span_seconds"] = round(span, 1)
+            out["records_per_second"] = round(self.records / span, 2)
+            out["bytes_per_second"] = round(self.stored_bytes / span, 1)
+            out["projected_mb_per_day"] = round(self.stored_bytes / span * 86400 / 1e6, 1)
+        return out
+
+
+def project(summary: dict, sim_seconds: float, max_rate_hz: float) -> dict:
+    """What one Tag's measured volume is worth on a robot running at real time (#326).
+
+    A simulator that runs at a fraction of real time makes wall-clock throughput
+    meaningless — every rate it produces is scaled by the real-time factor. Two things
+    survive that scaling: the bytes a Record costs, and the rate the robot publishes at,
+    which is measured in *simulated* seconds. Multiply them, after applying the rate
+    limiter that would bind on a real robot, and the projection is the robot's, not the
+    simulator's.
+
+    Args:
+        summary: one Tag's entry from `summarize_by_tag()`.
+        sim_seconds: simulated seconds the measurement window covered.
+        max_rate_hz: the profile's `raw.max_rate_hz` (0 = no limiter).
+    """
+    per_record = summary.get("bytes_per_record")
+    if not per_record or sim_seconds <= 0:
+        return {}
+    published_hz = summary["records"] / sim_seconds
+    shipped_hz = min(published_hz, max_rate_hz) if max_rate_hz else published_hz
+    bytes_per_second = per_record * shipped_hz
+    return {
+        "published_hz": round(published_hz, 2),
+        "shipped_hz": round(shipped_hz, 2),
+        "projected_bytes_per_second": round(bytes_per_second, 1),
+        "projected_gb_per_day": round(bytes_per_second * 86400 / 1e9, 3),
+    }
+
+
+def summarize_by_tag(path: str, arrival_rates: bool = True) -> dict:
+    """Per-Tag and total volume over a raw Destination's NDJSON file.
+
+    A line that isn't JSON still cost its bytes, so it counts toward the total under the
+    `(unparsable)` Tag rather than vanishing from the accounting.
+
+    Args:
+        path: the Destination's newline-delimited JSON.
+        arrival_rates: keep the timestamp-derived rates. Off for a simulated robot: the
+            sink timestamp is the Record's `date`, which for a message carrying a
+            `std_msgs/msg/Header` is the *message's* stamp — simulated time under a
+            simulator, and mixed with the Bridge's wall clock for messages without a
+            header. Spans across the two are meaningless; rates there come from the
+            simulated window instead (`project()`).
+    """
+    total = Volume()
+    per_tag: dict[str, Volume] = {}
+
+    with open(path) as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                event = json.loads(line)
+                tag = event.get("tag") or "(untagged)"
+                stored_at = event.get("timestamp")
+            except json.JSONDecodeError:
+                tag, stored_at = "(unparsable)", None
+            total.add_line(line)
+            if arrival_rates:
+                total.add_time(stored_at)
+            volume = per_tag.setdefault(tag, Volume())
+            volume.add_line(line)
+            if arrival_rates:
+                volume.add_time(stored_at)
+
+    return {
+        "tags": {
+            tag: {"records": v.records, **v.summary()}
+            for tag, v in sorted(per_tag.items(), key=lambda kv: -kv[1].stored_bytes)
+        },
+        "total": {"records": total.records, **total.summary()},
+    }