Add rosbag_blackbox ROS2 package with ring buffer and snapshot server

Co-authored-by: marc-hanheide <1153084+marc-hanheide@users.noreply.github.com>

Author: Copilot

.gitignore

@@ -4,6 +4,9 @@ build/
 bin/
 lib/
 install/
+*.egg-info/
+__pycache__/
+*.py[cod]
 msg_gen/
 srv_gen/
 msg/*Action.msg

src/rosbag_blackbox/README.md

@@ -0,0 +1,370 @@
+# rosbag_blackbox
+
+A ROS 2 **Humble** package that provides:
+
+1. **Ring Buffer Node** – continuously records configured topics into an
+   in-memory ring buffer and saves a *snapshot* (rosbag file) on demand.
+2. **Snapshot Server** – a combined **FastAPI** REST + **FastMCP** (Model
+   Context Protocol) server that lets REST clients and AI agents access and
+   introspect snapshot files.
+
+---
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration Reference](#configuration-reference)
+  - [Pattern Syntax](#pattern-syntax)
+  - [QoS Settings](#qos-settings)
+  - [Latched Topics](#latched-topics)
+- [Triggering a Snapshot](#triggering-a-snapshot)
+- [Snapshot Server](#snapshot-server)
+  - [REST Endpoints](#rest-endpoints)
+  - [MCP Tools](#mcp-tools)
+- [Launch File](#launch-file)
+- [Node Parameters](#node-parameters)
+- [Architecture](#architecture)
+
+---
+
+## Features
+
+| Feature | Details |
+|---------|---------|
+| **Glob topic matching** | Zenoh-style `*` (one segment) and `**` (any depth) |
+| **Per-topic QoS** | `reliability`, `durability`, `depth` fully configurable |
+| **Latched topics** | Set `durability: transient_local` per topic |
+| **Max frequency** | Discard messages arriving faster than configured Hz |
+| **Fixed-duration buffer** | Always keeps the last *N* seconds (default 10 s) |
+| **Memory efficiency** | Messages stored as raw serialised bytes; deque-based O(1) expiry |
+| **Dynamic type discovery** | Topic types resolved at runtime (like `ros2 bag record`) |
+| **On-demand snapshot** | `std_srvs/srv/Trigger` service writes the buffer to a rosbag |
+| **Configurable filename** | `{datetime}`, `{date}`, `{time}`, `{timestamp}` placeholders |
+| **REST API** | FastAPI server with OpenAPI docs at `/docs` |
+| **AI agent tools** | FastMCP server at `/mcp` (streamable-HTTP) |
+| **Image support** | `sensor_msgs/Image` and `CompressedImage` → PNG |
+
+---
+
+## Installation
+
+### ROS 2 dependencies
+
+```bash
+cd ~/ros2_ws
+rosdep install --from-paths src --ignore-src -r -y
+colcon build --packages-select rosbag_blackbox
+source install/setup.bash
+```
+
+### Optional Python dependencies (for the snapshot server)
+
+```bash
+pip install fastapi uvicorn fastmcp numpy Pillow
+# or
+pip install -r src/rosbag_blackbox/requirements.txt
+```
+
+The **ring buffer node** works without these dependencies.  The snapshot
+server prints a warning and exits if FastAPI / uvicorn are missing.
+
+---
+
+## Quick Start
+
+### 1 – Launch both nodes
+
+```bash
+ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
+    config_file:=/path/to/config.yaml
+```
+
+### 2 – Trigger a snapshot
+
+```bash
+ros2 service call /rosbag_blackbox/save_snapshot std_srvs/srv/Trigger
+```
+
+The response message contains the full path of the new rosbag directory.
+
+### 3 – Query the snapshot via REST
+
+```bash
+# List all snapshots
+curl http://localhost:8001/snapshots
+
+# Get metadata of a specific snapshot
+curl http://localhost:8001/snapshots/snapshot_20240101_120000/metadata
+
+# Get the 5 latest messages on a topic
+curl "http://localhost:8001/snapshots/snapshot_20240101_120000/messages?topic=/chatter&limit=5"
+```
+
+### 4 – OpenAPI docs
+
+Open `http://localhost:8001/docs` in your browser.
+
+---
+
+## Configuration Reference
+
+Copy `config/example_config.yaml` and adapt it:
+
+```yaml
+buffer:
+  duration_seconds: 10      # keep the last N seconds
+
+snapshot:
+  directory: /tmp/rosbag_blackbox_snapshots
+  name_pattern: "snapshot_{datetime}"   # see placeholders below
+
+topics:
+  - pattern: "/camera/**"
+    max_frequency: 10.0     # Hz; omit or 0 for unlimited
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 1
+
+  - pattern: "/tf_static"
+    qos:
+      reliability: reliable
+      durability: transient_local   # latched
+      depth: 1
+
+  - pattern: "/**"          # catch-all fallback
+    max_frequency: 5.0
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 10
+```
+
+**Filename placeholders**
+
+| Placeholder | Example |
+|-------------|---------|
+| `{datetime}` | `20240101_120000` |
+| `{date}` | `20240101` |
+| `{time}` | `120000` |
+| `{timestamp}` | `1704110400` (Unix epoch) |
+
+### Pattern Syntax
+
+The topic pattern follows **Zenoh selector** conventions:
+
+| Pattern | Matches |
+|---------|---------|
+| `/camera/*` | `/camera/left`, `/camera/right` |
+| `/camera/**` | `/camera/left`, `/camera/left/image_raw`, … |
+| `/**` | every topic |
+| `/*/scan` | `/lidar/scan`, `/front/scan`, … |
+| `/a/**/c` | `/a/c`, `/a/b/c`, `/a/b/d/c`, … |
+
+Rules are evaluated **in order**; the first matching pattern wins.
+
+### QoS Settings
+
+| Key | Values | Default |
+|-----|--------|---------|
+| `reliability` | `best_effort` \| `reliable` | `best_effort` |
+| `durability` | `volatile` \| `transient_local` | `volatile` |
+| `depth` | integer | `10` |
+
+### Latched Topics
+
+A **latched** publisher in ROS 2 uses `transient_local` durability.  To
+receive the cached message when your node (re-)joins, set:
+
+```yaml
+qos:
+  reliability: reliable
+  durability: transient_local
+  depth: 1
+```
+
+> **Note:** If the publisher uses `volatile` durability and the subscriber
+> requests `transient_local`, ROS 2 will refuse the connection.  Always match
+> the publisher's durability setting.
+
+---
+
+## Triggering a Snapshot
+
+Call the service exposed by the ring-buffer node:
+
+```bash
+ros2 service call /rosbag_blackbox/save_snapshot std_srvs/srv/Trigger
+```
+
+Or from Python:
+
+```python
+import rclpy
+from rclpy.node import Node
+from std_srvs.srv import Trigger
+
+rclpy.init()
+node = Node('snapshot_client')
+cli = node.create_client(Trigger, '/rosbag_blackbox/save_snapshot')
+cli.wait_for_service()
+future = cli.call_async(Trigger.Request())
+rclpy.spin_until_future_complete(node, future)
+print(future.result().message)
+```
+
+---
+
+## Snapshot Server
+
+Start manually (without the launch file):
+
+```bash
+ros2 run rosbag_blackbox snapshot_server \
+    --ros-args -p snapshot_dir:=/tmp/rosbag_blackbox_snapshots \
+               -p port:=8001
+```
+
+### REST Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/health` | Liveness check |
+| GET | `/snapshots` | List all snapshots |
+| GET | `/snapshots/latest` | Path of the latest snapshot |
+| GET | `/snapshots/{name}/metadata` | Topics, counts, duration |
+| GET | `/snapshots/{name}/messages` | Extract messages (filterable) |
+| GET | `/snapshots/{name}/image/{topic}` | Latest image as PNG |
+
+**Query parameters for `/messages`**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `topic` | string | Filter to a single topic |
+| `start` | float | Unix-time lower bound (seconds) |
+| `end` | float | Unix-time upper bound (seconds) |
+| `limit` | int | Max messages to return (default 100) |
+| `keys_only` | bool | Return only field names |
+| `truncate_arrays` | int | Truncate arrays to N elements |
+
+### MCP Tools
+
+The FastMCP server at `/mcp` exposes the following tools to AI agents:
+
+| Tool | Description |
+|------|-------------|
+| `list_snapshots(directory)` | Discover snapshot files |
+| `select_snapshot(bag_path)` | Set the active snapshot |
+| `get_snapshot_metadata(bag_path)` | Topics, counts, duration |
+| `list_topics(bag_path)` | All topics with message counts |
+| `search_topics(pattern, bag_path)` | Find topics by substring |
+| `get_messages(topic, ...)` | Extract & deserialise messages |
+| `get_image(topic, ...)` | Return image as base64-PNG |
+
+All tools default to the **latest** snapshot when `bag_path` is omitted.
+
+**Example agent interaction:**
+
+```
+agent → list_snapshots()
+      ← [{"path": "/tmp/.../snapshot_20240101_120000", ...}]
+
+agent → get_snapshot_metadata()
+      ← {"topics": [...], "total_messages": 1200, "duration_seconds": 10.0}
+
+agent → list_topics()
+      ← [{"name": "/camera/image_raw", "type": "sensor_msgs/msg/Image", ...}]
+
+agent → get_image(topic="/camera/image_raw")
+      ← {"image_base64": "iVBORw0KGgoAAAANS...", "image_format": "PNG"}
+
+agent → get_messages(topic="/tf", max_messages=3, truncate_arrays=5)
+      ← {"messages": [...], "count": 3}
+```
+
+---
+
+## Launch File
+
+```bash
+ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
+    config_file:=/path/to/config.yaml \
+    snapshot_dir:=/data/snapshots \
+    buffer_duration:=30.0 \
+    server_port:=8001
+```
+
+**Available arguments**
+
+| Argument | Default | Description |
+|----------|---------|-------------|
+| `config_file` | `share/…/example_config.yaml` | Path to YAML config |
+| `snapshot_dir` | `/tmp/rosbag_blackbox_snapshots` | Snapshot output dir |
+| `buffer_duration` | `10.0` | Ring buffer duration (seconds) |
+| `server_host` | `0.0.0.0` | Bind host for the API server |
+| `server_port` | `8001` | Port for the API server |
+
+---
+
+## Node Parameters
+
+### `ringbuffer_node`
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `config_file` | string | `''` | Path to YAML config file |
+| `buffer_duration` | double | `10.0` | Buffer duration in seconds |
+| `snapshot_dir` | string | `/tmp/…` | Snapshot output directory |
+| `snapshot_name_pattern` | string | `snapshot_{datetime}` | Filename pattern |
+
+### `snapshot_server`
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `snapshot_dir` | string | `/tmp/…` | Directory to search for bags |
+| `host` | string | `0.0.0.0` | Bind address |
+| `port` | int | `8001` | HTTP port |
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    ROS 2 Humble                              │
+│                                                             │
+│  Publishers                                                 │
+│     │ topic A ──→ ┌──────────────────────────────────┐      │
+│     │ topic B ──→ │  rosbag_blackbox node             │      │
+│     │ topic C ──→ │                                   │      │
+│     │             │  ┌────────────────────────────┐   │      │
+│     │             │  │  RingBuffer                │   │      │
+│     │             │  │  deque[(ts, topic, bytes)] │   │      │
+│     │             │  │  purge on every add()      │   │      │
+│     │             │  └────────────┬───────────────┘   │      │
+│     │             │               │ snapshot()         │      │
+│     │             │  ┌────────────▼──────────────┐    │      │
+│     │             │  │  save_snapshot service    │    │      │
+│     │             │  │  (std_srvs/Trigger)       │    │      │
+│     │             │  └────────────┬──────────────┘    │      │
+│     │             └───────────────┼──────────────────┘      │
+│     │                             │ writes                   │
+│     │                             ▼                          │
+│     │                    ┌────────────────┐                  │
+│     │                    │  rosbag file   │                  │
+│     │                    │  (SQLite3)     │                  │
+│     │                    └────────┬───────┘                  │
+│                                  │ reads                     │
+│  ┌───────────────────────────────▼───────────────────────┐   │
+│  │  snapshot_server node                                  │   │
+│  │                                                        │   │
+│  │  FastAPI  ──→  REST endpoints (/snapshots, /image, …) │   │
+│  │  FastMCP  ──→  /mcp  (streamable-HTTP)                │   │
+│  └────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────┘
+           ▲                         ▲
+           │ HTTP / REST             │ MCP
+     curl / browser           AI agent (Claude, etc.)
+```