Skip to content

43 - reorganic-documentation #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tuanhungngyn wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

43 - reorganic-documentation #47

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bdcb194
changed README
tuanhungngyn Jan 9, 2026
9b5f2d3
changed installation guide
tuanhungngyn Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 41 additions & 138 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,173 +21,76 @@ This agent is tested with:
* **Bandwidth constraints**: Filter and compress data before optionally replicating to a central server or the cloud.
* **Manual workflows**: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
* **Lack of filtering**: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
* **Ubuntu Core**: Future Snap integration to support deployment as part of the [Ubuntu Core observability stack](https://ubuntu.com/blog/ubuntu-core-24-robotics-telemetry).

## Structure

The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).

```yaml
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
downsample:
downsampling_mode: stride
stride_n: 5
```
* **Ubuntu Core**: Distributed as a Snap and suitable for deployment on Ubuntu Core–based robotic systems.

## Documentation

📘 **ROS 2 ReductStore Agent Documentation**
👉 https://www.reduct.store/docs/integrations/ros2-agent

📚 **ReductStore Platform Documentation**
👉 https://www.reduct.store/docs

See the [Configuration](#configuration) section for details on available parameters.
The website is the **single source of truth** for:
- YAML configuration options
- Pipeline behavior
- Storage quotas
- Compression & downsampling
- MCAP / CDR formats
- Performance tuning

## Installing
## Quickstart and Installation

This package requires Python dependencies (`reduct-py`, `mcap`, `mcap-ros2-support`) that are **not provided by ROS 2**.
To keep them isolated while still allowing `colcon` and `ros2` (installed system-wide) to work, we create a virtual environment with:
### 1. Start a ReductStore Instance

The agent requires a running ReductStore instance.

For local testing, you can start ReductStore using Docker:

```bash
python3 -m venv .venv --system-site-packages
docker run -p 8383:8383 reductstore/reductstore:latest
```
### 2. Install the ROS 2 Agent

The `--system-site-packages` flag is important because it lets the venv access ROS’s system-installed Python packages (`rclpy`, `rosbag2_py`, etc.).
Without it, `colcon` and ROS Python APIs won’t be visible inside the venv, and `ros2 run` would fail to find your node.
Choose one of the following installation methods:

### Build and Run
#### a) Snap Package (Recommended)

```bash
# 1. Create a workspace and clone the repository
sudo snap install reductstore-agent --edge
```

#### b) Build from Source

```bash
# Create workspace and clone repository
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..

# 2. Install system dependencies
# Install dependencies
rosdep install --from-paths src --ignore-src -r -y

# 3. Create and activate the virtual environment
# Set up virtual environment
python3 -m venv .venv --system-site-packages
source .venv/bin/activate

# 4. Install Python dependencies
# Install Python dependencies
pip install -U reduct-py mcap mcap-ros2-support

# 5. Build the package with colcon using the venv Python
# Build the package
python -m colcon build --symlink-install

# 6. Source the workspace and run the node
source install/setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config/params.yml
```

There is also a [Snap package](#snap-package) available for easier installation and updates.

## Configuration

The configuration file is a YAML file that defines the storage settings and pipelines. The `storage` section contains ReductStore connection details, and the `pipelines` section defines the individual pipelines for recording data.

### Storage Configuration

The `storage` section specifies the ReductStore instance to connect to:

* **`url`**: The URL of the ReductStore instance (e.g., `http://localhost:8383`).
* **`api_token`**: The API token for authentication. This is required to access the ReductStore instance.
* **`bucket`**: The bucket name where the data will be stored.
* **`quota_type`**: The type of quota to apply. Options are:
* `"FIFO"`: First In, First Out (oldest data is removed first).
* `"HARD"`: Hard limit (data is not accepted when the quota is reached).
* `"NONE"`: No quota applied.
* **`quota_size`**: The size of the quota in bytes. This is required if `quota_type` is set to `"FIFO"` or `"HARD"`.
* **`max_block_size`**: Maximum size of each block in bytes.
* **`max_block_records`**: Maximum number of records per block.

More information on how to setup ReductStore can be found in the [ReductStore Getting Started Guide](https://www.reduct.store/docs/getting-started).

### Pipeline Parameters

Each pipeline supports the following parameters:

* **`split`**:

* **`max_duration_s`**: Maximum duration (in seconds) for each data segment. Must be between `1` and `3600`.
* **`max_size_bytes`** *(optional)*: Maximum size (in bytes) for each segment. Must be between `1KB` and `1GB`.

* **`chunk_size_bytes`**: Size of each MCAP chunk in bytes. Defaults to `1MB`. Must be between `1KB` and `10MB`.

* **`compression`**: Compression algorithm to use. One of:

* `"none"`
* `"lz4"`
* `"zstd"` *(default)*

* **`enable_crcs`**: Whether to enable CRC checks. Defaults to `true`.

* **`spool_max_size_bytes`**: Maximum in-memory spool size before flushing. Defaults to `10MB`. Must be between `1KB` and `1GB`.

* **`include_topics`**: List of topics to include for recording. Supports regular expressions.
* **`exclude_topics`** *(optional)*: List of topics to exclude from recording. Supports regular expressions.

* **`static_labels`** *(optional)*: Fixed key-value labels to attach to each record.

* **`filename_mode`**: Determines how filenames are generated. One of:

* `"timestamp"` *(default)* — Use first topic timestamp for filenames.
* `"incremental"` — Use incrementing numbers (0, 1, 2, ...) for filenames.

* **`downsample_mode`**: Method discards messages. Choose one of:

* `"none"` *(default)*
* `"stride"` — (with separate parameter stride_n for the stride value) Only records records every `stride_n` message. Must be >=2.
* `"max_rate"` — (with separate parameter max_rate_hz for the rate limit) Specifies maximum frequency of recorded messages.

* **`output_format`**: Choose one of the output formats:

* `"mcap"` *(default)* — Send mcap files to ReductStore
* `"cdr"` — Send binary CDR to ReductStore

* **`mode`**: Choose one of dynamic labels mode:

* `"last"` *(default)* - use the most recent message
* `"first"` - use the first message in the current file
* `"max"` - use the maximum value across all messages in the file

## Snap Package

This snap provides the **ReductStore Agent ROS 2 node**, which records ROS 2 topics and streams them into ReductStore.

### Build and Install
### 3. Run the Recorder

```bash
# Install snapcraft if not already installed
sudo snap install snapcraft --classic

# Build the snap
snapcraft pack

# Install locally (for testing)
sudo snap install --dangerous reductstore-agent_*.snap
# For both Snap and source installs:
ros2 run reductstore_agent recorder --ros-args --params-file ./config/params.yml
```

Or install directly from the store (edge channel for now):

```bash
sudo snap install reductstore-agent --edge
```
> **Note:** The recorder requires a parameters file to start. Both the Snap and source distributions include a default configuration suitable for local testing.

## Links

Expand Down