Skip to content

Commit 496eea2

Browse files
takasehidekitakasehideki
committed
docs: update README to enhance clarity and structure of project overview and usage instructions
1 parent 7404915 commit 496eea2

1 file changed

Lines changed: 173 additions & 54 deletions

File tree

‎README.md‎

Lines changed: 173 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,103 @@
1-
# Scalability Evaluation Method for ROS 2 Distributed Communication
1+
# ros2-perf-multihost
22

3-
For the nested workspace that contains the ROS 2 node implementations used in this repository, see [ros2_node_impl_ws/README.md](./ros2_node_impl_ws/README.md).
3+
**Automated Coordination Framework for Objective Architecture Evaluation in Distributed Systems**
44

5-
## Generating Execution Scripts with a Shared Docker Image
5+
The "RMW Cambrian Explosion" in ROS 2 ecosystem following Zenoh’s integration presents developers with complex middleware choices and architectural challenges.
6+
Selecting the optimal RMW and system configuration requires empirical data from actual physical hardware.
67

7-
Instead of generating and building a Dockerfile for each topology, this repository reuses a single shared Docker image and generates only topology-specific execution scripts and Compose definitions.
8+
**ros2-perf-multihost** is an open-source framework for objectively evaluating the performance and architecture of ROS 2 systems in distributed environments on physical devices.
9+
It coordinates evaluation pipelines across multiple physical devices, and enables developers to quantify how node placement and network configurations impact overall stability.
10+
Our purpose is to provide a "scientific scale" for optimizing distributed system design across edge devices and servers with real-world networks, empowering data-driven decisions for large-scale robotic systems.
811

9-
### Generate Execution Scripts
12+
## Overview
1013

11-
Generate execution scripts (`host*_exec.sh`, `host*_run.sh`) and Docker Compose files from a JSON topology file.
14+
### Key Features 🚀
1215

13-
```bash
14-
cd manager_scripts
15-
python3 generate_exec_scripts.py ../topology_example/simple.json --rmw fastdds --ws-dir performance_ws
16-
```
16+
- **Manager-Host Coordination**: Deploy nodes in bulk to multiple target Hosts (Raspberry Pi, Jetson, servers, etc.) via REST API and remotely manage their lifecycle from a central Manager.
17+
- **Flexible Topology Configuration**: Define node relationships and QoS settings to the host assignments declaratively via JSON. Iterate complex topologies for multiple RMWs efficiently.
18+
- **RMW Neutrality**: Evaluate multiple RMW implementations (FastDDS, CycloneDDS, Zenoh) while using QoS and topology definitions for cross-RMW comparisons.
19+
- **Dual Execution Modes**: Support both Docker containerized and native ROS 2 environments for seamless evaluation across development as production-like setups.
20+
- **Precision Telemetry & Monitoring**: Record CPU and memory load on each host with trial-aligned timestamps, enabling time-correlated analysis with end-to-end communication metrics.
1721

18-
### Options Supported by Generated Scripts
22+
### 🏗 Architecture
1923

20-
Generated `host*_run.sh` and `local_run.sh` scripts support the runtime options below. `--eval-time` is applied to every launched node (Publisher / Subscriber / Intermediate). `payload_size` and `period_ms` must be specified in each Publisher / Intermediate topic entry in the topology JSON, and those values are passed directly to Publisher / Intermediate nodes. `--trial-idx` is available only on `host*_run.sh` and `local_run.sh`. For the JSON schema, see [topology_example/README.md](./topology_example/README.md).
24+
This framework employs a two-tier architecture:
2125

22-
| Option | Short | Description | Default |
23-
|---|---|---|---|
24-
| --eval-time | -t | Evaluation time in seconds | 60 |
25-
| --trial-idx | -r | Trial index for local execution | 1 |
26+
- **Manager**: Generates topology-specific scripts, coordinates execution across hosts via REST API, collects logs, and aggregates results.
27+
- **Hosts**: Operate a lightweight REST server to receive execution commands and launch ROS 2 nodes in either Docker containers or native environments.
28+
29+
The workflow proceeds as follows:
30+
31+
1. **Topology Definition**: Users define node placement, topic relationships, and QoS configuration in a topology JSON file.
32+
2. **Coordination**: The Manager generates execution scripts for the selected RMW and distributes them to each Host for execution.
33+
3. **Execution**: All hosts begin operation tests simultaneously while collecting system metrics in the background.
34+
4. **Data Aggregation**: After experiment completion, the Manager collates logs from all hosts and outputs analysis-ready CSV files.
35+
36+
### Observable Metrics 📊
37+
38+
The default pipeline correlates communication performance with host-level resource utilization:
39+
40+
| Category | Metrics | Per |
41+
| :-- | :-- | :-- |
42+
| **Communication** | End-to-end latency and message loss count | Per-trial |
43+
| **Throughput** | Aggregated throughput estimated from publish period, publisher count, payload size, and observed loss | Per-trial |
44+
| **Host Resource Usage** | CPU and memory usage, load average, and swap usage summary | Per-host / Per-trial |
2645

27-
#### Examples
46+
## Getting Started
47+
48+
### Prerequisites
49+
50+
Prepare the repository root on the manager host first.
2851

2952
```bash
30-
# Use default values
31-
./host1_exec.sh
53+
git clone https://github.com/hal-lab-u-tokyo/ros2-perf-multihost.git
54+
cd ros2-perf-multihost
55+
```
3256

33-
# Override eval-time
34-
./host1_run.sh --eval-time 60
57+
Before running the workflows below, make sure the following prerequisites are satisfied:
3558

36-
# Short options
37-
./host1_run.sh -t 60
59+
- Docker is available on hosts that will run the containerized workflow.
60+
- A ROS 2 environment is available on hosts that will run the native workflow.
61+
- The repository is placed at the same path on each host for multi-host execution.
62+
- SSH connectivity is available from the manager host to each target host.
63+
- For the REST workflow, install `python3-flask` and `python3-requests` on each target host.
64+
65+
```bash
66+
sudo apt update
67+
sudo apt install -y python3-flask python3-requests
3868
```
3969

40-
`--eval-time` is applied to all nodes launched through `*_run.sh` or `local_run.sh`. `payload_size` and `period_ms` are read from each Publisher/Intermediate entry in the topology JSON.
70+
### Quick Start: Local Verification
4171

42-
### Pull the Shared Docker Image
72+
Instead of generating and building a Dockerfile for each topology, this repository reuses a single shared Docker image and generates only topology-specific execution scripts and Compose definitions.
73+
74+
1. Pull the shared image.
4375

4476
```bash
4577
docker pull ghcr.io/hal-lab-u-tokyo/ros2-perf-multihost:latest
4678
```
4779

4880
Use the published GitHub Packages image [`ghcr.io/hal-lab-u-tokyo/ros2-perf-multihost:latest`](https://github.com/hal-lab-u-tokyo/ros2-perf-multihost/pkgs/container/ros2-perf-multihost). Generated `local_compose.yaml` and `host{N}_compose.yaml` files also reference the same image. For image build and push steps, see `docker/README.md`.
4981

50-
### Generate Scripts from the Project Root
82+
2. Generate execution scripts from a topology JSON file.
83+
84+
```bash
85+
python3 manager_scripts/generate_exec_scripts.py topology_example/simple.json --rmw fastdds --ws-dir performance_ws
86+
```
87+
88+
3. Run the generated local verification script.
89+
90+
```bash
91+
bash performance_ws/latest/exec_scripts/local_run.sh
92+
```
93+
94+
`local_run.sh` automatically sets `LOCAL_UID=$(id -u)` and `LOCAL_GID=$(id -g)` before running `docker compose`, which helps avoid root-owned files on bind mounts.
95+
96+
When using Zenoh, the script starts the Zenoh router first, then launches the host services, and stops the router automatically afterward.
97+
98+
### Generate Execution Scripts
99+
100+
Generate execution scripts (`host*_exec.sh`, `host*_run.sh`) and Docker Compose files from a JSON topology file.
51101

52102
```bash
53103
python3 manager_scripts/generate_exec_scripts.py <topology.json> [--rmw <rmw>] [--ws-dir <dir>] [--force|-f]
@@ -105,28 +155,63 @@ Generated files:
105155

106156
Each node launched from `host{N}_run.sh` or `local_run.sh` receives a `--log_dir` under `results/YYYY-MM-DD_hh-mm-ss/exec_logs/trial<trial_idx>/` inside the generated run directory. `results/latest` is updated as a symbolic link to the active run directory. Example: `performance_ws/latest/results/2026-04-26_13-21-45/exec_logs/trial1/`.
107157

108-
### Local Verification with Docker
158+
#### Runtime Options Supported by Generated Scripts
159+
160+
Generated `host*_run.sh` and `local_run.sh` scripts support the runtime options below. `--eval-time` is applied to every launched node (Publisher / Subscriber / Intermediate). `payload_size` and `period_ms` must be specified in each Publisher / Intermediate topic entry in the topology JSON, and those values are passed directly to Publisher / Intermediate nodes. `--trial-idx` is available only on `host*_run.sh` and `local_run.sh`. For the JSON schema, see [topology_example/README.md](./topology_example/README.md).
161+
162+
| Option | Short | Description | Default |
163+
|---|---|---|---|
164+
| --eval-time | -t | Evaluation time in seconds | 60 |
165+
| --trial-idx | -r | Trial index for local execution | 1 |
166+
167+
Examples:
168+
169+
```bash
170+
# Use default values
171+
./host1_exec.sh
172+
173+
# Override eval-time
174+
./host1_run.sh --eval-time 60
175+
176+
# Short options
177+
./host1_run.sh -t 60
178+
```
179+
180+
`--eval-time` is applied to all nodes launched through `*_run.sh` or `local_run.sh`. `payload_size` and `period_ms` are read from each Publisher/Intermediate entry in the topology JSON.
181+
182+
### Run with Docker
183+
184+
Use Docker when you want a containerized execution environment.
109185

110186
```bash
111187
bash performance_ws/latest/exec_scripts/local_run.sh
112188
```
113189

114-
`local_run.sh` automatically sets `LOCAL_UID=$(id -u)` and `LOCAL_GID=$(id -g)` before running `docker compose`, which helps avoid root-owned files on bind mounts.
190+
For multi-host execution, run the host-specific wrapper on each target host.
115191

116-
When using Zenoh, the script starts the Zenoh router first, then launches the host services, and stops the router automatically afterward.
192+
```bash
193+
# Run on host1
194+
bash performance_ws/latest/exec_scripts/host1_run.sh
195+
```
196+
197+
If needed, pull the image on each host in advance.
198+
199+
```bash
200+
docker pull ghcr.io/hal-lab-u-tokyo/ros2-perf-multihost:latest
201+
```
117202

118-
### Native Execution
203+
### Run Natively
119204

120-
To run in a native ROS 2 environment without Docker, set `ROS2_PERF_WS` to the project root.
205+
Use the native workflow when you want to execute ROS 2 nodes directly on the host without Docker.
121206

122207
```bash
123208
export ROS2_PERF_WS=$(pwd)
124209
bash performance_ws/latest/exec_scripts/host1_exec.sh
125210
```
126211

127-
If it is unset, the script falls back to the container default path.
212+
If `ROS2_PERF_WS` is unset, the script falls back to the container default path.
128213

129-
### Real Multi-Host Deployment
214+
### Multi-Host Deployment
130215

131216
Prepare the repository and the required Python environment at the same path on each host in advance.
132217

@@ -151,28 +236,10 @@ You can override the target paths on the command line.
151236
./manager_scripts/distribute_exec_scripts.sh --help
152237
```
153238

154-
```bash
155-
# Run on host1
156-
bash performance_ws/latest/exec_scripts/host1_run.sh
157-
```
158-
159-
If needed, pull the image on each host in advance.
160-
161-
```bash
162-
docker pull ghcr.io/hal-lab-u-tokyo/ros2-perf-multihost:latest
163-
```
164-
165-
## REST Server and Automated Performance Evaluation
239+
### Automated Benchmark via REST
166240

167241
In a multi-host setup, each Raspberry Pi runs a REST server implemented by `rest_server.py`. A controller script sends requests to those servers to automate benchmark execution.
168242

169-
Install the required Python packages with apt before starting the REST workflow.
170-
171-
```bash
172-
sudo apt update
173-
sudo apt install -y python3-flask python3-requests
174-
```
175-
176243
1. Start the REST server on every Raspberry Pi.
177244

178245
SSH into each host and launch `remote_hosts_scripts/rest_server.py` directly.
@@ -203,10 +270,62 @@ Main arguments:
203270
- `--scenario`: Scenario directory to use (default: `latest`)
204271
- `--eval-time`: Override evaluation time; if omitted, the default from `*_run.sh` or `*_exec.sh` is used
205272

206-
`performance_test.py` launches node groups via REST for each trial, then collects logs from each host with `scp`. On prepare, the manager creates `<ws-dir>/<scenario>/results/<session_timestamp>/` and updates `<ws-dir>/<scenario>/results/latest` to point to it. Trial logs are collected under `<ws-dir>/<scenario>/results/latest/logs/trial<N>/`, and aggregated outputs (for example `total_latency.csv`, `throughput.csv`, `host_trials_usage.csv`, `host_usage_summary.csv`) are written under `<ws-dir>/<scenario>/results/latest/csv/`.
207-
208273
When using Zenoh as the RMW, start the router on the manager host before running the benchmark.
209274

210275
```bash
211276
./manager_scripts/operate_zenoh_router.sh foreground
212277
```
278+
279+
### Results and Output Files
280+
281+
`performance_test.py` launches node groups via REST for each trial, then collects logs from each host with `scp`.
282+
283+
On prepare, the manager creates `<ws-dir>/<scenario>/results/<session_timestamp>/` and updates `<ws-dir>/<scenario>/results/latest` to point to it.
284+
285+
- Trial logs are collected under `<ws-dir>/<scenario>/results/latest/logs/trial<N>/`.
286+
- Aggregated outputs such as `total_latency.csv`, `throughput.csv`, `host_trials_usage.csv`, and `host_usage_summary.csv` are written under `<ws-dir>/<scenario>/results/latest/csv/`.
287+
288+
## Directory Structure
289+
290+
The main directories and their roles are as follows:
291+
292+
| Directory | Role |
293+
|---|---|
294+
| `manager_scripts/` | Generates topology-specific execution artifacts and provides helper scripts for distribution and router operation. |
295+
| `remote_hosts_scripts/` | Runs on each host (REST server, remote start orchestration, and host metrics collection). |
296+
| `performance_test/` | Executes trial automation, log collection, and CSV aggregation/analysis. |
297+
| `performance_ws/` | Stores generated scenarios, execution scripts, and run results. |
298+
| `topology_example/` | Provides example topology JSON files and schema guidance. |
299+
| `ros2_node_impl_ws/` | ROS 2 node implementation workspace used by generated execution scripts. |
300+
| `docker/` | Shared Docker image definition and compose-related assets. |
301+
302+
## Related Documents
303+
304+
For detailed usage in subdomains, see the following documents:
305+
306+
- [docker/README.md](./docker/README.md): Docker image build/push details and container workflow notes.
307+
- [topology_example/README.md](./topology_example/README.md): Topology JSON format and modeling guidance.
308+
- [ros2_node_impl_ws/README.md](./ros2_node_impl_ws/README.md): ROS 2 node workspace usage and build instructions.
309+
310+
## Troubleshooting
311+
312+
Common issues and fixes:
313+
314+
- `python3 manager_scripts/generate_exec_scripts.py ...` fails because output exists: rerun with `--force` or remove the existing scenario directory under `performance_ws/`.
315+
- `distribute_exec_scripts.sh` fails with SSH/SCP errors: verify hostnames, SSH keys, and that repository paths are identical across hosts.
316+
- REST benchmark does not start remote execution: ensure `python3 remote_hosts_scripts/rest_server.py` is running on every target host before calling `performance_test.py`.
317+
- Docker mode fails on remote hosts: pull `ghcr.io/hal-lab-u-tokyo/ros2-perf-multihost:latest` and confirm Docker permissions on each host.
318+
- Native mode cannot find workspace paths: set `ROS2_PERF_WS` to the project root before running `host*_exec.sh`.
319+
- Expected CSV outputs are missing: check `<ws-dir>/<scenario>/results/latest/logs/trial<N>/` for trial logs and inspect script stderr for analyzer failures.
320+
321+
## Contributing and License
322+
323+
Contributions are welcome. Please open an issue to discuss bugs, feature requests, or design changes before large modifications.
324+
325+
When submitting a pull request:
326+
327+
- Keep changes scoped and include a clear rationale.
328+
- Update documentation for user-facing behavior changes.
329+
- Include reproduction steps for bug fixes and benchmark-related changes.
330+
331+
This project is licensed under the terms in [LICENSE](./LICENSE).

0 commit comments

Comments
 (0)