Skip to content

Commit b0ef8c0

Browse files
JesusSilvaUtreraJesusSilvaUtrera
authored
Module 5 docs (#22)
* First draft of guide * Address comments * Final minor modifications * Refine guide with feedback --------- Co-authored-by: JesusSilvaUtrera <jesus.silva@ekumenlabs.com>
1 parent 39c1f28 commit b0ef8c0

1 file changed

Lines changed: 181 additions & 0 deletions

File tree

‎modules/module_5/README.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Module 5 – End-to-End Testing
2+
3+
In this module, the goal is to explore End-to-End (E2E) testing. This is the highest level of testing, designed to verify that the entire robotic system can successfully perform a complete "mission" or task from start to finish, just as it would in the real world.
4+
5+
- [Module 5 – End-to-End Testing](#module-5--end-to-end-testing)
6+
- [Objectives](#objectives)
7+
- [Motivation](#motivation)
8+
- [The rosbag Toolset](#the-rosbag-toolset)
9+
- [Recording Data](#recording-data)
10+
- [Inspecting Data](#inspecting-data)
11+
- [Playing Back Data](#playing-back-data)
12+
- [Manual End-to-End Testing](#manual-end-to-end-testing)
13+
- [Using Pre-recorded Rosbags](#using-pre-recorded-rosbags)
14+
- [Using Simulation Data](#using-simulation-data)
15+
- [Automated End-to-End Testing](#automated-end-to-end-testing)
16+
- [Replay\_testing](#replay_testing)
17+
- [References](#references)
18+
19+
## Objectives
20+
21+
By the end of the module, participants will be able to:
22+
23+
- Understand the purpose of E2E testing and how it differs from integration testing.
24+
- Use the `ros2 bag` command-line interface to record, inspect, and play data.
25+
- Perform a manual E2E test by launching a system and replaying a rosbag.
26+
- Understand the concept and workflow for automating E2E tests for CI/CD.
27+
28+
## Motivation
29+
30+
While integration tests (Module 4) confirm that a subset of nodes can collaborate (like checking the topics or services communication), they don't verify the entire "plot". End-to-end testing runs the entire system, bringing simulation into the picture.
31+
32+
Its purpose is to validate the robot's ability to meet a high-level requirement, such as "Given a map, navigate to the kitchen" or "Inspect all waypoints in the warehouse".
33+
34+
The importance of E2E testing in robotics includes:
35+
36+
<!-- TODO: Review this, and see if it matches more with field debugging -->
37+
38+
- **Validating the "Mission"**: It's the only test level that answers the question: "Does the robot actually achieve its goal?"
39+
- **Testing Against Reality**: By using data recorded from the real world (or a high-fidelity simulator), rosbags provide a "ground truth" scenario. This makes possible to test complex, emergent behaviors and edge cases that are impossible to script in a simple integration test.
40+
- **Ultimate Regression-Proofing**: An E2E test is the ultimate safety net. If a change in any package (perception, control, navigation) breaks the robot's ability to complete its mission, a good E2E test will catch it.
41+
- **Debugging Complex Failures**: When a robot fails in the field, a rosbag of that failure is invaluable. It can be replayed in a simulator over and over until the root cause (for example, a race condition, a state machine logic error) is found.
42+
43+
## The rosbag Toolset
44+
45+
The most important tool for E2E testing is rosbag (specifically, rosbag2 in ROS 2). It makes possible to capture and replay the entire message-passing state of a ROS system.
46+
47+
A rosbag is essentially a database (in ROS 2, from the Iron distro onwards, MCAP is used by default, in older distros it's a sqlite3 file) that stores all messages published on specific topics, along with their timestamps.
48+
49+
### Recording Data
50+
To record data, use the `ros2 bag record` command. It's possible to either specify topics or record everything:
51+
52+
```bash
53+
# Record all topics on the system
54+
ros2 bag record -a
55+
56+
# Record only specific topics
57+
ros2 bag record /scan /odom /tf
58+
59+
# Record to a specific bag file (directory)
60+
ros2 bag record -o my_mission_bag /scan /odom
61+
```
62+
63+
This last command creates a directory (for example, `my_mission_bag/`) containing the database file and metadata.
64+
65+
### Inspecting Data
66+
67+
Before using a bag, it's necessary to know what's in it.
68+
69+
```bash
70+
# Get a summary of the bag file
71+
ros2 bag info my_mission_bag/
72+
73+
# Example output:
74+
# Files: my_mission_bag.mcap
75+
# Bag size: 15.8 MiB
76+
# Storage id: mcap
77+
# Duration: 1m 10s
78+
# Start: Oct 17 2025 15:30:01.000
79+
# End: Oct 17 2025 15:31:11.000
80+
# Messages: 3013
81+
# Topic information:
82+
# Topic: /turtle1/cmd_vel | Type: geometry_msgs/msg/Twist | Count: 9 | Serialization Format: cdr
83+
# Topic: /turtle1/pose | Type: turtlesim/msg/Pose | Count: 3004 | Serialization Format: cdr
84+
```
85+
86+
### Playing Back Data
87+
88+
This is the core of the workflow. `ros2 bag play` republishes all the messages in the bag exactly as they were recorded. If the `--clock` flag is passed, it publishes `/clock` messages that makes the playback follow the original timing via simulated time, so nodes with the parameter `use_sim_time = True` experience the same timeline as when the bag was recorded, but those without this parameter set will still see real time.
89+
90+
```bash
91+
# Play a bag
92+
ros2 bag play my_mission_bag/
93+
94+
# Play in a loop (great for repeated testing)
95+
ros2 bag play -l my_mission_bag/
96+
97+
# Play at 2.0x speed
98+
ros2 bag play --rate 2.0 my_mission_bag/
99+
```
100+
101+
This command acts like a "data simulator", providing a perfectly repeatable stream of inputs to the system.
102+
103+
## Manual End-to-End Testing
104+
105+
This is the most common and intuitive form of E2E testing. It involves a human operator launching the system, providing a scenario (usually via a rosbag), and visual or log-based verification of the result.
106+
107+
This is perfect for debugging, or for a final "sanity check" before merging a major feature.
108+
109+
### Using Pre-recorded Rosbags
110+
111+
A typical manual test session looks like this:
112+
113+
1. Launch the System: Start the core nodes of the robot, along with visualization tools like `Rviz` to be able to monitor the progress.
114+
2. Provide Input: Instead of launching the drivers (like the camera or lidar node), use `ros2 bag play`. This feeds the recorded data (for example, `/scan`, `/camera/image_raw`) into the system.
115+
3. Observe and Verify: The engineer watches the output:
116+
- In `RViz`: "Does the robot's navigation visualization show it reaching the goal?"
117+
- In the terminal: "Did the mission control node log 'MISSION_COMPLETE'?"
118+
4. Analyze: If it fails, now it's possible to debug the running nodes, knowing the input data is identical every single time.
119+
120+
This workflow is incredibly powerful but has one major drawback: it's not automated. It relies on a human to launch, observe, and judge success.
121+
122+
> [!NOTE]
123+
> There will be cases where the rosbag won't be enough or where it's necessary to test other features in a known environment. In these cases, starting a simulator like `Gazebo` along with playing the rosbag is a good idea, but it's not common.
124+
125+
### Using Simulation Data
126+
127+
While rosbags are invaluable for reproducing real-world scenarios, simulation-based E2E testing provides complementary advantages. Instead of replaying fixed data, the simulator generates live sensor streams (for example, LiDAR, camera) and physics interactions in real time.
128+
129+
This is really useful specially for testing certain features:
130+
131+
- Test dynamic conditions such as moving obstacles or lightning changes.
132+
- Parameterize worlds and robot configurations to explore edge cases.
133+
- Ideal for CI, since there is the possibility of running simulations in headless mode on a server (using `--headless-rendering` flag in `Gazebo` for example).
134+
135+
Example workflow:
136+
137+
1. Launch the robot’s navigation and perception stacks inside the simulator.
138+
2. Define an automated mission (for example, "Navigate to the charging dock").
139+
3. Let the simulator generate data live, instead of replaying a rosbag.
140+
4. Use `launch_testing` (presented in Module 4) to check that the final goal is achieved (for example, via `/mission_status` or `/odom`).
141+
142+
This approach complements rosbag replay testing: the former focuses on realism and variability, the latter on repeatability and regression detection.
143+
144+
## Automated End-to-End Testing
145+
146+
The "holy grail" of robotics testing is to automate the manual workflow. This is key to build a robust Continuous Integration (CI) pipeline, as it will be covered in Module 6.
147+
148+
This approach combines everything learned so far:
149+
150+
1. Test Framework: Use `launch_testing` as the test runner.
151+
2. `generate_test_description()`: This function is now responsible for launching the entire system:
152+
- The robot's core nodes (navigation, perception, control).
153+
- A simulator (for example, Gazebo) with a specific world file.
154+
- A `ros2 bag play` command (using ExecuteProcess) to provide the test scenario.
155+
3. `unittest.TestCase`:
156+
- The test case can no longer "watch" RViz. It must check for success programmatically.
157+
- It creates its own temporary rclpy node.
158+
- It subscribes to a "result" topic (for example, `/mission_status`).
159+
- It waits for the rosbag to finish playing.
160+
- It asserts the final state. For example, it might subscribe to `/odom` and assert that the robot's final position is within 10cm of the desired goal.
161+
162+
This creates a fully self-contained test that can be run on a server with `colcon test`. It is the most complex type of test to write, but it provides the highest possible confidence in the system's stability.
163+
164+
It's important to highlight that using both the simulator and the rosbag can be misleading. One might think: "If I'm already using a simulation, why would I need a rosbag?" And this is true in some cases, but there are special cases when you might want to combine both, for example, with a probllematic trajectory pre-recorded that needs to be tested again but without assuming risks in the real world.
165+
166+
### Replay_testing
167+
168+
Polymath Robotics `replay_testing` tool provides a convenient wrapper for End-to-End testing with rosbags. It automates conventions like:
169+
170+
- Launching the system and a rosbag together.
171+
- Synchronizing `/clock` and simulated time.
172+
- Checking for topic availability and test completion conditions.
173+
174+
This makes it an excellent starting point for teams who want automated mission playback without writing custom `launch_testing` code from scratch.
175+
176+
## References
177+
178+
- [ROS 2 Documentation: ros2 bag CLI](https://docs.ros.org/en/jazzy/Tutorials/Beginner-CLI-Tools/Recording-And-Playing-Back-Data/Recording-And-Playing-Back-Data.html)
179+
- [ROS 2 rosbag2 GitHub Repository](https://github.com/ros2/rosbag2)
180+
- [replay_testing Polymath Robotics](https://github.com/PolymathRobotics/replay_testing)
181+
- [Example of system tests in Nav2](https://github.com/ros-navigation/navigation2/blob/main/nav2_system_tests/src/system/README.md)

0 commit comments

Comments
 (0)