Skip to content

Commit 5c5626a

Browse files
technolojinCopilot
andauthored
feat(tools): automatic system definition file generation (#50)
* feat: add inline autoware_launch parameter refiner script and documentation - Introduced a new Python script, `inline_autoware_launch_params.py`, to inline `autoware_launch` config dependencies into `*.parameter_set.yaml` files. This script resolves parameter file references, compares them with package defaults, and updates the YAML files accordingly. - Added a comprehensive README.md to document the tool's functionality, usage, options, and examples, enhancing user understanding and accessibility. These changes improve the management of parameter configurations in the Autoware System Designer, facilitating a more streamlined workflow. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance ROS 2 topology report functionality - Added new filtering options to include or exclude tool nodes and common topics in the topology report, improving customization for users. - Introduced functions to manage parameter service renames and namespace summaries, enhancing clarity in reporting changes. - Updated command-line arguments to allow users to control the visibility of tool nodes and common topics, providing a more tailored reporting experience. These changes improve the usability and flexibility of the ROS 2 topology report tool, facilitating better analysis of system configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add README for topology-analyzer tool - Introduced a comprehensive README.md for the topology-analyzer tool, detailing its purpose, usage, and workflow for capturing and comparing ROS 2 system graphs. - Documented the functionality of two scripts: `ros2_graph_snapshot.py` for capturing node graphs and `ros2_topology_report.py` for generating reports and diffs. - Included examples and options for both scripts, enhancing user understanding and accessibility of the tool. These changes provide essential documentation for users to effectively utilize the topology-analyzer in their ROS 2 projects. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance topology analyzer with composable node container support - Updated the README.md to include details about the new **Composable Node Containers** section, which lists containers and their composable nodes when `component_info` data is available. - Enhanced the `ros2_graph_snapshot.py` to capture component information, linking nodes to their respective containers. - Improved the `ros2_topology_report.py` to report container changes, including added, removed, and changed containers, and updated the matching algorithm to consider container names as a similarity factor. - Added documentation for the new container changes in the report output reference, improving user understanding of composable node transitions. These changes significantly enhance the functionality of the topology analyzer, allowing for better tracking and reporting of composable nodes within ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add process discovery features to topology analyzer - Updated README.md to include the new `--no-process` option, allowing users to skip OS process discovery when needed. - Enhanced `ros2_graph_snapshot.py` to capture detailed process information, including PID, executable path, package name, and loaded ROS libraries for each node. - Introduced a new section in the README detailing the `process` field in the snapshot JSON format, improving user understanding of the captured data. These changes significantly enhance the topology analyzer's capabilities, providing users with more granular insights into node processes within ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: implement process grouping and diffing in topology report - Added functionality to group nodes by OS PID, providing a summary of unique processes and nodes without process information in the topology report. - Introduced a new method to compare process information, capturing changes in executor type and package details. - Enhanced the report output to include detailed process summaries, improving insights into node execution contexts within ROS 2 systems. These changes significantly enhance the topology analyzer's reporting capabilities, offering users a clearer view of process-related data. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance topology matching algorithm and signature handling - Refactored the matching score calculation in `ros2_topology_report.py` to prioritize interface type composition over topic names, improving node identity recognition across namespace changes. - Updated the `_match_nodes` function to exclude endpoint similarity from fuzzy matching, focusing on type and name similarities for better accuracy. - Introduced a new function `iter_type_items` in `ros2_topology_common.py` to yield direction-qualified type tokens, facilitating type-composition-first matching. - Modified command-line arguments in `ros2_topology_similarity.py` to allow users to specify matching criteria (type, name, or both), enhancing flexibility in similarity comparisons. These changes significantly improve the robustness and usability of the topology analyzer, enabling more accurate comparisons of node signatures in ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance node matching algorithm for same-type compositions - Implemented a new mechanism in `ros2_topology_report.py` to disambiguate M:N node groups sharing the same type by utilizing name similarity, improving accuracy in node identity recognition. - Introduced a margin check to prevent false matches, ensuring robust mapping between old and new nodes. - Enhanced the `_match_nodes` function to build mutual-best name-similarity tables, facilitating better handling of nodes with identical types but different names. These changes significantly improve the topology analyzer's ability to accurately match nodes across different snapshots in ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: refactor topology analyzer with modular structure and enhanced functionality - Introduced a new modular architecture for the topology analyzer by creating a `lib` directory containing shared utilities, filters, matching algorithms, and rendering functions. - Moved common functions and constants from `ros2_topology_common.py` to `lib/common.py`, ensuring a single source of truth for shared resources. - Implemented new filtering functions in `lib/filters.py` to manage node and topic visibility, improving report customization. - Enhanced the matching algorithm in `lib/matching.py` to support more robust node identity recognition across snapshots. - Added comprehensive report rendering capabilities in `lib/render.py`, allowing for detailed output of topology changes and node relationships. These changes significantly improve the maintainability and extensibility of the topology analyzer, enabling better analysis and reporting of ROS 2 system configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: modularize ros2_graph_snapshot with new component detection and graph utilities - Refactored `ros2_graph_snapshot.py` to utilize a modular structure, enhancing maintainability and functionality. - Introduced new modules in `lib/snapshot` for component detection, graph management, and parameter collection, improving the overall architecture. - Added `components.py` for querying composable node containers, `graph.py` for node graph information, and `params.py` for parameter handling. - Enhanced process discovery capabilities in `proc.py`, allowing for detailed mapping of ROS 2 nodes to their respective processes and components. These changes significantly improve the topology analyzer's ability to capture and report on ROS 2 system configurations, facilitating better analysis and insights. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: introduce auto-system-config-generator for generating Autoware YAML configs - Added `generate_system_config.py` to parse flattened ROS 2 launch XML and produce Autoware System Designer YAML configurations. - Created a README.md detailing usage, prerequisites, and workflow for the new tool. - Introduced `component_map.yaml` for customizing component names and entity references. - Implemented supporting modules in `lib` for connection resolution, node grouping, and YAML emission, enhancing the overall functionality and maintainability of the system configuration generation process. These additions significantly improve the automation of system configuration generation for Autoware, streamlining the setup process for users. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: update README and enhance connection resolution logic - Updated README.md to reflect the correct repository link for `launch_unifier`. - Modified `connection_resolver.py` to skip processing for root namespace groups, improving connection resolution accuracy. - Refined internal connection handling in `emitter.py` to utilize deduplicated instance names, enhancing YAML output consistency. - Improved port name handling in `launch_parser.py` to replace slashes with underscores for better format compatibility. These changes enhance the functionality and usability of the auto-system-config-generator, ensuring more accurate and reliable YAML configuration generation. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * refactor: enhance connection resolution and introduce namespace tree structure - Updated `connection_resolver.py` to improve port name resolution by incorporating group namespace handling. - Refined `launch_parser.py` to preserve slashes in port names, enhancing compatibility with topic structures. - Introduced `namespace_tree.py` to build a recursive namespace tree, facilitating better organization and management of nodes and containers. - Enhanced `emitter.py` to support the new namespace structure, improving the overall configuration generation process. These changes significantly improve the accuracy and usability of the auto-system-config-generator, enabling more effective YAML configuration generation for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add graph JSON support for enhanced node remapping - Introduced `graph_parser.py` to parse ROS 2 graph snapshot JSON and merge topics into node records. - Updated `generate_system_config.py` to include a new `--graph-json` argument for specifying the graph snapshot file. - Enhanced connection resolution in `connection_resolver.py` and `emitter.py` to accommodate synthetic remaps from the graph data. - Revised `README.md` to document the new graph JSON feature and its impact on configuration generation. These changes improve the auto-system-config-generator's ability to integrate hard-coded topics, enhancing the overall configuration generation process for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add node configuration generation for undeclared ROS 2 nodes - Introduced functionality in `generate_system_config.py` to generate `*.node.yaml` files for node entities not already defined in known package directories. - Added a new `--node-configs` argument to the command-line interface for enabling this feature. - Created `node_emitter.py` to handle the logic for collecting node records, checking for existing definitions, and emitting the corresponding YAML files. - Enhanced the overall configuration generation process for Autoware by ensuring that all node entities are properly defined. These changes improve the auto-system-config-generator's ability to manage node configurations, facilitating better integration and usability in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance node configuration generation with namespace support - Updated `generate_system_config.py` to create subdirectories for node YAML files based on their common namespace, improving organization. - Introduced `namespace_for_entity` and `_common_namespace` functions in `node_emitter.py` to determine the appropriate namespace for node records. - Refined filtering logic in `emitter.py` to exclude components that are part of the same namespace, enhancing connection resolution. These changes improve the structure and clarity of generated node configurations, facilitating better management in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: introduce launch unifier for ROS 2 launch file processing - Added `run_launch_unifier.py` to flatten ROS 2 launch files into XML and generate PlantUML diagrams, enhancing usability without requiring modified `launch_ros`. - Implemented a series of modules including `filter.py`, `launch_maker.py`, `parser.py`, and `serialization.py` to handle entity filtering, launch file generation, and serialization of launch entities. - Introduced `patches.py` to apply necessary monkey-patches for compatibility with unmodified `launch_ros`. - Created templates for generating launch files and PlantUML diagrams, improving the overall configuration generation process for ROS 2 environments. These additions significantly enhance the auto-system-config-generator's capabilities, streamlining the management and visualization of ROS 2 launch configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add support for extra arguments in composable nodes - Introduced `extra_arguments` attribute in `FinalAttributes` to handle additional parameters for composable nodes. - Updated `_patch_load_composable_nodes` to evaluate and append extra arguments to the final attributes. - Enhanced serialization in `serialization.py` to include `extra_arguments` when making entities serializable. - Modified Jinja2 template to render extra arguments in the generated launch files. These changes improve the flexibility and configurability of composable nodes in ROS 2 launch files. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: enhance system configuration generation with unified pipeline and live snapshot support - Updated `generate_system_config.py` to support a unified pipeline for generating Autoware System Designer YAML configs from ROS 2 launch files. - Introduced new command-line arguments for specifying launch files, arguments, and optional live snapshot capture. - Added helper functions in `lib/unifier.py` and `lib/snapshot.py` to facilitate launch file flattening and live graph snapshot capturing. - Enhanced `README.md` to document the new usage patterns and options for the configuration generator. These changes improve the usability and flexibility of the auto-system-config-generator, streamlining the process of generating system configurations for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * refactor: streamline output directory structure and enhance error messaging - Removed the `--unifier-output-dir` argument, consolidating output under a single `--output-dir`. - Updated functions in `generate_system_config.py` to reflect the new output structure, including subdirectories for unified launch files, snapshots, and system design files. - Improved error messages for launch arguments to clarify expected formats. - Enhanced serialization in `serialization.py` to support dynamic output directory paths. These changes improve the organization of generated files and enhance user experience by providing clearer feedback on input errors. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * refactor: update command-line arguments and output structure for system configuration generation - Replaced the `--node-configs` argument with `--no-node-configs` to clarify its purpose in skipping the generation of YAML files for undefined node entities. - Consolidated output directory structure to include a timestamp, enhancing organization of generated files. - Updated the README.md to reflect changes in command-line options and the new output directory format. These modifications improve clarity and usability in the configuration generation process for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: implement auto-system-config-generator with unified pipeline for YAML generation - Introduced a new script `generate_system_config.py` to facilitate the generation of Autoware System Designer YAML configurations from ROS 2 launch files. - Added support for a unified pipeline that includes parsing launch files, capturing live ROS 2 graph snapshots, and merging topics into node records. - Created a structured output directory for generated files, including system, module, and node YAML configurations. - Enhanced the README.md to provide comprehensive usage instructions and document new features. These changes significantly improve the usability and flexibility of the auto-system-config-generator, streamlining the process of generating system configurations for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: correct namespace handling in Jinja2 template for composable node containers - Updated the Jinja2 template to strip trailing slashes from the expanded node namespace when rendering composable node containers. This change ensures that the generated launch files have the correct namespace format, improving compatibility and clarity in the output. This modification enhances the accuracy of the generated launch files, contributing to better configuration management in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * enhance namespace handling in configuration generation - Added logic to move nodes whose full_path matches a child namespace into that namespace, preventing duplicate instances in the namespace tree. - This improvement streamlines the organization of namespaces, enhancing clarity and efficiency in the generated configurations for ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * refactor: improve entity handling and shutdown logic in launch unifier - Updated entity retrieval logic in `launch_maker.py` and `plantuml.py` to check for "entity" presence instead of relying on "children" key, enhancing robustness in entity processing. - Modified shutdown handling in `parser.py` to ensure proper coroutine execution only when necessary, improving resource management during service shutdown. - Enhanced `_patch_load_composable_nodes` in `patches.py` to resolve target container names without loading nodes, streamlining composable node handling. These changes enhance the efficiency and reliability of the launch unifier's entity management and service lifecycle operations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * update README.md for improved clarity and formatting - Changed code block syntax from plain to text for better readability. - Enhanced table formatting for sub-tools, flags, and options to improve clarity and organization. - Updated output directory example to use text formatting for consistency. These modifications enhance the documentation's usability and presentation, making it easier for users to understand the system configuration generation process. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * refactor: streamline code formatting and improve readability in parameter set refiner - Consolidated multiline statements into single lines for better clarity in `inline_autoware_launch_params.py`. - Enhanced table formatting in `README.md` for options to improve organization and readability. - Updated various sections in the codebase to maintain consistent formatting and improve overall code quality. These changes enhance the maintainability and usability of the parameter set refiner tool, making it easier for developers to navigate and understand the code. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add port name shortening functionality to connection resolver and emitter Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Update tools/system-config-generator/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Update tools/topology-analyzer/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: update README.md and improve XML file naming in system config generator - Updated the README.md to reflect the correct script path for generating launch XML files. - Enhanced the XML file naming convention in `serialization.py` to use a slugified format based on entity attributes, improving file organization and clarity. These changes enhance documentation accuracy and improve the output structure of generated XML files. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> --------- Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 92aa3e5 commit 5c5626a

46 files changed

Lines changed: 7123 additions & 1157 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# parameter-set-refiner
2+
3+
A tool to remove `autoware_launch` config file dependencies from
4+
`*.parameter_set.yaml` files by inlining only the parameter values that
5+
actually differ from each package's own defaults.
6+
7+
## Background
8+
9+
`*.parameter_set.yaml` files in the autoware system designer framework can
10+
reference config files from `autoware_launch` via `param_files` entries:
11+
12+
```yaml
13+
- node: /perception/object_recognition/detection/centerpoint/validation
14+
param_files:
15+
- obstacle_pointcloud_based_validator_param_path: >-
16+
$(find-pkg-share autoware_launch)/config/perception/.../obstacle_pointcloud_based_validator.param.yaml
17+
param_values: []
18+
```
19+
20+
This creates a hard dependency on `autoware_launch` at design time. The
21+
refiner resolves each such reference, compares the `autoware_launch` override
22+
against the package's own default config, and rewrites the entry so only the
23+
differing values are kept as inline `param_values`:
24+
25+
```yaml
26+
- node: /perception/object_recognition/detection/centerpoint/validation
27+
param_files: []
28+
param_values:
29+
- name: using_2d_validator
30+
value: true
31+
```
32+
33+
If the two files are identical, the reference is removed with no `param_values`
34+
added.
35+
36+
## Usage
37+
38+
```bash
39+
python3 inline_autoware_launch_params.py <parameter_set.yaml> [OPTIONS]
40+
```
41+
42+
### Options
43+
44+
| Option | Description |
45+
| --------------------- | ---------------------------------------------------------------------------------- |
46+
| `--workspace-src DIR` | Root of the colcon `src/` tree. Auto-detected from the input file path if omitted. |
47+
| `--dry-run` | Print what would change without writing the file. |
48+
| `--output FILE` | Write the result to `FILE` instead of modifying in place. |
49+
50+
### Example
51+
52+
```bash
53+
python3 inline_autoware_launch_params.py /path/to/autoware/src/launcher/autoware_launch/autoware_sample_designs/design/parameter_set/sample_system_perception.parameter_set.yaml --workspace-src /path/to/autoware/src
54+
```
55+
56+
## How it works
57+
58+
For each `param_files` entry referencing `$(find-pkg-share autoware_launch)/...`:
59+
60+
1. **Resolve the autoware_launch config** — maps the `$(find-pkg-share ...)` substitution to an actual file path in the workspace.
61+
62+
2. **Find the matching `*.node.yaml`** — searches the workspace for node definitions that declare the same `param_file` key. When multiple candidates exist (e.g. the generic `param_path` key appears in many nodes), the best match is selected using this priority:
63+
- Node whose declared default filename matches the autoware_launch config filename (exact match).
64+
- Node whose package contains `config/<autoware_launch_config_filename>`.
65+
- Node whose package name best matches the directory path of the autoware_launch config (path-segment scoring).
66+
67+
3. **Resolve the package default config** — locates the baseline config file to compare against. Prefers `<pkg>/config/<autoware_launch_config_filename>` (exact filename match) over the filename declared in `node.yaml`, to handle cases where a node.yaml's declared default points to a sibling config for a different mode.
68+
69+
4. **Diff recursively** — compares the autoware_launch config against the package default using a recursive, leaf-level diff. Only the parameters that actually differ are recorded, using dotted-path names (e.g. `association.can_assign.unknown`) so that identical sibling fields are not written.
70+
71+
5. **Rewrite the entry** — appends the differing values to `param_values` (skipping any already set), then removes the `autoware_launch` reference from `param_files`. Non-autoware_launch `param_files` entries are preserved unchanged.
72+
73+
### Output format
74+
75+
- Scalars: `value: true` / `value: 0.5`
76+
- Arrays: inline flow style `value: [0.098, 0.147, 0.078]`
77+
- Nested dicts: block mapping under the dotted-path name, e.g.:
78+
79+
```yaml
80+
- name: association.can_assign.unknown
81+
value:
82+
[polygon_tracker, multi_vehicle_tracker, pedestrian_and_bicycle_tracker]
83+
```
84+
85+
## Requirements
86+
87+
```text
88+
ruamel.yaml
89+
```
90+
91+
Install with:
92+
93+
```bash
94+
pip install ruamel.yaml
95+
```

0 commit comments

Comments
 (0)