feat(tools): automatic system definition file generation#50
Merged
technolojin merged 31 commits intoautowarefoundation:mainfrom Apr 21, 2026
Merged
feat(tools): automatic system definition file generation#50technolojin merged 31 commits intoautowarefoundation:mainfrom
technolojin merged 31 commits intoautowarefoundation:mainfrom
Conversation
technolojin
changed the title
…tation - Introduced a new Python script, `inline_autoware_launch_params.py`, to inline `autoware_launch` config dependencies into `*.parameter_set.yaml` files. This script resolves parameter file references, compares them with package defaults, and updates the YAML files accordingly. - Added a comprehensive README.md to document the tool's functionality, usage, options, and examples, enhancing user understanding and accessibility. These changes improve the management of parameter configurations in the Autoware System Designer, facilitating a more streamlined workflow. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Added new filtering options to include or exclude tool nodes and common topics in the topology report, improving customization for users. - Introduced functions to manage parameter service renames and namespace summaries, enhancing clarity in reporting changes. - Updated command-line arguments to allow users to control the visibility of tool nodes and common topics, providing a more tailored reporting experience. These changes improve the usability and flexibility of the ROS 2 topology report tool, facilitating better analysis of system configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Introduced a comprehensive README.md for the topology-analyzer tool, detailing its purpose, usage, and workflow for capturing and comparing ROS 2 system graphs. - Documented the functionality of two scripts: `ros2_graph_snapshot.py` for capturing node graphs and `ros2_topology_report.py` for generating reports and diffs. - Included examples and options for both scripts, enhancing user understanding and accessibility of the tool. These changes provide essential documentation for users to effectively utilize the topology-analyzer in their ROS 2 projects. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Updated the README.md to include details about the new **Composable Node Containers** section, which lists containers and their composable nodes when `component_info` data is available. - Enhanced the `ros2_graph_snapshot.py` to capture component information, linking nodes to their respective containers. - Improved the `ros2_topology_report.py` to report container changes, including added, removed, and changed containers, and updated the matching algorithm to consider container names as a similarity factor. - Added documentation for the new container changes in the report output reference, improving user understanding of composable node transitions. These changes significantly enhance the functionality of the topology analyzer, allowing for better tracking and reporting of composable nodes within ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Updated README.md to include the new `--no-process` option, allowing users to skip OS process discovery when needed. - Enhanced `ros2_graph_snapshot.py` to capture detailed process information, including PID, executable path, package name, and loaded ROS libraries for each node. - Introduced a new section in the README detailing the `process` field in the snapshot JSON format, improving user understanding of the captured data. These changes significantly enhance the topology analyzer's capabilities, providing users with more granular insights into node processes within ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Added functionality to group nodes by OS PID, providing a summary of unique processes and nodes without process information in the topology report. - Introduced a new method to compare process information, capturing changes in executor type and package details. - Enhanced the report output to include detailed process summaries, improving insights into node execution contexts within ROS 2 systems. These changes significantly enhance the topology analyzer's reporting capabilities, offering users a clearer view of process-related data. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Refactored the matching score calculation in `ros2_topology_report.py` to prioritize interface type composition over topic names, improving node identity recognition across namespace changes. - Updated the `_match_nodes` function to exclude endpoint similarity from fuzzy matching, focusing on type and name similarities for better accuracy. - Introduced a new function `iter_type_items` in `ros2_topology_common.py` to yield direction-qualified type tokens, facilitating type-composition-first matching. - Modified command-line arguments in `ros2_topology_similarity.py` to allow users to specify matching criteria (type, name, or both), enhancing flexibility in similarity comparisons. These changes significantly improve the robustness and usability of the topology analyzer, enabling more accurate comparisons of node signatures in ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Implemented a new mechanism in `ros2_topology_report.py` to disambiguate M:N node groups sharing the same type by utilizing name similarity, improving accuracy in node identity recognition. - Introduced a margin check to prevent false matches, ensuring robust mapping between old and new nodes. - Enhanced the `_match_nodes` function to build mutual-best name-similarity tables, facilitating better handling of nodes with identical types but different names. These changes significantly improve the topology analyzer's ability to accurately match nodes across different snapshots in ROS 2 systems. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…functionality - Introduced a new modular architecture for the topology analyzer by creating a `lib` directory containing shared utilities, filters, matching algorithms, and rendering functions. - Moved common functions and constants from `ros2_topology_common.py` to `lib/common.py`, ensuring a single source of truth for shared resources. - Implemented new filtering functions in `lib/filters.py` to manage node and topic visibility, improving report customization. - Enhanced the matching algorithm in `lib/matching.py` to support more robust node identity recognition across snapshots. - Added comprehensive report rendering capabilities in `lib/render.py`, allowing for detailed output of topology changes and node relationships. These changes significantly improve the maintainability and extensibility of the topology analyzer, enabling better analysis and reporting of ROS 2 system configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
… graph utilities - Refactored `ros2_graph_snapshot.py` to utilize a modular structure, enhancing maintainability and functionality. - Introduced new modules in `lib/snapshot` for component detection, graph management, and parameter collection, improving the overall architecture. - Added `components.py` for querying composable node containers, `graph.py` for node graph information, and `params.py` for parameter handling. - Enhanced process discovery capabilities in `proc.py`, allowing for detailed mapping of ROS 2 nodes to their respective processes and components. These changes significantly improve the topology analyzer's ability to capture and report on ROS 2 system configurations, facilitating better analysis and insights. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…YAML configs - Added `generate_system_config.py` to parse flattened ROS 2 launch XML and produce Autoware System Designer YAML configurations. - Created a README.md detailing usage, prerequisites, and workflow for the new tool. - Introduced `component_map.yaml` for customizing component names and entity references. - Implemented supporting modules in `lib` for connection resolution, node grouping, and YAML emission, enhancing the overall functionality and maintainability of the system configuration generation process. These additions significantly improve the automation of system configuration generation for Autoware, streamlining the setup process for users. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Updated README.md to reflect the correct repository link for `launch_unifier`. - Modified `connection_resolver.py` to skip processing for root namespace groups, improving connection resolution accuracy. - Refined internal connection handling in `emitter.py` to utilize deduplicated instance names, enhancing YAML output consistency. - Improved port name handling in `launch_parser.py` to replace slashes with underscores for better format compatibility. These changes enhance the functionality and usability of the auto-system-config-generator, ensuring more accurate and reliable YAML configuration generation. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…structure - Updated `connection_resolver.py` to improve port name resolution by incorporating group namespace handling. - Refined `launch_parser.py` to preserve slashes in port names, enhancing compatibility with topic structures. - Introduced `namespace_tree.py` to build a recursive namespace tree, facilitating better organization and management of nodes and containers. - Enhanced `emitter.py` to support the new namespace structure, improving the overall configuration generation process. These changes significantly improve the accuracy and usability of the auto-system-config-generator, enabling more effective YAML configuration generation for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Introduced `graph_parser.py` to parse ROS 2 graph snapshot JSON and merge topics into node records. - Updated `generate_system_config.py` to include a new `--graph-json` argument for specifying the graph snapshot file. - Enhanced connection resolution in `connection_resolver.py` and `emitter.py` to accommodate synthetic remaps from the graph data. - Revised `README.md` to document the new graph JSON feature and its impact on configuration generation. These changes improve the auto-system-config-generator's ability to integrate hard-coded topics, enhancing the overall configuration generation process for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Introduced functionality in `generate_system_config.py` to generate `*.node.yaml` files for node entities not already defined in known package directories. - Added a new `--node-configs` argument to the command-line interface for enabling this feature. - Created `node_emitter.py` to handle the logic for collecting node records, checking for existing definitions, and emitting the corresponding YAML files. - Enhanced the overall configuration generation process for Autoware by ensuring that all node entities are properly defined. These changes improve the auto-system-config-generator's ability to manage node configurations, facilitating better integration and usability in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Updated `generate_system_config.py` to create subdirectories for node YAML files based on their common namespace, improving organization. - Introduced `namespace_for_entity` and `_common_namespace` functions in `node_emitter.py` to determine the appropriate namespace for node records. - Refined filtering logic in `emitter.py` to exclude components that are part of the same namespace, enhancing connection resolution. These changes improve the structure and clarity of generated node configurations, facilitating better management in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Added `run_launch_unifier.py` to flatten ROS 2 launch files into XML and generate PlantUML diagrams, enhancing usability without requiring modified `launch_ros`. - Implemented a series of modules including `filter.py`, `launch_maker.py`, `parser.py`, and `serialization.py` to handle entity filtering, launch file generation, and serialization of launch entities. - Introduced `patches.py` to apply necessary monkey-patches for compatibility with unmodified `launch_ros`. - Created templates for generating launch files and PlantUML diagrams, improving the overall configuration generation process for ROS 2 environments. These additions significantly enhance the auto-system-config-generator's capabilities, streamlining the management and visualization of ROS 2 launch configurations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Introduced `extra_arguments` attribute in `FinalAttributes` to handle additional parameters for composable nodes. - Updated `_patch_load_composable_nodes` to evaluate and append extra arguments to the final attributes. - Enhanced serialization in `serialization.py` to include `extra_arguments` when making entities serializable. - Modified Jinja2 template to render extra arguments in the generated launch files. These changes improve the flexibility and configurability of composable nodes in ROS 2 launch files. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…nd live snapshot support - Updated `generate_system_config.py` to support a unified pipeline for generating Autoware System Designer YAML configs from ROS 2 launch files. - Introduced new command-line arguments for specifying launch files, arguments, and optional live snapshot capture. - Added helper functions in `lib/unifier.py` and `lib/snapshot.py` to facilitate launch file flattening and live graph snapshot capturing. - Enhanced `README.md` to document the new usage patterns and options for the configuration generator. These changes improve the usability and flexibility of the auto-system-config-generator, streamlining the process of generating system configurations for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…saging - Removed the `--unifier-output-dir` argument, consolidating output under a single `--output-dir`. - Updated functions in `generate_system_config.py` to reflect the new output structure, including subdirectories for unified launch files, snapshots, and system design files. - Improved error messages for launch arguments to clarify expected formats. - Enhanced serialization in `serialization.py` to support dynamic output directory paths. These changes improve the organization of generated files and enhance user experience by providing clearer feedback on input errors. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…em configuration generation - Replaced the `--node-configs` argument with `--no-node-configs` to clarify its purpose in skipping the generation of YAML files for undefined node entities. - Consolidated output directory structure to include a timestamp, enhancing organization of generated files. - Updated the README.md to reflect changes in command-line options and the new output directory format. These modifications improve clarity and usability in the configuration generation process for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…r YAML generation - Introduced a new script `generate_system_config.py` to facilitate the generation of Autoware System Designer YAML configurations from ROS 2 launch files. - Added support for a unified pipeline that includes parsing launch files, capturing live ROS 2 graph snapshots, and merging topics into node records. - Created a structured output directory for generated files, including system, module, and node YAML configurations. - Enhanced the README.md to provide comprehensive usage instructions and document new features. These changes significantly improve the usability and flexibility of the auto-system-config-generator, streamlining the process of generating system configurations for Autoware. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…e containers - Updated the Jinja2 template to strip trailing slashes from the expanded node namespace when rendering composable node containers. This change ensures that the generated launch files have the correct namespace format, improving compatibility and clarity in the output. This modification enhances the accuracy of the generated launch files, contributing to better configuration management in ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Added logic to move nodes whose full_path matches a child namespace into that namespace, preventing duplicate instances in the namespace tree. - This improvement streamlines the organization of namespaces, enhancing clarity and efficiency in the generated configurations for ROS 2 environments. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Updated entity retrieval logic in `launch_maker.py` and `plantuml.py` to check for "entity" presence instead of relying on "children" key, enhancing robustness in entity processing. - Modified shutdown handling in `parser.py` to ensure proper coroutine execution only when necessary, improving resource management during service shutdown. - Enhanced `_patch_load_composable_nodes` in `patches.py` to resolve target container names without loading nodes, streamlining composable node handling. These changes enhance the efficiency and reliability of the launch unifier's entity management and service lifecycle operations. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
- Changed code block syntax from plain to text for better readability. - Enhanced table formatting for sub-tools, flags, and options to improve clarity and organization. - Updated output directory example to use text formatting for consistency. These modifications enhance the documentation's usability and presentation, making it easier for users to understand the system configuration generation process. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
technolojin
force-pushed
the
parameter-tools
branch
from
5cbc6f6 to
1ce874e
Compare
…eter set refiner - Consolidated multiline statements into single lines for better clarity in `inline_autoware_launch_params.py`. - Enhanced table formatting in `README.md` for options to improve organization and readability. - Updated various sections in the codebase to maintain consistent formatting and improve overall code quality. These changes enhance the maintainability and usability of the parameter set refiner tool, making it easier for developers to navigate and understand the code. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…nd emitter Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a unified toolchain for generating Autoware System Designer definition files from a ROS 2 launch file, optionally enriched with live runtime topology data, and expands the topology-analyzer utilities to support richer snapshot/diff/report workflows.
Changes:
- Enhanced
topology-analyzerto capture component container + OS process metadata in snapshots, and added a name-agnostic matching/diff/report implementation. - Added
system-config-generatorpipeline to flatten launch files (vendoredlaunch_unifier), optionally merge live graph topics, and emit system/module/node/parameter-set YAML skeletons. - Added
parameter-set-refinerto inlineautoware_launchparameter overrides into*.parameter_set.yamlfiles.
Reviewed changes
Copilot reviewed 43 out of 46 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/topology-analyzer/ros2_graph_snapshot.py | Refactors snapshot logic and enriches nodes with component/process info; adds --no-process. |
| tools/topology-analyzer/lib/snapshot/proc.py | New /proc-based process/executor/library/component-class discovery. |
| tools/topology-analyzer/lib/snapshot/params.py | Extracted parameter collection into reusable helper. |
| tools/topology-analyzer/lib/snapshot/graph.py | Introduces shared NodeGraphInfo + graph helper utilities. |
| tools/topology-analyzer/lib/snapshot/components.py | Adds component container detection via /_container/list_nodes. |
| tools/topology-analyzer/lib/snapshot/init.py | Package initializer for snapshot helpers. |
| tools/topology-analyzer/lib/render.py | Adds Markdown rendering for single snapshot + diff reports. |
| tools/topology-analyzer/lib/process.py | Adds helpers for grouping composable nodes and processes in reports. |
| tools/topology-analyzer/lib/matching.py | Implements multi-pass node matching across snapshots. |
| tools/topology-analyzer/lib/filters.py | Adds node/topic filtering helpers and param-service rename suppression. |
| tools/topology-analyzer/lib/diff.py | Implements endpoint/edge/component/process diff calculations. |
| tools/topology-analyzer/lib/common.py | Centralizes shared types/constants/utilities used by topology-analyzer. |
| tools/topology-analyzer/lib/init.py | Library package initializer. |
| tools/topology-analyzer/functions/ros2_topology_similarity.py | Updates similarity CLI to support --match-by modes. |
| tools/topology-analyzer/functions/ros2_topology_common.py | Replaces duplicated logic with shim that re-exports lib/common.py. |
| tools/topology-analyzer/README.md | New documentation for snapshot/report/diff workflow and formats. |
| tools/system-config-generator/scripts/unify_launch.py | Adds standalone runner for vendored launch_unifier. |
| tools/system-config-generator/pipeline/port_utils.py | Adds unique-suffix shortening for port names. |
| tools/system-config-generator/pipeline/namespace_tree.py | Builds recursive namespace tree for generation. |
| tools/system-config-generator/pipeline/launch_runner.py | Runs launch_unifier with monkey patches and writes flattened XML/PUML. |
| tools/system-config-generator/pipeline/launch_parser.py | Parses flattened XML into structured node/container records. |
| tools/system-config-generator/pipeline/graph_snapshot.py | Captures live snapshot by running topology-analyzer as a subprocess. |
| tools/system-config-generator/pipeline/graph_parser.py | Parses graph.json and merges runtime topics into node remap records. |
| tools/system-config-generator/pipeline/emitter/system.py | Emits top-level *.system.yaml (tree-mode) and node group definitions. |
| tools/system-config-generator/pipeline/emitter/parameter.py | Emits per-component *.parameter_set.yaml. |
| tools/system-config-generator/pipeline/emitter/node.py | Emits missing *.node.yaml entity definitions and discovers existing ones. |
| tools/system-config-generator/pipeline/emitter/module.py | Emits recursive *.module.yaml and module-level wiring. |
| tools/system-config-generator/pipeline/emitter/init.py | Exposes emitter API. |
| tools/system-config-generator/pipeline/connection_resolver.py | Resolves cross-component topic connections from remap entries. |
| tools/system-config-generator/pipeline/init.py | Pipeline package initializer. |
| tools/system-config-generator/launch_unifier/templates/launch_plantuml.jinja2 | Template for PlantUML entity-tree visualization. |
| tools/system-config-generator/launch_unifier/templates/launch_generated.jinja2 | Template for flattened XML launch output. |
| tools/system-config-generator/launch_unifier/serialization.py | Serializes resolved launch entities for XML/template generation. |
| tools/system-config-generator/launch_unifier/plantuml.py | Renders PlantUML from serialized entity tree. |
| tools/system-config-generator/launch_unifier/patches.py | Monkey-patches launch_ros to expose resolved attributes without launching nodes. |
| tools/system-config-generator/launch_unifier/parser.py | Builds and filters entity tree from launch evaluation. |
| tools/system-config-generator/launch_unifier/launch_maker.py | Generates flattened XML using Jinja templates. |
| tools/system-config-generator/launch_unifier/filter.py | Filters unnecessary/invalid launch containers/entities. |
| tools/system-config-generator/launch_unifier/init.py | Vendored package header. |
| tools/system-config-generator/launch_unifier/LICENSE | Includes Apache-2.0 license for vendored launch_unifier. |
| tools/system-config-generator/generate_system_config.py | Main unified CLI pipeline producing system/module/node/parameter-set YAML. |
| tools/system-config-generator/config/component_map.yaml | Default namespace → component naming/entity override mapping. |
| tools/system-config-generator/README.md | Documentation for generator workflow, options, and output layout. |
| tools/parameter-set-refiner/inline_autoware_launch_params.py | Inlines autoware_launch param overrides into parameter_set YAMLs. |
| tools/parameter-set-refiner/README.md | Documentation for parameter-set-refiner usage and behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
…nerator - Updated the README.md to reflect the correct script path for generating launch XML files. - Enhanced the XML file naming convention in `serialization.py` to use a slugified format based on entity attributes, improving file organization and clarity. These changes enhance documentation accuracy and improve the output structure of generated XML files. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>
technolojin
force-pushed
the
parameter-tools
branch
from
3a5f001 to
5e993fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
import launcher_unifier developed by @xmfcx https://github.com/xmfcx/launch_unifier_ws
implemented system-config-generator
updated topology analyzer for better matching algorithm
parameter-set-refiner is added. it is to flattening out from copied parameter files.