fix(ros): make the interop gate count interop tests - #276
Open
YuanYuYuan wants to merge 2 commits into
Open
Conversation
The gate added in #271 asserted a non-zero total and then reported it as "N ROS interop tests ran against rmw_zenoh_cpp". That total is the whole hiroz-tests package. On a healthy run it printed 125, of which only 41 are interop -- the rest are lifecycle, cache, parameter_tests and friends. Deleting every interop test would still have printed a confident number. That is the defect #271 existed to remove, reintroduced in the fix. Three changes: - Count the six interop binaries by name and require each to be present. A run with 84 passing non-interop tests now fails instead of reporting "84 ROS interop tests ran". - Fail when any test self-skipped for a missing ros2 CLI. Those return early, still pass, and are still counted -- a pass with no interop in it. #271 named this failure mode in its own description and did not close it. - Gate the run that decided the outcome. The retry path re-ran nextest but the gate still parsed the first attempt, so a passing retry was validated against the discarded output, and a first attempt that died before producing a summary failed an otherwise-green run. Also drops the `0 tests run` branch: cargo-nextest 0.9.138 defaults --no-tests to fail, so it was unreachable. The repo already knew this -- test.yml passes --no-tests=warn precisely because the default fails. Proven in four directions: healthy output passes; 84 non-interop tests fail; a self-skip line fails; empty output fails.
The suite is `#![cfg(not(ros_humble))]` -- Humble has no type description service to interoperate with -- so it compiles to an empty binary there and contributes no tests. Requiring it on every distro failed a healthy humble run. Every other interop suite stays mandatory everywhere.
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The interop gate added in #271 does not count interop tests. It asserts a non-zero package total, then reports that total as interop:
Only 41 of those are interop tests. Counting test functions in
crates/hiroz-tests/tests/:pubsub_interopdemo_nodesservice_interoptype_description_interopaction_interopparameter_interopThe rest of the package is
lifecycle,cache,hu_meter,parameter_testsand friends. Delete every interop test and the gate still prints a confident number. That is the defect #271 was written to remove, reintroduced in the fix.What this PR does
One file changes:
scripts/test-ros.nu.ros2 CLI not availableros2CLI is no longer a pass0 tests runbranch--no-testsdefault is to failNotes on each:
Suite list is distro-dependent.
type_description_interop.rsis# — Humble has no type description service to interoperate with — so it is required on every distro except Humble. The other five suites are required everywhere.python_interopis deliberately absent: it is#![cfg(feature = "python-interop")], which this command does not enable.The self-skip check is not uniform across suites, and the message is what makes it detectable.
check_ros2_available()isCommand::new("ros2").arg("--version").output().is_ok()(crates/hiroz-tests/tests/common/mod.rs:295) — true if the process merely spawned, so a runner with a brokenrmw_zenoh_cppstill passes it. Of the suites that call it, onlypubsub_interopself-skips silently: it printsSkipping …: ros2 CLI not availableand returns, so the test still passes and still counts (lines 116, 198).demo_nodes,service_interop,parameter_interopandtype_description_interoppanic!with the same substring, so they already fail the run; the gate's check catches both spellings.action_interophas no such guard at all. #271 named this failure mode in its own description and did not close it.The
0 tests runbranch was unreachable. Established by reading, not by executing a zero-test run:.github/workflows/test.yml:167-171passes--no-tests=warnfor thehiroz-unioninvocation precisely because the default fails the job.Evidence
The new check fired on a real empty suite, on this branch's first CI run. The Humble job failed with:
The old gate had reported that same Humble run as
116 ROS interop tests ran. The cause turned out to be a legitimate distro difference rather than a defect, and is now exempted (47ddff36) — but it is the proof that the detector detects rather than merely never firing.Beyond that, the gate logic was exercised by feeding recorded nextest output through it. This is a manual check; there is no automated test for the gate in this repo.
ros2 CLI not availablelineBreaking changes
scripts/test-ros.nufails when interop suites are absent or self-skippedros2CLINo library, API or message-format change. CI-only; the gate becomes strictly stricter.
Coverage this does not have
Interop tests with ROS 2 <distro>jobs are ungated.github/workflows/test.ymlrunscargo nextest rundirectly (lines 160-171) and never callsscripts/test-ros.nu. Itshiroz-unioninvocation passes--no-tests=warn, so a zero-test run there is green by designinterop_suites