Skip to content

Conversation

@MarcoPolo
Copy link
Contributor

interop testing the next go-libp2p release

@dhuseby
Copy link
Contributor

dhuseby commented Jan 24, 2026

The "contract" has changed between the test framework and the test applications. It is fully documented in: write-a-transport-test-app.md

If you don't want to change the test application in the go-libp2p repository, you can add a patch file in the images/ subfolders like I did for the images/rust/v0.56 transport interop test and then you add patchPath and patchFile to the implementation object in images.yaml

Signed-off-by: Dave Grantham <[email protected]>
@dhuseby
Copy link
Contributor

dhuseby commented Jan 24, 2026

The app built and the tests ran but some are failing.

✗ Failed: 24

  • go-v0.47-pre x rust-v0.56 (tcp, tls, yamux)
  • go-v0.47-pre x rust-v0.56 (ws, noise, yamux)
  • go-v0.47-pre x rust-v0.56 (ws, tls, yamux)
  • go-v0.47-pre x rust-v0.56 (quic-v1)
  • go-v0.47-pre x rust-v0.56 (webrtc-direct)
  • rust-v0.56 x go-v0.47-pre (webrtc-direct)
  • rust-v0.56 x go-v0.47-pre (ws, tls, yamux)
  • rust-v0.56 x go-v0.47-pre (tcp, noise, yamux)
  • rust-v0.56 x go-v0.47-pre (quic-v1)
  • rust-v0.56 x go-v0.47-pre (ws, noise, yamux)
  • rust-v0.56 x go-v0.47-pre (tcp, tls, yamux)
  • go-v0.47-pre x js-v1.x (ws, noise, yamux)
  • go-v0.47-pre x js-v2.x (tcp, noise, yamux)
  • go-v0.47-pre x js-v2.x (ws, noise, yamux)
  • go-v0.47-pre x js-v3.x (tcp, noise, yamux)
  • js-v1.x x go-v0.47-pre (tcp, noise, yamux)
  • js-v1.x x go-v0.47-pre (wss, noise, yamux)
  • js-v1.x x go-v0.47-pre (ws, noise, yamux)
  • js-v2.x x go-v0.47-pre (tcp, noise, yamux)
  • js-v2.x x go-v0.47-pre (wss, noise, yamux)
  • js-v2.x x go-v0.47-pre (ws, noise, yamux)
  • js-v3.x x go-v0.47-pre (tcp, noise, yamux)
  • js-v3.x x go-v0.47-pre (wss, noise, yamux)
  • js-v3.x x go-v0.47-pre (ws, noise, yamux)

If you download the artifacts there's a logs subfolder with the log captures from each individual test.

@MarcoPolo
Copy link
Contributor Author

Why did the contract change?

@dhuseby
Copy link
Contributor

dhuseby commented Jan 26, 2026

To normalize it across all tests and to minimize dependencies. This makes it easier to write new tests (e.g. Kademlia) while also simplifying local development/debugging/performance testing. There are many improvements with this new test-plans framework.

@MarcoPolo
Copy link
Contributor Author

This is a cosmetic change of upper vs lower case env vars. All of the original interop tests used lower case env vars. If the argument is to normalize it, it should be normalized to lower case as that is what the existing code and spec defined. I'm not convinced that using upper case env vars makes anything easier.

Can we please change it back? It seems painful to have to require every prior version of this test to include a patch to workaround a design issue that can still be fixed.

@dhuseby
Copy link
Contributor

dhuseby commented Jan 26, 2026

The output format changed to YAML as well. Upper case is convention for environment variables. Has been for decades. It's already changed and you can just land the patch I wrote for you.

@MarcoPolo
Copy link
Contributor Author

The issue is not only with this version, but with all previous versions of this interop test.

Upper case is convention for environment variables. Has been for decades.

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

Environment variable names used by the utilities in the Shell and Utilities volume of POSIX.1-2017 consist solely of uppercase letters, digits, and the ( '_' ) from the characters defined in Portable Character Set and do not begin with a digit. Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. Uppercase and lowercase letters shall retain their unique identities and shall not be folded together. The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities.

Emphasis added. These parameters are part of the application namespace.

But even putting that aside, it would be nice if we didn't break existing applications for no good reason. Upper vs lower is cosmetic issue. JSON vs YAML is also a cosmetic issue (as an aside, JSON is valid YAML syntax, you could parse the json output as YAML just fine).

@dhuseby
Copy link
Contributor

dhuseby commented Jan 26, 2026

I hear your arguments and I understand where you're coming from. Most developers recognize ENVIRONMENT_VARIABLES as such, despite the fine print in the POSIX spec. There's good arguments either way. As for the YAML/JSON output issue, we're minimizing dependencies and wanted to use YAML for the input config files because they allow us to add comments which JSON doesn't. Switching to YAML for everything eliminates an extra dependency. Plus there's a schema that aids in upstream reporting that isn't obvious in the transport tests but becomes apparent in the perf tests.

I wrote you a patch to save you the work. The new framework supports local patches against test applications to decouple the testing from upstreaming changes in implementation repos. The rest of the test applications will be patched in short order since, as you point out, it's a trivial change.

You might think this is an unnecessary and irrelevant change but the upper casing of environment variables brings this test framework in line with common convention and lowers the barrier to entry for developers looking at it for the first time. Also, the consistency across tests will pay off in the long term maintenance and expansion of this framework.

Would you like me to submit the patch as a PR against go-libp2p?

@dhuseby
Copy link
Contributor

dhuseby commented Jan 26, 2026

@MarcoPolo the next spec meeting is on February 10th: https://luma.com/yamxhbko That's where we include time to discuss this repo and the improvements for enforcing the specifications. Attending that is the best way to have influence over the test-plans roadmap.

@acul71 acul71 mentioned this pull request Jan 29, 2026
@acul71
Copy link
Contributor

acul71 commented Jan 29, 2026

@dhuseby @MarcoPolo Take a look at #801 ( Updates / supersedes #788. Same bump-go changes but from my fork so the latest commit (fix: make go new transport spec compliant) is included. )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants