Should we do a test matrix for performance testing?

[dhuseby]

I was thinking about the performance testing today and I don't think it makes sense to do a testing matrix (i.e. rust x py, py x dotnet, etc). I think it only makes sense to have rust x rust, py x py, etc. Why? Because I think this isolates each implementation from the others and we get true relative comparisons between each impl.

Think of it this way, as a hypothetical let us assume the Rust QUIC implementation is slow. We know it's slow because when Rust QUIC dials Rust QUIC, the throughput is 1 Gbps. Also let us assume that Python QUIC is much faster. We know that because when Python QUIC dials Python QUIC, the throughput is 5 Gbps. What good is it to have Python QUIC dial Rust QUIC and have the number be 1 Gbps? It doesn't tell us anything we don't already know (i.e. Rust QUIC is slower).

I think the only reason we would want to do a full test matrix is to see if there are any combinations that are outliers.

If I refactor the test to be Rust x Rust, Python x Python, Dotnet x Dotnet, then we'll get a good sense of which implementations and which transport + secure channel + muxer is most efficient.

What do you think?

@seetadev @acul71 @MarcoPolo @paschal533 @parth-soni07

[paschal533]

Hi @dhuseby, I think same-impl testing is the right default... it gives us clean baselines without cross-impl noise muddying the numbers. That said, I'd keep the full matrix as an optional mode we can run on-demand. Cross-impl outliers are rare but when they show up they usually point to a protocol-level bug (like a bad muxer negotiation path) that same-impl tests would never catch.
So maybe: same-impl runs on every CI trigger, full matrix runs nightly or on-demand? That way we get fast feedback on regressions without losing the ability to spot weird cross-impl interactions when we need to.

[dhuseby]

@paschal533 what if I add a --run-full-matrix option to the perf run.sh that, when added, it does a full matrix perf test. Otherwise, it just does single impl tests. You have a good point that we should be able to do a full matrix pass if we're looking for cross-impl outliers.

[paschal533]

That works perfectly. A --run-full-matrix flag keeps the default fast and focused while still giving us the escape hatch when we need it. Happy to help implement that once you've got the scaffolding in place

[acul71]

@paschal533 what if I add a --run-full-matrix option to the perf run.sh that, when added, it does a full matrix perf test. Otherwise, it just does single impl tests. You have a good point that we should be able to do a full matrix pass if we're looking for cross-impl outliers.

Yes, a --run-full-matrix option would be best. Also to see if some file transfers fails because of protocol mismatch

[dhuseby]

@paschal533 create a fork of unified-testing and add me as a collaborator that can create branches. I'll pull your fork, add a branch and do the scaffolding and push it to your fork. Then you can add the rest and when we're done, submit a PR upstream.

[paschal533]

Done... just added you as a collaborator on my fork. Looking forward to building on top of the scaffolding once it's ready.