Skip to content

Version-aware test gate for cross-version runs (SupportedFromVersion) #5121

Description

@islamaliev

Problem

The cross-version multiplier replays existing P2P tests with one node running an
older release. Some tests assert behavior that only exists in newer versions, so
they cannot pass against an older node and must be skipped.

Right now the only way to skip them is MultiplierExcludes, which names the
multiplier:

MultiplierExcludes: []string{multiplier.CrossVersionOldSource},

That works while there is one version pair, but it says the wrong thing. The test
is not incompatible with the multiplier, it is incompatible with the older
version. As the LTS window grows to the last three minors, every version pair
becomes another multiplier name, and each excluded test has to list all of them.
Miss one and the test runs against a version that cannot support it.

Proposal

Let the test state the version it needs, once:

SupportedFromVersion: "v1.1.0",

The multiplier implements ShouldSkip (already part of testo's
ActionAwareSkipper) and compares that value against the version it is targeting.
Adding a new version pair then needs no change to any excluded test.

Notes

  • Deliberately not done as part of the first multiplier. v1.0.0 is currently the
    only post-1.0 release, so a version comparison would have exactly one value to
    compare against, and MultiplierExcludes is the pattern the codebase already
    uses (signed_docs).
  • Worth doing when the second supported version pair appears, whichever comes
    first with the LTS window widening.
  • Needs a small version comparison helper. Release tags are semver, so this
    should stay simple.
  • MultiplierExcludes should keep working. This is for the version-specific
    case, not a replacement.

Part of #5066.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingRelated to any test or testing suite

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions