test_snap_versions lint failure due to snapshot key ordering mismatch (and stale snapshot persistence) #3969
Description
Description of the bug
Description
I have encountered two related behaviors regarding nf-test snapshots and nf-core modules lint that lead to false failures and stale data management issues.
Bug 1: Snapshot Ordering Mismatch causes Lint Failure
When a module has multiple tests (e.g., a script test and a stub test), the order in which entries appear in the *.nf.test.snap file can drift from the order in main.nf.test. This causes nf-core modules lint to report missing versions, even when they exist.
Steps to Reproduce:
Create a module with two tests in main.nf.test:
Test A: A "real" script test (initially configured without snapshotting output/versions).
Test B: A stub test (configured with assertions/snapshots).
Run nf-test test.
Result: main.nf.test.snap is created containing only Test B data.
Modify Test A to assert/snapshot versions.yml.
Run nf-test test again.
Result: Test A data is appended to the bottom of the existing main.nf.test.snap.
Current File State:
main.nf.test: Order is [Test A, Test B].
main.nf.test.snap: Order is [Test B, Test A].
Run nf-core modules lint.
Observed Behavior:
The linter fails with the error:
test_snap_versions: versions not found in snapshot
This occurs because the linter appears to expect the snapshot keys to appear in a specific order or struggles to parse the versions when the script test appears after the stub test in the snapshot file.
Expected Behavior:
The linter should be able to identify the versions block regardless of its position in the snapshot file relative to other tests, OR nf-test should respect the source file order when updating snapshots.
Bug 2: Stale Snapshot Persistence (Silent Passing)
Additionally, I observed potentially suboptimal behavior regarding outdated version files.
Observation:
If a snapshot previously contained a versions file check, and the test is subsequently modified to remove that check (or the file is no longer generated), the old entry remains in the *.snap file. nf-test passes successfully because it ignores the extra data in the snapshot, effectively skipping verification for that section.
Resolution:
I had to manually delete the .snap file and regenerate it to clear the stale version entries.
Command used and terminal output
System information
nf-core, version 3.5.1
nf-test 0.9.2