DRAFT: rework testing code#695
Draft
MDr164 wants to merge 11 commits intotheupdateframework:masterfrom
Draft
Conversation
MDr164
force-pushed
the
rework-testing
branch
from
abaeb88 to
201ebbb
Compare
Contributor
Author
|
We also want to drop the use of static assets like the root files and keys from repository_data |
Merged
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Remove TempDirManager (callers use t.TempDir()), RunTableTest/TestCase
generic wrappers, and the BenchmarkOperation abstraction. Replace with
a simpler flat set of functions that each call t.Helper() directly.
Key changes:
- Rename StripWhitespaces → StripWhitespace (singular)
- Rename NoError → AssertNoError, ErrorContains → AssertErrorContains
- Migrate FuzzDataGenerator from deprecated math/rand to math/rand/v2
with PCG source; change NewFuzzDataGenerator(seed int64) to
NewFuzzDataGenerator(seed1, seed2 uint64)
- Add Build{Root,Targets,Snapshot,Timestamp}JSON builder functions that
do not require *testing.T, eliminating the &testing.T{} anti-pattern
in fuzz seed corpus setup
- Keep CreateTest*JSON(t) as thin t.Helper wrappers for backward
compatibility
- Replace containsString/findSubstring reimplementation in updater.go
with strings.Contains from the standard library
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Co-authored-by: Claude <noreply@anthropic.com>
Callers of the rewritten helpers package required updates:
marshal_test.go:
- helpers.NoError → helpers.AssertNoError (3 call sites)
- SuccinctRoles BitLength 256 → 8 (256 is out-of-range, valid max is 32)
- Null_values wantErr false → true (null HexBytes fails to unmarshal)
metadata_fuzz_test.go:
- NewFuzzDataGenerator(time.Now().UnixNano()) →
NewFuzzDataGenerator(uint64(time.Now().UnixNano()), 0) (new two-arg API)
- helpers.CreateTestRootJSON(&testing.T{}) → helpers.BuildRootJSON()
(eliminates the &testing.T{} anti-pattern in f.Add seed setup)
metadata_bench_test.go:
- All for i := 0; i < b.N; i++ loops → for b.Loop() (Go 1.24+ API)
- math/rand → math/rand/v2 for generateRandomString helper
metadata_table_test.go:
- Replace TempDirManager with t.TempDir() (4 call sites)
config/config_table_test.go:
- Full rewrite: old tests had wrong assumptions about EnsurePathsExist
(uses os.MkdirAll — it creates dirs, not validates existence) and
about config.New (sets RemoteMetadataURL, not LocalMetadataDir)
- New table tests: TestUpdaterConfigNew, TestUpdaterConfigDefaults,
TestEnsurePathsExistTable, TestUpdaterConfigCopy,
TestUpdaterConfigCustomFetcher
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Co-authored-by: Claude <noreply@anthropic.com>
Add the client-under-test binary required by the tuf-conformance test suite (https://github.com/theupdateframework/tuf-conformance). cmd/tuf-conformance-client/main.go: Implements the three-command CLI protocol from CLIENT-CLI.md: - init <trusted-root> copies root.json into --metadata-dir; no network requests are made - refresh runs updater.Refresh() to update top-level metadata from --metadata-url - download refreshes metadata then downloads and verifies the artifact named by --target-name, caching if already present All diagnostic output goes to stderr; the process exits 0 on success and 1 on any error, satisfying the test suite's expectations. .github/workflows/conformance.yml: Runs the tuf-conformance GitHub Action on push, pull_request, and a weekly schedule (Wednesday 06:30 UTC) so the public conformance report stays current. Makefile: Add 'build-conformance-client' and 'conformance' targets for local development runs using a locally-installed tuf-conformance suite. Signed-off-by: Marvin Drees <marvin.drees@9elements.com> Co-authored-by: Claude <noreply@anthropic.com>
…ble tests Add infrastructure to make updater integration tests easier to write and read, then exercise it with a table-driven test file. simulator/builder.go: SimulatorBuilder is a fluent API for configuring a RepositorySimulator before a test starts. It covers the most common dimensions: consistent-snapshot flag, target files, delegations, succinct roles, expired/unsigned roles, version overrides, and root rotations. simulator/test_repository.go: TestRepository encapsulates all per-test state (simulator, metadata dir, targets dir, root bytes) in one struct. Temp dirs are created with t.TempDir() so they are cleaned up automatically; Cleanup() is a kept-for-compatibility no-op. Convenience methods cover the mutations and assertions most updater tests need. metadata/updater/updater_table_test.go: Table-driven tests for the TUF client workflow: root updates (version rollback, non-consecutive version, expired intermediate root), timestamp and snapshot unsigned/expired/version-mismatch scenarios, targets unsigned/expired/version-mismatch, fast-forward recovery for all three roles, version rollback detection, consistent-snapshot fetch sequence, and hash-mismatch detection. internal/testutils/README.md: Rewritten to document the current API (AssertNoError, AssertErrorContains, Build*JSON builders, FuzzDataGenerator, TestRepository, SimulatorBuilder) and remove references to removed types (TempDirManager, NoError, AssertEqual, RunTableTest). Signed-off-by: Marvin Drees <marvin.drees@9elements.com> Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft should provide an overview and ground for discussion in order to tackle the open TODO items related to testing. This includes proper table structured testing for every package, fuzzing and benchmarks where applicable and lastly aiming for better coverage and more test cases in general.
The general plan to address the current test situation:
Each of these will be split up into their own PR to not bloat the scope of this one. This draft will eventually be closed once all above items are addressed.