Do not merge, non-draft for workflow testing: ci: fix test caching, use consistent Go version

[anacrolix]

Summary

Overhaul CI workflows to improve caching, reduce redundancy, and speed up test runs.

New composite actions

• setup-erigon: Consolidates Go setup, cleanup-space, mtime normalization, Windows temp redirect, and dependency installation (build-essential, choco make) into a single reusable action.
• restore-build-cache / save-build-cache: Content-hashed Go build cache with per-workflow build tags. Saves on failure so partial results aren't lost.
• save-mod-cache: Separate content-addressed module cache entry.

Caching improvements

• Normalize all file/directory mtimes to a fixed timestamp so Go's test cache keys are stable across CI runs.
• Split cache into restore/save steps (instead of save-always) so partial caches are saved even when tests fail.
• Include Go version and content hashes in cache keys for proper invalidation.
• Per-test-group build cache tags for race tests.
• Shallow submodule clones and selective LFS pulls (only files actually needed by tests).

Workflow restructuring

• Break monolithic ci.yml into separate workflows: test-all-erigon.yml, reproducible-build.yml, sonar.yml.
• Combine Linux/macOS/Windows test jobs into single multi-OS matrix.
• Add workflow_dispatch to workflows that were missing it.
• Remove unused test-win-downloader workflow.

Test output filtering

• Extract grep filter pipeline from Makefile into tools/filter-test-output with proper exit code handling.
• Apply filter to race test and caplin test workflows.

Other

• Derive Go version from go.mod instead of hardcoding.
• Drop GOEXPERIMENT=synctest (standard in Go 1.25).
• Export CGO_CXXFLAGS from Makefile instead of per-command forwarding.
• Split lint (fast + full, for local) and lintci (full only, for CI).
• Document workflow triggering, required checks, merge queues, and debugging in .github/README.md.
• Add skip-uncaching PR label to skip Docker-based test workflows (Hive, Kurtosis) that don't use setup-erigon caching.

[Copilot]

Pull request overview

Refactors CI workflows to improve Go test/build caching and reduce redundant setup across jobs, while reorganizing workflows and test grouping utilities.

Changes:

• Added composite actions for standardized Go setup and explicit restore/save caching of Go build + module caches.
• Split and reorganized CI workflows (tests, race, sonar, reproducible build), adding matrix-based grouping and output filtering.
• Refactored execution test utilities by moving matcher logic into execution/tests/testutil and updating tests accordingly.

Reviewed changes

Copilot reviewed 50 out of 51 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/write-test-groups	New script to partition Go packages into named CI test groups and write group files/JSON.
tools/filter-test-output	New bash filter to reduce noisy go test output while preserving failure exit codes.
sonar-project.properties	Excludes .github/** from Sonar analysis.
node/interfaces	Updates submodule pointer.
execution/tests/transaction_test.go	Switches to testutil.TestMatcher API.
execution/tests/testutil/matcher.go	Adds shared TestMatcher implementation and JSON test file walking/loading.
execution/tests/state_test.go	Migrates to testutil.TestMatcher and changes temp dir handling.
execution/tests/rlp_test.go	Switches to testutil.TestMatcher API.
execution/tests/invalid_receipt_hash_high_mgas_test.go	Switches to testutil.TestMatcher API.
execution/tests/init_test.go	Removes old matcher/JSON helpers; uses testutil.TestMatcher.
execution/tests/eest_prague_calldata/block_test.go	Adds isolated EEST Prague calldata blockchain test package.
execution/tests/eest_osaka_clz/block_test.go	Adds isolated EEST Osaka CLZ blockchain test package.
execution/tests/eest_frontier_scenarios/block_test.go	Adds isolated EEST Frontier scenarios blockchain test package.
execution/tests/eest_cancun_blobs/block_test.go	Adds isolated EEST Cancun blobs blockchain test package.
execution/tests/difficulty_test.go	Switches to testutil.TestMatcher API.
execution/tests/block_test.go	Switches to testutil.TestMatcher and adjusts skip/slow handling.
execution/tests/benchmark_engine_x_test.go	Switches to testutil.TestMatcher and updates parallel control.
cl/spectest/Makefile	Adjusts go test invocation flags for consensus spec tests.
agents.md	Adds workflow guidance note.
TODO	Adds CI/workflow-related TODOs.
Makefile	Exports CGO_CXXFLAGS, refactors test filtering, adds test-group helpers, and updates lint targets.
.gitignore	Ignores generated test group files.
.github/workflows/test-win-downloader.yml	Removes unused workflow.
.github/workflows/test-kurtosis-assertoor.yml	Collapses regular/pectra jobs into a matrix; adds skip label logic.
.github/workflows/test-integration-caplin.yml	Uses setup/cleanup composite actions and filters output.
.github/workflows/test-hive.yml	Adds skip label gate for “uncaching” workflows.
.github/workflows/test-all-erigon.yml	Reworks test workflow to use setup/cleanup composite actions and matrix.
.github/workflows/test-all-erigon-race.yml	Builds dynamic test-group matrix using write-test-groups; uses composite caching.
.github/workflows/sonar.yml	New dedicated Sonar workflow using composite setup/cleanup.
.github/workflows/reusable-release-build-debian-pkg.yml	Makes apt output quieter.
.github/workflows/reproducible-build.yml	New reproducible build workflow across OS matrix.
.github/workflows/qa-rpc-performance-tests.yml	Uses partial clone filter (blob:none).
.github/workflows/qa-rpc-performance-comparison-tests.yml	Uses partial clone filter (blob:none).
.github/workflows/manifest.yml	Derives Go version from go.mod.
.github/workflows/lint.yml	Reworks triggers, caching, and uses composite setup/cleanup.
.github/workflows/docker-tags.yml	Uses partial clone filter (blob:none).
.github/workflows/ci.yml	Removes monolithic legacy CI workflow.
.github/test-caching	Adds helper script to rerun workflows to warm caches on PRs.
.github/actions/setup-erigon/action.yml	New composite action: checkout strategy, mtime normalization, Go setup, mod/build cache restore.
.github/actions/save-mod-cache/action.yml	New composite action to hash and save Go module download cache content-addressably.
.github/actions/save-build-cache/action.yml	New composite action to save Go build cache with workflow/job/scoped keys.
.github/actions/restore-build-cache/action.yml	New composite action to restore Go build cache with tiered fallbacks.
.github/actions/cleanup-erigon/action.yml	New composite action to save caches and warn about uncached tests.
.github/README.md	Adds documentation for required checks, merge queues, triggers, and caching strategy.
.github/PROPOSAL.md	Adds CI/workflow proposal notes.
.gitattributes	Forces LF for go.sum to stabilize hashFiles('go.sum') cache keys across OSes.
.claude/settings.json	Expands allowed tools/actions for Claude config.
.claude/rules/lint.md	Adds lint rules doc.
.claude/rules/conventions.md	Adds commit message conventions doc.
.claude/rules/build-and-test.md	Adds build/test instructions doc.
.claude/rules/architecture.md	Adds architecture overview doc.

Comments suppressed due to low confidence (6)

execution/tests/testutil/matcher.go:1

• readJSONFile passes &value to json.Unmarshal, but callers already pass a pointer (e.g. m.Addr().Interface()). Unmarshaling into *interface{} will populate the interface variable rather than the pointed-to map, leaving the map empty. Pass value directly to json.Unmarshal so it fills the provided pointer target.
  execution/tests/testutil/matcher.go:1
• CheckFailureWithName ignores its name parameter and matches patterns against t.Name() instead. This breaks callers that want to evaluate expected failures for a different logical test name (e.g., aggregated names or file/key names). Use name for the match instead of t.Name().
  execution/tests/testutil/matcher.go:1
• If os.Stat(dir) returns an error other than IsNotExist, dirinfo will be nil and dirinfo.IsDir() will panic. Handle err != nil first (skip only on not-exist; otherwise fail the test), and only call IsDir() when err == nil.
  tools/filter-test-output:1
• The script only checks PIPESTATUS[1] (the second grep) and ignores failures from the first grep; it also documents SIGPIPE (141) but currently exits 141, which can fail callers that intentionally truncate output. Consider: (1) validating both grep exit codes and returning 2 if either grep errors, and (2) treating 141 like non-fatal for a filter (similar to exit 1 handling).
  cl/spectest/Makefile:1
• These targets previously passed -tags=spectest but no longer do. If any spec tests are gated behind //go:build spectest (or similar), this will silently skip compiling/running them. Either restore -tags=spectest here or ensure the tag is provided via GOFLAGS/environment consistently.
  agents.md:1
• Use the standard hyphenation 'cross-platform' in documentation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

sha256sum is not available by default on macOS runners (they typically provide shasum -a 256). Since this action is used by macOS workflows, this step can fail. Use a portable hashing fallback (e.g., prefer sha256sum else shasum -a 256) consistently anywhere hashing is done in composite actions.

Suggested change

	hash=$(git ls-tree -r HEAD | awk '$2 == "commit" {print $3}' | sort | sha256sum | cut -d' ' -f1)
	if command -v sha256sum >/dev/null 2>&1; then
	hash_cmd="sha256sum"
	else
	hash_cmd="shasum -a 256"
	fi
	hash=$(git ls-tree -r HEAD | awk '$2 == "commit" {print $3}' | sort | ${hash_cmd} | cut -d' ' -f1)

[Copilot]

Same portability issue as elsewhere: sha256sum is not guaranteed on macOS/Windows bash environments. Since this cache action may run on macOS jobs, replace sha256sum usage with a portable hash helper (fallback to shasum -a 256).

Suggested change

	hash=$(cd '${{ inputs.modcache-path }}/cache/download' && find . -type d -name '*@*' | sort | sha256sum | cut -d' ' -f1)
	portable_sha256() {
	if command -v sha256sum >/dev/null 2>&1; then
	sha256sum "$@"
	elif command -v shasum >/dev/null 2>&1; then
	shasum -a 256 "$@"
	else
	echo "Error: no SHA-256 checksum utility found (sha256sum or shasum)" >&2
	return 1
	fi
	}
	hash=$(cd '${{ inputs.modcache-path }}/cache/download' && find . -type d -name '*@*' | sort | portable_sha256 | cut -d' ' -f1)

[Copilot]

This downloads and executes a script directly from raw.githubusercontent.com without pinning to a commit SHA or verifying a checksum. To reduce supply-chain risk, pin the URL to an immutable commit (or vendor the script into the repo) and/or verify an expected checksum before executing.

Suggested change

	curl -sL https://raw.githubusercontent.com/MestreLion/git-tools/main/git-restore-mtime -o "$RUNNER_TEMP/git-restore-mtime"
	# Pin git-restore-mtime to a specific commit to avoid running mutable code from "main"
	GIT_RESTORE_MTIME_URL="https://raw.githubusercontent.com/MestreLion/git-tools/8dfb1b78c57d20593d3536bc3a1e9bdfb1ef82b4/git-restore-mtime"
	curl -sL "$GIT_RESTORE_MTIME_URL" -o "$RUNNER_TEMP/git-restore-mtime"

[Copilot]

This doc has a few typos ('teh', 'tho', 'accomodate') that reduce readability/searchability. Please correct them.

[Copilot]

This doc has a few typos ('teh', 'tho', 'accomodate') that reduce readability/searchability. Please correct them.

[anacrolix]

There's about 3 separate PRs here, but I have most CI jobs down to about 3-4 mins. It will be broken up and submitted in separate PRs.