fix(test-suite): freeze orchestrated fhevm baseline

[Eikix]

Summary

Freeze orchestrated fhevm e2e baselines from the PR base SHA instead of resolving floating latest-main during the reusable workflow boot path.

Closes zama-ai/fhevm-internal#1263.

Why

Merge-queue / orchestrated e2e was still resolving latest-main live when no lock artifact was provided. That made the boot path depend on live GitHub and GHCR state and caused flaky failures from:

• package metadata scope/auth issues
• GitHub API / fetch failures and rate limiting
• image sets where latest-main points at a commit whose full orchestrated stack is not available yet

The goal of this PR is narrow:

• resolve one baseline lock from github.event.pull_request.base.sha
• reuse that frozen lock across dry-run and boot
• keep head-image overrides only for components rebuilt by the PR
• let existing and future SHA/tag-based compat rules reason from the resolved base SHA

Change

• add a resolve-baseline job to .github/workflows/test-suite-orchestrate-e2e-tests.yml
• resolve the baseline with ./fhevm-cli resolve --target sha --sha "$BASE_SHA" --reset
• upload the produced lock as an artifact and thread lock-artifact-name into the reusable e2e workflow
• require non-manual orchestrated runs to use that frozen lock instead of falling back to latest-main
• keep workflow_dispatch orchestrated runs able to resolve latest-main manually when no artifact is supplied
• simplify sha resolution so it constructs the repo-owned image tags from the requested SHA without branch/ref/release logic

Testing

• cd test-suite/fhevm && bun test src/resolve.test.ts src/cli.test.ts
• cd test-suite/fhevm && bun run check
• ruby -e 'require "yaml"; YAML.load_file(ARGV.fetch(0)); puts "ok"' .github/workflows/test-suite-orchestrate-e2e-tests.yml
• ruby -e 'require "yaml"; YAML.load_file(ARGV.fetch(0)); puts "ok"' .github/workflows/test-suite-e2e-tests.yml
• cd test-suite/fhevm && bun run src/cli.ts resolve --target sha --sha $(git rev-parse --short=7 origin/main) --reset
• PR orchestrated validation run passed before this simplification: https://github.com/zama-ai/fhevm/actions/runs/24889356352

[Eikix]

@claude /pr-review

[claude]

Claude finished @Eikix's request. — View run

[claude]

[AUTOMATED]

Code Review

Summary

This PR freezes orchestrated fhevm e2e baselines by resolving from the PR base SHA instead of floating latest-main. Key changes:

• New resolve-baseline job in the orchestration workflow that resolves and uploads a lock artifact from the PR base SHA
• New git.ts module replacing GitHub API calls with local git history for sha resolution
• --ref parameter support enabling branch-aware baseline resolution (including release branches)
• Differentiated build-result handling for relayer/test-suite (empty on skip vs base-tag on skip) to let the frozen lock provide baseline versions
• Guard steps ensuring orchestrated non-manual runs require a frozen lock artifact

Review Findings

Bugs & Security: No issues found. Reviewed for null/undefined dereferences, logic errors, command injection, and path traversal. The SAFE_REF regex in git.ts and requireRef validation provide adequate input sanitization for git commands.

Guideline Compliance: No violations. The code follows established project patterns — error types (GitHubApiError, PreflightError), CLI argument parsing via parseUpInput, and test structure are all consistent with the existing codebase.

Error Handling: Adequate. All error paths throw with descriptive messages. The catch → re-throw as GitHubApiError pattern matches existing code in the same files.

Nit (non-blocking)

Unused ref parameter in applyReleaseBaselineDefaults:

export const applyReleaseBaselineDefaults = (
  bundle: VersionBundle, defaults: Record<string, string>, ref: string
): VersionBundle => {

The ref parameter is accepted but never used in the function body. Consider prefixing with _ or removing it if it's not needed for future use.

Notes

• The removal of the missingRepoPackages check from the sha target path is intentional — the frozen lock artifact now ensures image availability, and assertCommitOnRef + assertSupportedShaBundle provide the necessary validation.
• The floor >= 0 guard in assertSupportedShaBundle correctly handles release branches where anchor commits (SIMPLE_ACL_MIN_SHA, SHA_RUNTIME_COMPAT_MIN_SHA) may not exist in the branch history — skipping the check rather than incorrectly failing.
• Good test coverage: new tests for --ref validation, release baseline defaults, stale sha lock rejection, and branch history without main cutover anchors.

No blocking issues found. ✅

[claude]

The review has been posted successfully. Here's a summary of what was done:

PR Review Complete

PR #2278: fix(test-suite): freeze orchestrated fhevm baseline

Review Process

• ✅ Pre-review validation (PR is open and non-trivial)
• ✅ Context gathering (PR details, guidelines discovery)
• ✅ Bug Hunter Agent 1 (opus) — No bugs found
• ✅ Bug Hunter Agent 2 (opus) — No bugs found
• ✅ Guideline Compliance Agent (sonnet) — No violations
• ✅ Error Handling Auditor (sonnet) — No critical/high issues

Result

No blocking issues found. The PR is well-structured with:

• Proper input validation and sanitization for git commands
• Consistent error handling following existing patterns
• Good test coverage for new functionality
• Clear separation between orchestrated/manual/non-orchestrated flows

One non-blocking nit was noted: an unused ref parameter in applyReleaseBaselineDefaults.