ELF loader conformance vectors and svm-test-harness integration

[cmoyes-jump]

TL;DR: Introduce a minimal, developer-friendly conformance harness focused on ELF loader behavior, wire a small curated set of ELF test vectors, and stand up a non-blocking CI job. These conformance tests will help provide a shared definition of correctness across teams and projects.

Crucially, these tests make regressions visible and ensure refactorings and releases do not introduce unexpected behavior, breaking consensus for other validators. We currently maintain and utilize these tests today in our everyday workflows. To minimize disruptions, we will keep the port light, and get early feedback before broadening coverage.

Definitions

• Test vectors: Small, reproducible inputs with expected outcomes that encode correctness for specific behaviors. Test vectors are generated directly from real ledgers and fuzz runs, and many of them correspond to real bugs we have encountered in the past.
• Fuzz harnesses: Lightweight execution entrypoints that run vectors deterministically in an isolated / reproducible environment, making debugging simpler.

What this is and how to use it

What is "conformance testing"?

• Think of it as golden files for the runtime: a set of small inputs ("test vectors") with their expected results. Developers can re-run them any time (locally or in CI) to help establish that changes made do not break consensus and to quickly reproduce and triage detected behavioral changes.

What is a "test vector"?

• A tiny file generated from real-world ledgers and test runs that contains:
  • the input (an ELF binary with a few flags)
  • the effects (the expected output: rodata bytes, text layout, entry point, call destinations, or an error code)
• Formats used: protobuf files for existing fixture files (.fix) that encode the same info.

Why start with the ELF loader?

• It's the simplest meaningful "atom" we can validate end‑to‑end without appreciable bank/runtime complexity, while remaining high value.
• Many validator mismatch investigations begin with answering the question "was this ELF loaded consistently?"

Where do the vectors live?

• Initially in the monorepo: For the SVM harness: svm-test-harness/test-vectors/.
• The initial set will be only ~2 MB in size.
• You can drop additional .fix files into that directory to extend coverage.
• Later we can consider a separate test vectors depot (e.g. using git LFS or an S3 bucket).

How do I run them locally?

• From the repo root:
  • Run all ELF vectors:
    • cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader
  • Run one by name:
    • cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader --case <name>
  • Obtain additional logs:
    • cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader --verbose
• The test runner:
  • Prints each case as it runs ("PASS"/"FAIL").
  • Stops on first failure (so you can debug quickly).
  • Prints a final summary (total / passed / failed).

What if something fails? How do I debug it?

• The runner prints:
  • Which fixture failed (full path)
  • A human-readable "expected vs. actual" effects diff
• Typical next steps:
  • Rerun the specific individually failing test cases locally with --verbose.
  • Check if fixtures are outdated: cargo run -p solana-svm-test-harness --features fuzz --bin update_fixtures -- --suite elf_loader --input-dir svm-test-harness/test-vectors/elf_loader
  • Update outdated fixtures (if behavior change is intentional): Add --update flag to the above command.
  • If this is a deliberate behavior change, rework the PR to follow the expected practices, including writing a SIMD and gating with a feature, so that conformance with other validators is not broken.

Do I need to install new tools?

• Today, CI already includes an install-proto.sh in the monorepo that we can reuse.
• Protobuf codegen is already committed for this tiny subset, so a plain cargo build should work.
• If you do need to regenerate test fixtures, use protoc from PATH, or set PROTOC_EXECUTABLE=/full/path/to/protoc.
• CI jobs can be run locally with system packages as well.

What am I looking at in a fixture?

• Input: ELF bytes and feature flags for the loader.
• Output (effects):
  • rodata/rodata_sz
  • text_cnt/text_off
  • entry_pc
  • calldests (function destinations discovered)
  • error (0 for success, non‑zero for specific loader errors)

Proposal

• Introduce ELF loader conformance smoke tests.
• Provide an initial ELF loader test harness in svm-test-harness.
• Keep CI non-blocking at first.
• Gate CI later once harnesses are stable and developers gain familiarity.

Proposed changes

• Add and wire the following in-repo within Agave's svm-test-harness:
  • Crate scaffolding: Cargo.toml, build.rs
  • Public entrypoints: src/lib.rs
  • ELF logic: src/elf_loader.rs
  • Utilities used by ELF: src/utils/{mod.rs,err_map.rs}
  • Initial test vectors: test-vectors/elf_loader/**
  • Schema/types: protosol (protobuf definitions and prost codegen within external crate)
• Place an initial tiny, curated set of ELF vectors and wire a script to run them locally and in CI.

Test vectors (ELF)

• Start with a very small subset (20 test cases / ~2.5 MB) that:
  • Exercise happy-path loads, edge case quirks, and representative error cases.
  • Are easy to run locally and explain in review.
• Format: Protobuf (v1)
• We will include a per-suite manifest so tests can be iterated or annotated without code changes.

Developer experience

We want a frictionless local experience while keeping the port light and the repo size small.

• Keep the port lightweight
• Minimize disruptions to developers' workflows.
• Avoid checking in large third-party trees or heavy tooling.
• Rely on system packages in CI.
• Aim to strictly improve and not regress productivity.
• Use environment variables (PROTOC_EXECUTABLE) to point codegen to system tools.
• Prefer a minimal footprint until we get feedback and know what sticks.

[RFC] Options for local developer setup

1. Require local protoc, with crisp instructions and pre-flight checks.

• Pros: Small repo, simplest maintenance, consistent with CI.
• Con: Devs must install new binaries (Linux/macOS/Windows instructions needed).
• Con: Deterministic reproducibility suffers if local environments differ.

2. Vendor protoc binaries for major OSes under tools/.

• Pros: Zero-install for most devs.
• Cons: Increases repo size/churn, platform matrix upkeep, multi-architecture support burden.

3. Commit the code-generated sources for the initial test vectors subset. Only CI regenerates.

• Pros: Out-of-the-box builds without new binaries. Still keep original test vectors as the source of truth.
• Cons: Need discipline around regeneration and review of generated diffs.

Recommendation

• Ship with Option 1 as the default policy (documented install with preflight steps).
• Also adopt Option 3 for the small ELF subset to guarantee cargo build works from a clean checkout without additional installation steps.
• Re-litigate a full decision (including Option 2) later.

Documentation to add

• Quickstart: Run vectors locally and debugging workflow guide
• Install notes: OS-specific package manager snippets for protoc
• Troubleshooting: PATH checks, PROTOC_EXECUTABLE override, common errors

CI plan (non-blocking initially)

• GitHub Actions job:
  • Build svm-test-harness (already done).
  • Run cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader against the curated test vectors subset.
  • Surface actionable conformance test suite results in the UI with a single pass/fail determination.
  • Use system packages in CI runners for protoc (set PROTOC_EXECUTABLE).
• Keep the job non-blocking until fully landed and stable, then flip to merge blocking.

Debugging workflow

• Test runner: cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors
  • Examples:
    • Run all ELF vectors: cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader
    • Run a single case: cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader --case <name> --verbose
• Update outdated fixtures: cargo run -p solana-svm-test-harness --features fuzz --bin update_fixtures
  • Check which fixtures are outdated: cargo run -p solana-svm-test-harness --features fuzz --bin update_fixtures -- --suite elf_loader --input-dir svm-test-harness/test-vectors/elf_loader
  • Apply updates: Add --update flag to update fixtures to match current behavior
• Generate new fixtures: cargo run -p solana-svm-test-harness --features fuzz --bin gen_fixture
  • From invalid ELF (smoke test): cargo run -p solana-svm-test-harness --features fuzz --bin gen_fixture -- --suite elf_loader --name test.pb
  • From real ELF file: cargo run -p solana-svm-test-harness --features fuzz --bin gen_fixture -- --suite elf_loader --elf path/to/program.so --name test.pb
• Output makes it easy to locate the exact vector file and expected vs. actual behavior for quick triaging.
• Provide a short "How to debug mismatches?" guide with handy tips.

Acceptance criteria

• From a clean checkout:
  • cargo build succeeds for svm-test-harness without additional installs
  • OR: When binaries are needed, developers get clear, one-command install instructions and a crisp self-documenting, descriptive error messages.
• A minimal ELF suite (20 vectors) runs locally and in CI.
• Non merge-blocking CI job publishes pass/fail/skipped with clear summaries.
• Documentation provided: Quickstart, install, and debugging guides.
• Clear decision on where test vectors will live in the long term.

Request for feedback

• Binary handling approach for local dev (choose options 1, 2, or 3)?
• Preferred location for additional test vectors?
• Any constraints on adding a non-blocking CI job now, with intent to gate later?

Next steps (upon approval)

1. Land the ELF loader test harness into svm-test-harness.
2. Add the curated ELF vectors (20 representative test cases).
3. Add the Rust test tooling (run_test_vectors, gen_fixture, update_fixtures).
4. Add the CI job (non-blocking).
5. Write and contribute Quickstart / Install / Debugging docs.

Available Tools

• run_test_vectors - Execute test vectors for a suite

  • Supports: --suite <name>, --case <fixture>, --verbose
  • Usage: cargo run -p solana-svm-test-harness --features fuzz --bin run_test_vectors -- --suite elf_loader

• gen_fixture - Generate new test fixtures

  • Supports: --suite <name>, --elf <path>, --deploy-checks, --out-dir <dir>, --name <file>
  • Usage: cargo run -p solana-svm-test-harness --features fuzz --bin gen_fixture -- --suite elf_loader --elf my_program.so

• update_fixtures - Update outdated fixtures after behavior changes

  • Supports: --suite <name>, --input-dir <dir>, --update, --verbose
  • Usage: cargo run -p solana-svm-test-harness --features fuzz --bin update_fixtures -- --suite elf_loader --input-dir svm-test-harness/test-vectors/elf_loader --update

[t-nelson]

@yihau does this sounds sane to integrate into ci? i know the corpus downloads have screwed us in when we tried this in the past. i think i'm pretty opposed to committing it directly

cc/ @steviez

[yihau]

looks like the test vectors are ~ 2.5 mb, so we should be fine including them. I don't have a strong preference at current status. my bigger concern is whether we might keep adding more test vectors in the future and how we ensure they remain valid.

[t-nelson]

my bigger concern is whether we might keep adding more test vectors in the future and how we ensure they remain valid

same and i'd prefer that our git repo not be the collateral damage in the meantime. is there an off the shelf solution that we could use in a separate ci step to sync the corpus, then the actual tests we can just point to that directory? ideally we can cache the fetch on each ci agent to avoid constantly redownloading

@cmoyes-jump is there anything agave version dependent about the corpus format? i vaguely recall a previous version of this needing recompiled from time to time

[cmoyes-jump]

we have some options:

• dedicated git repo
• git LFS
• storage buckets (e.g. S3)
• artifactory, nexus

for our own internal use, we have our own dedicated repo that we cache on the CI worker to avoid constant redownloading

which of these options are you leaning towards?

is there anything agave version dependent about the corpus format?

no, the corpus format is dependent on the protosol protocol buffer version used