Skip to content

README.md

Latest commit

 

History

History
414 lines (329 loc) · 17.8 KB

README.md

File metadata and controls

414 lines (329 loc) · 17.8 KB
Raw

Rust E2E Test Infrastructure

Purpose

This is a mostly Claude-generated document describing the Rust end-to-end test framework. It reflects the implementation that lives in crates/e2e-tests/ and is intended as a reference for engineers writing new tests.

This is a living document — the earlier draft (#2446) set the direction; the sections below have been revised to match what was actually built.

Architecture Overview

An E2E test runs real mpc-node OS processes against a real neard sandbox with a freshly deployed MPC contract. The crate exposes four components that cooperate:

graph TD
    Cluster[MpcCluster] -->|owns| Sandbox[NearSandbox]
    Cluster -->|owns| Blockchain[NearBlockchain]
    Cluster -->|owns N| Node[MpcNode / MpcNodeSetup]
    Cluster -->|owns| Ports[TestPorts]

    Sandbox --- S1(neard process + genesis + boot node info)
    Blockchain --- B1(near-kit RPC client)
    Blockchain --- B2(Account creation, contract deploy, view/call)
    Node --- N1(Spawn mpc-node binary)
    Node --- N2(Generate secrets.json + start_config.toml)
    Node --- N3(Scrape /metrics, toggle block ingestion)
    Ports --- P1(Deterministic per-test port ranges)
Loading

Key design decisions

  1. Real neard sandbox, real mpc-node binaries. Each test starts a near-sandbox process (via the near-sandbox crate, which downloads a real neard binary) and spawns N real mpc-node processes built from this repo. The mpc-node binary uses its built-in NEAR indexer, which in turn runs its own small neard peered with the sandbox over P2P — i.e. we exercise the production code path end-to-end, including config parsing, P2P networking, the indexer, and Prometheus metrics.

  2. RPC-only contract interaction. Tests never touch the sandbox process directly; everything happens through a near-kit RPC client. The NearBlockchain and DeployedContract wrappers accept any RPC URL, so the same test code could in principle run against a deployed testnet (though this isn't exercised in CI yet).

  3. Deterministic ports per test. cargo nextest runs each test in its own process, so tests execute concurrently by default. Each test declares a unique port_seed; TestPorts::e2e_tests maps that seed to a non-overlapping range of ports for the sandbox, the mpc-node web/P2P/pprof endpoints, and the per-node neard instances. Seeds are centralised as constants in tests/common.rs to avoid collisions.

  4. Support for mixed node versions. MpcClusterConfig::binary_paths takes a vec of paths; if it has a single entry, all nodes use it, otherwise each node uses the corresponding entry. This enables compatibility tests across mpc-node versions (not yet in routine use, but wired up).

  5. Separate "operator" keys. Each node account has two full-access keys on chain: one used by the mpc-node process itself for its NEAR transactions (node_keys) and a second key held by the test harness (operator_keys) for casting votes. Disjoint key sets keep the node's nonce sequence clean and let the harness issue contract calls without racing the node.

  6. Crate location. The framework lives at crates/e2e-tests/. Its src/ exposes the components as library APIs; actual tests live in tests/ and are declared via the [[test]] name = "e2e" entry in Cargo.toml, which compiles them as submodules of tests/e2e.rs (one binary, one neard download, parallel test execution via nextest).

Components

Note: The snippets below capture the shape of each component, not exhaustive signatures. Read the source for the authoritative API.

1. NearSandbox — neard process wrapper

Starts a near-sandbox binary at a configurable version, on ports assigned by TestPorts. Exposes everything an mpc-node indexer needs to peer with it: genesis path, boot-node string (<pubkey>@127.0.0.1:<port>), and chain ID.

pub struct NearSandbox { /* wraps near_sandbox::Sandbox */ }

impl NearSandbox {
    pub async fn start(ports: &TestPorts, version: &str) -> anyhow::Result<Self>;
    pub fn rpc_url(&self) -> String;
    pub fn genesis_path(&self) -> PathBuf;
    pub fn boot_nodes(&self) -> anyhow::Result<String>;
    pub fn chain_id(&self) -> anyhow::Result<String>;
}

The inner Sandbox kills the process and deletes its temp directory on drop.

2. NearBlockchain — RPC client

Wraps a near_kit::Near client signed as the sandbox root account. All contract interaction, account creation, and WASM deployment goes through it.

pub struct NearBlockchain { /* root_client + rpc_url */ }

impl NearBlockchain {
    pub fn new(rpc_url: &str, chain_id: &str, root_account: &str,
        root_secret: near_kit::SecretKey) -> anyhow::Result<Self>;
    pub async fn create_account_with_keys(&self, name: &str, balance_near: u128,
        keys: &[SigningKey]) -> anyhow::Result<()>;
    pub async fn create_account_and_deploy(&self, name: &str, balance_near: u128,
        key: &SigningKey, wasm: &[u8]) -> anyhow::Result<DeployedContract>;
    pub fn client_for(&self, account_id: &str, key: &SigningKey)
        -> anyhow::Result<NearKitCaller<ExecutedOptimistic>>;
}

DeployedContract wraps the contract's account ID plus its own near-kit client. It exposes account_id(), client(), call (signed by the contract account), call_from_with_deposit (untyped escape hatch for calls with no typed method yet), view, and state() (parsed ProtocolContractState).

NearKitCaller<T> — signer-bound caller (caller module)

Binds a signer to a non-contract account (nodes voting, users submitting sign requests) and implements the CallContract transport trait, s.t. typed calls can go through MpcContractHandle The T parameter is the wait level: how far a call waits before returning (e.g. Final, ExecutedOptimistic,...). Finality influences the return type (T::Response).

pub trait CallMpc: Sized {
    fn call_mpc(self, contract_id: &AccountId) -> MpcContractHandle<Self>;
}

pub trait WithWaitLevel {
    fn with_wait_level<U: WaitLevel>(self) -> MpcContractHandle<NearKitCaller<U>>;
}

3. MpcNode / MpcNodeSetup — node process manager

MpcNodeSetup owns everything needed to start a node (binary path, home dir, signing keys, ports, contract account, sandbox chain info, foreign-chain config). It writes secrets.json and start_config.toml on creation and exposes start() to spawn the process, returning an MpcNode.

MpcNode owns a running child process via a ProcessGuard that SIGKILLs on drop. It exposes:

  • kill() -> MpcNodeSetup and restart() -> MpcNode — lifecycle.
  • has_exited() — sanity check to surface early crashes.
  • get_metric(name) — scrape /metrics and parse an i64.
  • set_block_ingestion(active) — writes a flag file that pauses block ingest (requires the network-hardship-simulation feature, which CI builds with).
  • web_address(), pprof_address(), setup() for accessors.

MpcNodeSetup also offers wipe_db() (targeted SST/MANIFEST cleanup) and reset_mpc_state() (nuke the entire home dir — used after SIGKILL where the indexer state may be corrupt).

4. MpcCluster — orchestration

The entry point for tests. MpcCluster::start(config) does everything:

  1. Create TestPorts via TestPorts::e2e_tests(config.port_seed).
  2. Create a per-test artifact directory (TestDir, kept on failure, see Debugging a failure).
  3. Start the NearSandbox.
  4. Build a NearBlockchain signed as the sandbox root.
  5. Generate deterministic signing keys for each node (near signer, p2p, operator, plus a separate contract deployer key).
  6. Deploy the compiled MPC contract WASM to mpc.sandbox.
  7. Create nodeN.sandbox accounts, each with a near_signer_key and a disjoint operator_key as full-access keys.
  8. Call init() on the contract with the initial participants.
  9. Call submit_participant_info for each initial participant (with a {"Mock": "Valid"} attestation — enough to satisfy the contract in tests).
  10. Deploy the tee-verifier WASM to tee-verifier.sandbox and vote it in from every participant, for topology parity with production. Mock attestations are verified without calling it, so the verifier stays idle; the cross-contract flow is covered by the mpc-contract sandbox tests.
  11. Spawn the mpc-node binaries (start before adding domains so key generation has running nodes to talk to).
  12. Sleep briefly and assert no node exited early.
  13. If config.domains is non-empty, vote add_domains from each participant and wait for Running state.
  14. Create user accounts for signing/CKD/verify requests.

The returned cluster exposes:

  • Node lifecycle: kill_nodes, start_nodes, reset_and_start_nodes (wipe + start + wait for health), kill_all.
  • Contract state: get_contract_state, wait_for_state, wait_for_node_healthy, get_tee_accounts.
  • Resharing: start_resharing, start_resharing_and_wait, vote_cancel_resharing_from, add_domains_and_wait, start_add_domains.
  • Metrics: get_metric_all_nodes, wait_for_metric_all_nodes.
  • Data management: wipe_db, set_block_ingestion.
  • Request submission: send_sign_request, send_ckd_request, send_verify_foreign_transaction.
  • Foreign chains: view_foreign_chains_supported_by_contract, view_foreign_chain_configurations, view_available_foreign_chains, view_foreign_chains_configs, view_allowed_foreign_chain_providers, register_foreign_chain_config, whitelist_foreign_chains, wait_for_foreign_chains_registrations, wait_for_available_foreign_chains
  • User accounts: user_client, default_user_account.

Drop kills all running nodes; the artifact directory is held via test_dir and removed when the cluster is dropped, unless the test failed (see Debugging a failure).

pub struct MpcClusterConfig {
    pub num_nodes: usize,
    pub threshold: usize,
    pub domains: Vec<DomainConfig>,
    pub binary_paths: Vec<PathBuf>,             // one or num_nodes
    pub contract_wasm: Vec<u8>,                 // pre-compiled by the test
    pub tee_verifier_wasm: Vec<u8>,             // pre-compiled by the test
    pub port_seed: u16,
    pub triples_to_buffer: usize,
    pub presignatures_to_buffer: usize,
    pub sandbox_version: String,
    pub home_base: Option<PathBuf>,
    pub initial_participant_indices: Vec<usize>,
    pub migration_targets: Vec<usize>,          // source node indices
    pub init_format: ContractInitFormat,
    pub foreign_chains: ForeignChainsClusterConfig,
}

pub struct ForeignChainsClusterConfig {
    pub node_configs: Vec<ForeignChainsConfig>, // per-node; empty = default for all
    pub whitelist: BTreeMap<ForeignChain, ChainEntry>, // voted in during setup
}

impl MpcClusterConfig {
    pub fn default_for_test(port_seed: u16, contract_wasm: Vec<u8>, tee_verifier_wasm: Vec<u8>) -> Self;
    pub fn participant_indices(&self) -> Vec<usize>;
}

5. TestPorts::e2e_tests — deterministic port layout

Each test declares a port_seed: u16 (the allocator's test_id). Ports are computed as:

e2e base (20000) + test_id * ports_per_test() + offset

For the e2e scheme, ports_per_test is 82 (2 cluster ports + 8 per-node ports × 10 maximum nodes); the mpc-node scheme uses a different block size, so this number is specific to TestPorts::e2e_tests. Cluster-level ports cover the sandbox RPC and network; per-node ports cover p2p, web UI, migration web UI, pprof, and the node's internal neard RPC/network.

The allocator is test_port_allocator::TestPorts, the same struct the mpc-node integration tests use — each constructor pins its scheme, and the e2e scheme's range (20000+) is kept disjoint from TestPorts::mpc_node_tests (10000+) and reserve_port (40000+).

Centralising seeds in tests/common.rs (e.g. CKD_VERIFICATION_PORT_SEED = 9) keeps parallel tests from colliding. If a test crashes and leaves an orphan mpc-node holding its ports, the next run will fail to bind — clean up with cargo make kill-orphan-mpc-nodes.

6. Foreign chain mocks

foreign_chain_mock.rs exposes setup_bitcoin_mock, setup_evm_mock, and setup_starknet_mock. Each takes an httpmock::MockServer and attaches a POST / handler that returns hardcoded JSON-RPC responses for the methods the MPC nodes call during verify_foreign_transaction. Tests own the MockServers directly (rather than a wrapping struct) so they can wire the URLs into the per-node ForeignChainsConfig via MpcClusterConfig::foreign_chains.

Writing a test

Tests live under crates/e2e-tests/tests/<feature>.rs and must be declared as a module from tests/e2e.rs. tests/common.rs provides helpers used by most tests (must_setup_cluster, wait_for_presignatures, must_load_contract_wasm, send_sign_request, etc.).

// tests/request_lifecycle.rs
use crate::common;

#[tokio::test]
async fn request_lifecycle__signature_request_succeeds() {
    let (cluster, running) =
        common::must_setup_cluster(common::SIGN_REQUEST_PER_SCHEME_PORT_SEED, |_| {}).await;

    let mut rng = rand::thread_rng();
    let user = cluster.default_user_account().clone();
    common::send_sign_request(&cluster, &running, &mut rng, &user).await;
}

must_setup_cluster builds the default 3-node / 2-of-3 / 3-domain cluster, waits for Running, and blocks until presignatures are buffered. Pass a closure to override fields on MpcClusterConfig.

Prerequisites

neard sandbox binary

NearSandbox uses the near-sandbox crate, which downloads a precompiled neard binary on first run. That binary is linked against a specific glibc and will crash on systems whose glibc is incompatible (common on older/minimal Linux distributions and some CI images).

If you hit a crash or "GLIBC not found" error from the sandbox, point the tests at a locally built neard via NEAR_SANDBOX_BIN_PATH:

export NEAR_SANDBOX_BIN_PATH=/path/to/neard   # e.g. a neard built from nearcore
cargo make e2e-tests

Running the tests

cargo make e2e-tests                            # Build required binaries and run all tests
cargo make e2e-tests-skip-build                 # Reuse binaries from a previous run
cargo make e2e-tests-skip-build <name>          # Run tests matching a substring filter
cargo make e2e-tests-skip-build -E 'test(request_lifecycle)' --no-capture  # Forward nextest flags
cargo make kill-orphan-mpc-nodes                # Recover from ports held by crashed runs

Everything after the task name is forwarded to cargo nextest run unchanged, so any nextest filter or flag works (substring filters, -E expressions, --no-capture, --retries, -j, etc.). With no arguments, the whole suite runs with the ci-e2e profile. Do not put flags after a -- separator: it is forwarded verbatim, and nextest only accepts filters, not flags, after --.

The task runner builds five things before tests run: the mpc-node binary with the network-hardship-simulation feature, the MPC contract WASM, the tee-verifier WASM, the test parallel contract WASM, and the backup CLI. WASM paths are passed to tests via the MPC_CONTRACT_WASM, MPC_TEE_VERIFIER_WASM and MPC_PARALLEL_CONTRACT_WASM environment variables, read by the must_load_* helpers in tests/common.rs; if the env var is unset and no pre-built WASM is found, test-utils::contract_build::ContractBuilder builds it on the fly (useful for local iteration).

CI runs the same task via the mpc-e2e-tests job.

Debugging a failure

Each test gets an artifact directory (/tmp/mpc-e2e-<random> by default) with a node<i>/ subdirectory per node holding its config, secrets, RocksDB data, the embedded neard home, stdout.log (the node's tracing output) and stderr.log (panics only). TestDir (src/test_dir.rs) keeps it when the test failed and prints the path to the test's stderr; a startup failure also carries it in the error:

failed to start cluster: cluster artifacts preserved in /tmp/mpc-e2e-AbC123:
mpc-node 0 exited early, check /tmp/mpc-e2e-AbC123/node0 (stdout.log holds its
logs, stderr.log any panic)
Variable Effect
E2E_KEEP_TMP=1 Keep artifacts for passing tests too. 0, false, no or off (any case) deletes them even for failing ones.
E2E_HOME_BASE=<dir> Parent of the artifact directories, created if missing. Same as MpcClusterConfig::home_base, without editing the test.
MPC_NODE_LOG=<filter> RUST_LOG for the spawned mpc-node processes (default DEBUG).
MPC_NODE_BACKTRACE=<0|1|full> RUST_BACKTRACE for the spawned mpc-node processes (default 1).

Test layout conventions

  • Follow the <subject>__should_<assertion> or <subject>__<scenario> naming from docs/engineering-standards.md.
  • Reserve a unique port_seed constant in tests/common.rs before adding a new test. Don't reuse someone else's seed, even if the test is short.
  • Prefer common::must_setup_cluster over calling MpcCluster::start directly; it initialises tracing_subscriber and waits for presignatures.
  • Tests must be deterministic across parallel execution. Use the port allocator, the per-cluster artifact directory (cluster.test_dir), and the deterministic key generation rather than creating state outside the cluster.
  • Arithmetic in tests uses raw +/-/*//; overflow panics are the desired failure mode (see CLAUDE.md).

Related documents