runner: this repo now owns benchmark operations — port the campaign runner out of stellar-rpc

.github/workflows/shellcheck.yml

@@ -0,0 +1,36 @@
+name: Shellcheck runner
+
+# Lint the campaign-runner shell scripts. This is a static gate only — the
+# runner is exercised for real on the benchmark devbox, not in CI.
+on:
+  push:
+    branches: [main]
+    paths:
+      - "runner/**"
+      - ".github/workflows/shellcheck.yml"
+  pull_request:
+    paths:
+      - "runner/**"
+      - ".github/workflows/shellcheck.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  shellcheck:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Syntax-check (bash -n)
+        run: |
+          for f in runner/*.sh; do
+            echo "bash -n $f"
+            bash -n "$f"
+          done
+
+      - name: Shellcheck
+        # The .cfg is a sourced bash fragment and carries its own
+        # `shellcheck shell=bash` directive, so it lints too.
+        run: shellcheck runner/*.sh runner/*.cfg

README.md

@@ -10,7 +10,8 @@ step, no server: the numbers live in git and the site is just HTML/JS reading th
 ## What this is
 
 The bench suite (`stellar-rpc bench-ingest cold|hot`, `bench-query cold|hot`) runs in
-campaigns on an AWS NVMe devbox (`m6id.2xlarge`). Each campaign is several configurations
+campaigns on an AWS NVMe devbox (`m6id.2xlarge`), driven by the config-driven runner in
+[`runner/`](runner/) (see "Run a campaign" below). Each campaign is several configurations
 × 5 fresh-process runs; every run writes CSVs
 (`stage,n,n_items,total_ns,p50_ns,p90_ns,p99_ns,max_ns`) into its own directory, and the
 results are mirrored to GCS under `gs://rpc-full-history/benchmarks/`.
@@ -51,6 +52,38 @@ To smoke-test the viewer headlessly (loads each run in a jsdom DOM, asserts zero
 errors and the expected figure/section counts and sanity values), run `make smoke`
 (needs Node; installs `jsdom` under `tests/smoke/` on first run).
 
+## Run a campaign
+
+Campaigns run on the benchmark devbox via the scripts in [`runner/`](runner/) — this
+repo's operations side. The runner treats stellar-rpc as a **black box**: it maintains a
+build clone of it under `$BENCH_ROOT/src`, builds the configured ref, and drives the
+bench subcommands — no standalone stellar-rpc checkout is needed anywhere. See
+[runner/README.md](runner/README.md) for the bundle layout it produces and the minimum
+stellar-rpc ref it requires (the compatibility floor).
+
+```bash
+# 0. One-time on a fresh devbox (and again after every instance stop/start,
+#    which wipes the NVMe instance store): provision the machine.
+./runner/bootstrap.sh
+
+# 1. Write a campaign config (copy runner/example-campaign.cfg, adjust the keys)
+#    and sanity-check the full command plan. --dry-run builds, downloads, and
+#    runs nothing — it works on any machine, e.g. a laptop:
+./runner/campaign.sh my-campaign.cfg --dry-run
+
+# 2. Run it (in tmux — campaigns run for hours). Results land in
+#    $BENCH_ROOT/results/<NAME>-<sha>-<stamp>/, tarred to /tmp so the bundle
+#    survives an instance stop.
+./runner/campaign.sh my-campaign.cfg
+
+# 3. Publish the bundle to GCS. This happens automatically when the config
+#    sets PUBLISH_URI; run it by hand otherwise (or to retry a failed upload):
+./runner/publish.sh /mnt/nvme/bench/results/<run-id> gs://rpc-full-history/benchmarks
+```
+
+The published bundle is exactly what the next section converts into a committed run
+JSON — closing the loop: campaign config → run → publish → convert → viewer.
+
 ## Add a run locally (the primary flow today)
 
 On a laptop that's authenticated to GCS (`gcloud auth login`), pull a results directory
@@ -158,7 +191,13 @@ stellar-rpc-benchmarks/
 ├── SCHEMA.md                # run JSON schema v1 (the data contract)
 ├── .github/
 │   └── workflows/
-│       └── ingest.yml       # workflow_dispatch: GCS results dir → committed run
+│       ├── ingest.yml       # workflow_dispatch: GCS results dir → committed run
+│       └── shellcheck.yml   # lint runner/ scripts on every PR that touches them
+├── runner/                  # benchmark operations: devbox scripts producing result bundles
+│   ├── bootstrap.sh         # provision the devbox (idempotent, no builds)
+│   ├── campaign.sh          # campaign config → results bundle (see runner/README.md)
+│   ├── publish.sh           # bundle → gs:// or s3://
+│   └── example-campaign.cfg # annotated config to copy from
 ├── converter/
 │   ├── convert.py           # results dir → docs/runs/<id>.json (+ manifest), stdlib only
 │   ├── facts/               # per-unit sidecar facts (e.g. synthetic model/tps/pack)

SCHEMA.md

@@ -239,7 +239,8 @@ The converter auto-detects the input bundle layout from its subdirectory names:
 - **synthetic** — `synth-{cold,hot}-<profile>-run<R>`.
 - **pubnet** — `ingest-{cold,hot}-<chunk>-run<R>`, `query-{cold,hot}-<chunk>-run<R>`,
   `golden-download-<chunk>` (a timed sourcing leg surfaced as the `golden` section).
-- **campaign** — produced by `campaign.sh`. Timed dirs sit at the bundle root as
+- **campaign** — produced by `runner/campaign.sh` (the producer-side bundle layout is
+  documented in `runner/README.md`). Timed dirs sit at the bundle root as
   `{ingest,query}-{cold,hot}-<dataset>-c<chunk>-run<R>`; the unit id is the composite
   `<dataset>-c<chunk>` (e.g. `sac-6000-c1`). Untimed prep dirs `golden-<dataset>-c<chunk>`
   are dataset preparation, **not results** — the converter skips them and warns. The

runner/README.md

@@ -0,0 +1,88 @@
+# Campaign runner
+
+The operations side of this repo: a config-driven runner that produces the result
+bundles the rest of the pipeline consumes. It treats **stellar-rpc as a black box** — it
+clones it, builds the requested ref, and drives its `bench-ingest` / `bench-query`
+subcommands; it never lives inside a stellar-rpc checkout and never modifies one.
+
+```
+runner/
+├── bootstrap.sh          # provision the devbox (NVMe, apt, Go, Rust, native libs, env)
+├── campaign.sh           # run one campaign from a config file
+├── publish.sh            # upload a finished bundle to gs:// or s3://
+└── example-campaign.cfg  # annotated config to copy from
+```
+
+`campaign.sh`'s header comment is the authoritative reference for config keys and
+dataset kinds; the [top-level README](../README.md#run-a-campaign) walks through the
+operator flow end to end.
+
+## Compatibility floor
+
+The runner requires a stellar-rpc ref whose bench subcommands **write `invocation.json`
+into every `--out` directory** — that is stellar-rpc's `bench-run-metadata` branch or any
+descendant of it (its merge commit into `feature/full-history`, once merged). The default
+`REF=feature/full-history` satisfies this only after that merge lands; until then, set
+`REF=bench-run-metadata` (or a descendant) in the campaign config. Older refs produce
+bundles without per-invocation manifests, which the converter accepts but with weaker
+provenance (see `SCHEMA.md` § Inputs).
+
+## `$BENCH_ROOT` layout
+
+Everything the runner touches lives under `$BENCH_ROOT` (default `/mnt/nvme/bench`, the
+devbox's NVMe instance store — wiped on instance stop/start; everything here is
+re-creatable):
+
+```
+$BENCH_ROOT/
+├── src/       persistent build clone of $REPO (re-pointed, fetched, and hard-reset
+│              every campaign; gitignored build caches survive, so rebuilds are
+│              incremental)
+├── bin/       versioned binaries: stellar-rpc-<sha>
+├── golden/    immutable prepared datasets, one dir per dataset name
+│              (rm -rf golden/<name> to force a re-fetch)
+├── fixture/   staging area for generated fixture packs
+├── scratch/   cold-ingest output, deleted before every run
+├── hot/       hot DBs; the last run's DB is kept for the hot query suite
+└── results/   campaign bundles: <NAME>-<sha>-<stamp>/
+```
+
+The finished bundle is also tarred to `/tmp/bench-results-<NAME>-<sha>-<stamp>.tgz` (EBS
+root, survives an instance stop) and, when `PUBLISH_URI` is set, uploaded to
+`<PUBLISH_URI>/<NAME>-<sha>-<stamp>/`.
+
+## Campaign bundle layout — the cross-repo contract
+
+A campaign bundle is what `publish.sh` uploads and what `converter/convert.py` consumes
+(as the **campaign** input layout). Two repos write into it, so its shape is a contract:
+
+```
+<NAME>-<sha>-<stamp>/                        # run_id = the bundle basename
+├── <config>.cfg                             # the campaign config, verbatim
+├── binary.txt                               # benchmarked binary identity (free text)
+├── machine-metadata.txt                     # machine facts (free text)
+├── metadata.json                            # ← written by campaign.sh (THIS repo)
+├── golden-<dataset>-c<chunk>/               # untimed dataset prep — not results;
+│                                            #   the converter skips these and warns
+├── ingest-{cold,hot}-<dataset>-c<chunk>-run<R>/
+│   ├── driver.csv, hot.csv, *.csv           # ← written by stellar-rpc bench subcommands
+│   └── invocation.json                      # ← written by stellar-rpc bench subcommands
+└── query-{cold,hot}-<dataset>-c<chunk>-run<R>/
+    └── …same shape…
+```
+
+Who owns what:
+
+- **`metadata.json`** (bundle root, `schema_version` 1) — written by `campaign.sh` here.
+  Run identity (`run_id`, `started_at`), the campaign config knobs (incl.
+  `close_interval`), the dataset list, structured `hardware`, and `hostname`.
+- **`invocation.json`** (each `--out` dir, `schema_version` 1) — written by stellar-rpc's
+  `bench-ingest` / `bench-query`. Binary identity (`binary.{commit_hash, branch, version,
+  build_timestamp}`) and the resolved subcommand flags.
+
+The consumer side of this contract — exactly which fields the converter reads, and the
+precedence rules between the manifests, the free-text metadata, and CLI arguments — is
+documented in [`SCHEMA.md` § Inputs](../SCHEMA.md#inputs--result-bundle-layouts--manifests).
+Changing either manifest's shape, the bundle directory naming, or the CSV columns is a
+cross-repo change: update the producer (here or in stellar-rpc), the converter, and
+`SCHEMA.md` together.

runner/bootstrap.sh

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+#
+# Idempotent bootstrap for a full-history benchmark machine: an EC2 instance
+# with a local NVMe instance store (e.g. m6id.2xlarge) running Ubuntu 24.04.
+# It only provisions — NVMe mount, apt packages, Go, Rust, native libs, env;
+# campaign.sh does all cloning-current and building. Safe to re-run any time —
+# in particular after an instance stop/start, which wipes the NVMe instance
+# store (golden packs are re-downloaded and the build clone re-created by the
+# next bootstrap/campaign run).
+#
+# Usage (on the machine):
+#   ./runner/bootstrap.sh
+#
+# Overridable: NVME_DEV (default /dev/nvme1n1), BENCH_ROOT (default
+# /mnt/nvme/bench), REPO (git URL or local path of stellar-rpc, default
+# https://github.com/stellar/stellar-rpc.git).
+#
+set -euo pipefail
+
+NVME_DEV="${NVME_DEV:-/dev/nvme1n1}"
+MOUNT=/mnt/nvme
+BENCH_ROOT="${BENCH_ROOT:-$MOUNT/bench}"
+REPO="${REPO:-https://github.com/stellar/stellar-rpc.git}"
+SRC=$BENCH_ROOT/src
+
+note() { echo "== $*"; }
+
+# --- NVMe instance store: format if raw, mount if unmounted -----------------
+[ -b "$NVME_DEV" ] || { echo "error: $NVME_DEV is not a block device" >&2; exit 1; }
+model=$(lsblk -no MODEL "$NVME_DEV" | head -1)
+case "$model" in
+  *"Instance Storage"*) ;;
+  *) echo "error: refusing to touch $NVME_DEV — model '$model' is not the EC2 instance store" >&2; exit 1 ;;
+esac
+if ! sudo blkid "$NVME_DEV" >/dev/null 2>&1; then
+  note "no filesystem on $NVME_DEV (fresh instance store) — formatting"
+  sudo mkfs.ext4 -m0 "$NVME_DEV"
+fi
+if ! mountpoint -q "$MOUNT"; then
+  sudo mkdir -p "$MOUNT"
+  sudo mount -o noatime "$NVME_DEV" "$MOUNT"
+  sudo chown "$USER" "$MOUNT"
+fi
+mkdir -p "$BENCH_ROOT"/{golden,scratch,hot,results}
+
+# --- fsync honesty probe: the whole reason this machine exists --------------
+probe=$(dd if=/dev/zero of="$MOUNT/.fsync-probe" bs=4k count=2000 oflag=dsync 2>&1 | tail -1)
+rm -f "$MOUNT/.fsync-probe"
+note "fsync probe: $probe"
+case "$probe" in
+  *GB/s*) echo "WARNING: GB/s-scale dsync writes — fsync is being absorbed; hot-commit numbers would be fiction" >&2 ;;
+esac
+
+# --- system packages ---------------------------------------------------------
+note "apt packages"
+sudo apt-get update -qq
+sudo apt-get install -y -qq build-essential git jq pkg-config cmake ninja-build \
+  tmux libsnappy-dev liblz4-dev zlib1g-dev
+
+# --- cloud CLIs: only some campaigns need them, so warn rather than fail -----
+# gcloud: packs-gs datasets and gs:// publishing. aws: bsb-s3 datasets and
+# s3:// publishing. Neither ships in apt in a form worth installing here.
+command -v gcloud >/dev/null 2>&1 ||
+  echo "WARNING: gcloud not found — packs-gs datasets and gs:// PUBLISH_URI will fail; install it: https://cloud.google.com/sdk/docs/install" >&2
+command -v aws >/dev/null 2>&1 ||
+  echo "WARNING: aws not found — bsb-s3 datasets and s3:// PUBLISH_URI will fail; install it: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" >&2
+
+# --- Go (>= 1.26; Noble's apt Go is too old) ---------------------------------
+if ! /usr/local/go/bin/go version 2>/dev/null | grep -Eq 'go1\.(2[6-9]|[3-9][0-9])'; then
+  note "installing Go"
+  GOVER=$(curl -fsSL 'https://go.dev/VERSION?m=text' | head -1)
+  curl -fsSL "https://go.dev/dl/${GOVER}.linux-amd64.tar.gz" -o /tmp/go.tgz
+  # decompress as the user: sudo'd tar cannot always exec gzip
+  gunzip -f /tmp/go.tgz
+  sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xf /tmp/go.tar
+fi
+
+# --- Rust --------------------------------------------------------------------
+if [ ! -x "$HOME/.cargo/bin/rustc" ]; then
+  note "installing Rust"
+  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+fi
+
+# --- build clone --------------------------------------------------------------
+# The box needs no standalone stellar-rpc checkout: seed the persistent build
+# clone campaign.sh maintains at $BENCH_ROOT/src, and run the native-lib
+# install scripts below from it. campaign.sh re-points, fetches, and checks
+# out this clone per campaign (and re-clones it itself if this step is ever
+# skipped).
+if [ ! -d "$SRC/.git" ]; then
+  note "cloning $REPO into $SRC"
+  git clone "$REPO" "$SRC"
+fi
+
+# --- native libs, mirroring CI's setup-go action ------------------------------
+[ -e "$HOME/.zstd/lib/libzstd.so" ] ||
+  (cd "$SRC" && PREFIX="$HOME/.zstd" ./scripts/install-zstd.sh)
+[ -e "$HOME/.rocksdb/lib/librocksdb.so" ] ||
+  (cd "$SRC" && PREFIX="$HOME/.rocksdb" ZSTD_HOME="$HOME/.zstd" ./scripts/install-rocksdb.sh)
+
+# --- environment: persist for future shells, set for this run ----------------
+if ! grep -q '# bench-campaigns env' "$HOME/.bashrc"; then
+  cat >> "$HOME/.bashrc" <<'EOF'
+# bench-campaigns env
+export PATH=/usr/local/go/bin:$HOME/go/bin:$HOME/.cargo/bin:$PATH
+export CGO_CFLAGS="-I$HOME/.zstd/include -I$HOME/.rocksdb/include"
+export CGO_LDFLAGS="-L$HOME/.zstd/lib -L$HOME/.rocksdb/lib"
+export LD_LIBRARY_PATH="$HOME/.zstd/lib:$HOME/.rocksdb/lib"
+EOF
+fi
+export PATH=/usr/local/go/bin:$HOME/go/bin:$HOME/.cargo/bin:$PATH
+export CGO_CFLAGS="-I$HOME/.zstd/include -I$HOME/.rocksdb/include"
+export CGO_LDFLAGS="-L$HOME/.zstd/lib -L$HOME/.rocksdb/lib"
+export LD_LIBRARY_PATH="$HOME/.zstd/lib:$HOME/.rocksdb/lib"
+
+note "bootstrap OK — campaign.sh builds the benchmark binary on first run"