testops: add unified lab controller - #1
Draft
pingqiu wants to merge 38 commits into
Draft
Conversation
Standardize sw-test-runner as a cross-product TestOps platform (block/S3/VFS): - docs/testops-handbook.md: drop-and-test onboarding manual (env access, run, watch+report, build->seed->test->cleanup, write a scenario, rules, gotchas) - docs/cross-product-testops-standard.md: architecture, scenario contract, placement rule, multi-project shared-lab isolation, global dashboard - packs/s3 + cmd/sweeds3 + scenarios/s3-smoke-chain.yaml: reference S3 pack, proven end-to-end on m02 (build->seed->test->cleanup, 10/10, zero residue) - cmd/testops-dashboard: read-only global runs view across projects, serves result.html, renders docs at /docs (goldmark); -emit-md for a wiki feed - cli.go: list shows new product-pack tiers (s3/vfs) without editing the slice Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/docs now lists only the four docs an agent should follow, in reading order (Handbook -> Standard -> Scenario Spec -> Tutorial), via a docReadingOrder whitelist; internal roadmap/evidence md are hidden from the list (still renderable by direct URL). Adds a "read in order" hint that points at the Standard as the contract that makes runs observable in the shared results root. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the index runs table (.86em->1.12em), status pills (.72em->.9em), metadata (.8em->.9em) and header title, plus a little more cell padding — the run list is the thing people stare at, so make it legible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
manifest.json carries no status field — the runner writes the authoritative state (running/pass/fail) to status.json. scan() now reads status.json and overrides the run status with its state, so in-progress runs show a pulsing blue "running" pill plus "current_phase done/total" progress, and the header gains a running count. Falls back to manifest.status for legacy bundles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Whitelist rdma-vfs-s3-testing-start-here.md (sra-next developer testing start-here: clone, basic tests, native RC/DC + UCX builds, rdma-loader usage, current status) as a product testing guide after the core four, and note "product testing guides" in the reading-order hint. The doc itself is a snapshot served from the shared SMB docs dir (sourced from the sra-next repo); the whitelist only references the filename, so it shows when present. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a MkDocs Material wiki under docs/ (mirrors the seaweed-block wiki):
- mkdocs.yml: Material theme, nav = Start Here / Standard / Architecture /
Product Testing Guides / Scenarios / Live Dashboard. docs_dir=docs reuses the
existing top-level docs (handbook, standard, scenario-spec, tutorial).
- docs/wiki/: index (home), code-map, packs-and-binaries (accurate cmd->pack
table), scenario-catalog (public set), dashboard (live view + deploy + janitor),
guides/{block,s3,rdma} product testing entry points.
- requirements-docs.txt (mkdocs-material) + Makefile (build/test/vet/wiki) +
README "Docs and wiki" + "Build" sections.
Tidy (additive only — no moves to dev-owned packs/scenarios):
- .gitignore: anchor binary names to repo root (bare names matched cmd/<name>/
source dirs), add /bin/, *.exe, /site/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bare `sw-test-runner` .gitignore entry matched the cmd/sw-test-runner/ directory, so the all-in-one binary's entry point was never tracked — a fresh clone from origin could not build `sw-test-runner` at all. The anchored .gitignore fix un-hides it; commit the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add free-form, dashboard-friendly run metadata so multiple agents/teams sharing one dashboard can tell runs apart and filter them: - Scenario gains an optional `metadata:` block (test_id, team, owner, ...) that flows into manifest.json via CreateRunBundle. - `run -meta key=value` (repeatable) merges run-context (project, run_by, ...) into the manifest via the new RunBundle.MergeMetadata. Additive: manifest.metadata is omitempty; nothing else changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read manifest.metadata (and an optional meta.json sidecar so agents can annotate post-hoc). Show test_id under the scenario, run_by under the host, team as a clickable chip; project link + team chip filter the view (?project= / ?team=) with a filter banner. metadata.project overrides the dir-derived project. Doc the three ways to set metadata in the wiki dashboard page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wiki design docs with mermaid diagrams (Material renders them via the superfences custom_fence, now enabled in mkdocs.yml): - how-it-works.md: core+packs+binaries model, run pipeline (sequence), phases, variables, status, distributed mode — component + sequence diagrams. - lifecycle.md: build->seed->test->cleanup (state diagram), cluster attach/managed control (flow), the shared-lab cleanup discipline. - deploy.md: build, lab topology (flow), nodes/infra.Node, dashboard systemd, janitor. - syntax.md: scenario YAML quick reference (top-level/phase/action fields) from the actual structs, annotated example. - code-map.md flow redrawn as mermaid; index + nav (new Design / Deploy sections). Handbook: new "Tag your run (metadata)" section so testers pass -meta project/run_by/team (and scenario metadata: block / meta.json sidecar). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the reusable emit_provenance core action (the one place a gate declares the common envelope) and onboard three products through it, so ONE qa-assert validates all four (rdma already emits rdma-prefixed keys): - actions/provenance.go + register: emit_provenance (TierCore) emits __product/ __gate_pass/__gate_status/__tested_ref/__tested_sha/__lab_run_id + pass-through __-prefixed product rows (auto-propagated into result.json vars). - scenarios/s3-smoke-chain.yaml: S3 http gate emits the envelope (+ s3.expect). - scenarios/vfs-cross-access-chain.yaml: VFS proves S3<->VFS shared-data cross-access (M02 filer/volume, M01 FUSE mount, SHA parity both ways; + vfs.expect). - block.expect: block first-volume floors (gate wired in seaweed_block). - docs/unified-testops.md: overview + taxonomy (object/KV data model with s3/vfs/kv access methods x transports; block distinct) + product status matrix; wiki vfs guide + nav. Verified: go test ./actions/ + root pass; rdma/s3/block/vfs bundles all print QA_BUNDLE_ASSERT_OK via scripts/qa-assert.sh --profile <product>.expect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- controller: register the vfs suite (project vfs-ci, SubmitEnabled) so POST /api/vfs/submit + status?suite=vfs work like rdma. block stays disabled until its worker deploys. - scripts/run-vfs-ci.sh: the per-suite run script (mirrors run-rdma-ci.sh) — runs the self-contained vfs-cross-access-chain gate via sweeds3, overrides ssh_key to the M01 linux key, accepts the bundle with qa-assert --profile vfs. The generic testops-ci-worker.sh drives it (QUEUE_DIR/LOCK_FILE/RUN_SCRIPT env). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- controller: block suite SubmitEnabled=true, DefaultRef=sha-28a99ce4f644 (the published sw-block image tag under test), DefaultTestID=helm-single-node-first-volume. - scripts/run-block-ci.sh (executable): syncs the SMB-staged block harness (scenario+scripts+charts) into product_root on m02 (m02 mounts SMB -> local copy), runs the helm first-volume gate via ./bin/swblock against the published images, accepts via qa-assert --profile block. The tested artifact is the image tag (REF). Driven by the generic testops-ci-worker.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- controller: register the s3 suite (project s3-ci, SubmitEnabled) so POST /api/s3/submit + status?suite=s3 work like rdma/block/vfs. - scripts/run-s3-ci.sh (executable): runs the self-contained s3-smoke-chain via ./bin/sweeds3, accepts via qa-assert --profile s3. Driven by the generic worker. All four products (rdma/block/vfs/s3) now submit through one controller + one generic worker + one qa-assert + one emit_provenance envelope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All four products now submit via 9099 through the generic worker and pass the same qa-assert. Update the overview status matrix (§5) to 4/4 full, §3 to note all suites have workers, and deploy.md with the per-suite worker table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e/client)
New wiki/submitting.md consolidates the dev-facing mechanics that were scattered:
- two ways to run (CI submit to M01 9099 vs local CLI);
- run an existing gate = submit a `ref` only (git branch/SHA for rdma/s3/vfs,
image tag for block) — no scenario YAML per run; each suite has one canonical gate;
- add a NEW gate = write a scenario once (+ emit_provenance + qa-profile);
- define topology (service<->client): topology.nodes + node: dispatch, advertise a
reachable IP (not 127.0.0.1), cross-node value passing via save_as + {{ }}.
Also add a service<->client example to syntax.md; nav wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The datapath rc_push row is a sampled no-op-callback bench; it cannot see a copy-path (载体) regression while datapath stays fast. Add a floor on __rdma_perf_rc_push_discard_mib_s (proxy fetch + copy-out + drop), guarding the migration ">5 GB/s client" contract. Set 5120 with margin below the observed ~6040 to avoid run-to-run variance flakes. Also applied to the deployed copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This turns
sw-test-runnerinto the shared TestOps controller for the lab.It adds:
9099;/api/<suite>/submitand/api/controller/status?suite=<suite>;rdma,s3,block, andvfs;/mnt/smb/work/share/testops;qa-assert.shplus per-suite profiles;Why
The team needs one place to submit a branch/ref, run the right lab gate, and see the bundle/result. RDMA already uses this pattern; this PR makes it reusable for S3, block, and VFS instead of each project inventing a separate harness.
How to Use
Submit a pushed ref:
Then watch:
The same shape applies to
s3,block, andvfsonce their worker service is installed on M01.Validation
go test ./... python -m mkdocs build git diff --checkNotes:
mkdocs buildpasses.mkdocs build --strictis currently blocked by an upstream Material for MkDocs warning about future MkDocs 2.0 changes, not by a broken repo link. The broken wiki link found during PR prep was fixed.Docs to Review First
docs/unified-testops.mddocs/wiki/submitting.mddocs/control-plane-product-contract.mddocs/qa-bundle-assert.md