test(devnet): rc19 gap coverage for #1241/#1229/#1234 + sweep stale oxigraph

[branarakic]

What

Adds devnet coverage for three new-since-rc17 features that had zero devnet tests (surfaced by the rc19 readiness review), plus a devnet-harness fix. All three scripts are green on a 6-node fleet.

New gap-test scripts

Script	Feature	Result	Proves
devnet-test-seal-decouple.sh	#1241 / #1116 CG-independent seal	10/10	finalize seals on a never-registered CG; /vm/publish auto-registers the CG and mints; full share seals-by-default; skipSeal→unsealed SWM share; seal-in-SWM (finalize layer:swm in place); 400 + strict-boolean guards
devnet-test-cg-registration-deposit.sh	#1229 / OT-RFC-53 deposit	6/6	dormant (param=0 → no escrow) → armed (governance sets param; register pulls the deposit gross into the CSS vault via the adapter's lazy approve-and-retry) → param reset to 0 at teardown
devnet-test-agents-meta-bounded.sh	#1234 agents/_meta bloat	12/12	agent-registry DATA graph populated while <agents>/_meta stays bounded on every node (no unbounded tentative records)

Harness fix (scripts/devnet.sh)

stop_oxigraph_servers() only removed the docker oxigraph containers (nodes 5-6); it never killed the local daemon-managed oxigraph-server binaries (nodes 1-N). A SIGKILL'd node orphans them → the :79xx port stays bound → the next devnet.sh start dies with Address already in use and those nodes silently fail to boot (the main source of devnet flakiness on repeated boots — not memory). Now a path-scoped sweep (only this devnet's orphaned oxigraph procs) runs on stop/clean.

Notes

• Scripts are self-contained, reuse a running ./scripts/devnet.sh start 6 fleet, and create their own CGs.
• The deposit test arms a global param and resets it to 0 at teardown (dormant-regression-safe for other suites).
• The deposit governance setter uses the well-known public hardhat account[0] key — local 31337 devnet only, not a real deployer key.

🤖 Generated with Claude Code

[otReviewAgent]

🔴 Bug: devnet-test-agents-meta-bounded.sh can report success without exercising the fleet. Missing token files are skipped, and an unreachable node is also skipped instead of counted as a failure; if the devnet is not running or fewer than NUM_NODES nodes are provisioned, the loop can finish with FAIL=0 and exit 0. That gives a false green for the bounded _meta business behavior the script is meant to prove. Treat missing/unreachable expected nodes as fatal, and require at least one checked node, preferably exactly NUM_NODES.

[otReviewAgent]

🟡 Issue: The deposit test mutates a global chain parameter and only resets it on the normal fall-through path, always to hard-coded 0. If the script is interrupted or times out after arming the deposit, later devnet suites run with a live deposit; if the initial value was not zero, the teardown also clobbers that original value. Capture P0 and install a trap before line 77 to restore it on EXIT/INT/TERM, or abort before mutation when the baseline is not the expected dormant devnet value.

[otReviewAgent]

🟡 Issue: The deposit test always resets contextGraphRegistrationDeposit to 0 instead of restoring the baseline value read in P0. If the script is run against a devnet where the parameter was intentionally nonzero, or after a previous suite armed it, this teardown silently changes global chain state for subsequent tests. Capture the original value and restore that value, ideally via a trap after any successful mutation.

[otReviewAgent]

🟡 Issue: After this line the script mutates the shared devnet chain parameter, but cleanup only happens at the normal bottom-of-file path. Any failure, timeout, or Ctrl-C before the teardown leaves contextGraphRegistrationDeposit nonzero for later suites, despite the script's isolation contract. Register a cleanup trap before arming the parameter so the reset runs on every exit path.

[otReviewAgent]

🟡 Issue: This script arms the global contextGraphRegistrationDeposit but only resets it at the normal end of the script. If the process is interrupted or killed after this setter succeeds and before the teardown block runs, the devnet remains in the non-dormant deposit mode, which can break later suites that assume the default zero-deposit chain state. Add an EXIT/INT/TERM cleanup trap before arming the parameter and clear it once teardown succeeds.

After arming contextGraphRegistrationDeposit, cleanup depends on reaching the final teardown block. An interrupt, timeout, or killed shell between the setter and line 98 leaves the shared devnet chain parameter live despite the script promising dormant-regression-safe teardown. Install a trap immediately after a successful arm to reset the parameter on EXIT INT TERM, and make the final check use that same cleanup path.

After setting the global registration deposit live, cleanup only runs if the script reaches the teardown block. An interrupt, timeout, or hung command after this line leaves contextGraphRegistrationDeposit at 1 TRAC and can poison subsequent devnet suites despite the script's dormant-regression-safe claim. Install a trap immediately after a successful arm to reset the parameter on EXIT/INT/TERM.

[otReviewAgent]

🟡 Issue: This assertion only checks sealed=false for the skipSeal:true path, even though the documented API contract under test is {sealed:false, publishReady:false}. A regression that marks an unsealed SWM share as publishReady:true would still pass here and then allow publish gating bugs to slip through. Assert both booleans explicitly, as the script already extracts READY_B.

The script claims to validate publishReady=false for skipSeal:true, but it only asserts sealed=false; READY_B is logged but never checked. A regression that marks an unsealed share publish-ready would still pass this test. The same pattern exists for the default share path at line 91, where publishReady=true is expected but not asserted.

[otReviewAgent]

🟡 Issue: The skipSeal contract documented by the test is {sealed:false, publishReady:false}, but the assertion only checks sealed=false and merely logs publishReady. A regression that marks an unsealed share as publish-ready would still pass this section, so the test does not actually validate the advertised HTTP contract.

[otReviewAgent]

🟡 Issue: The skipSeal case claims to verify {sealed:false, publishReady:false}, but the assertion only checks sealed=false and merely logs publishReady. A regression that leaves unsealed SWM shares publish-ready would pass this test. Assert READY_B=false here.

[otReviewAgent]

🟡 Issue: The three new devnet proof scripts each define their own tiny test framework plus JSON/curl helpers (ok/bad, token/port resolution, field parsing, API wrappers). The duplicated inline Node one-liners are already diverging in behavior: this script's qcount accepts several response shapes, while the deposit and seal scripts use a different dot-path parser, and only the seal script has an HTTP wrapper. That makes future endpoint or response-shape changes easy to fix in one proof but miss in the others. Consider extracting a small devnet-test-helpers.sh for token/port lookup, JSON field/count parsing, HTTP calls, and summary accounting, then keep each proof script focused on scenario steps.

[otReviewAgent]

🟡 Issue: This path scoping is implemented as a raw pkill -f regular expression built from $DEVNET_DIR. Because DEVNET_DIR is user-configurable and not regex-escaped, characters like ., [, +, or spaces change the match semantics, so the cleanup behavior is hard to reason about and can match more than the intended devnet-owned oxigraph processes. For maintainability and safer ownership boundaries, prefer recording/using per-node oxigraph pid files or matching against an escaped canonical path with pgrep -f plus explicit verification before killing.

[otReviewAgent]

🟡 Issue: The loop silently skips missing token files and unreachable nodes, then exits success when FAIL remains 0. A stale or absent .devnet, wrong DEVNET_DIR, or fully stopped fleet will produce PASS=0 FAIL=0 and a green exit, so this validation can report success without checking any node. Count checked nodes and fail when fewer than the expected fleet are reachable, or at minimum fail when zero nodes were tested.

[otReviewAgent]

🔴 Bug: This script can report success without checking any node. Missing token files are skipped, and unreachable nodes only log and continue, so a stopped or partially provisioned devnet leaves PASS=0/FAIL=0 and exits 0. Track how many nodes were actually checked and fail when it is zero, and consider treating expected fleet members as failures unless explicitly optional.

[otReviewAgent]

🟡 Issue: The agents-meta proof can pass without checking the fleet. Missing auth tokens and unreachable nodes are skipped, and the script exits success whenever FAIL remains zero, so an empty/stopped devnet can report ALL AGENTS-META CHECKS PASSED with PASS=0 FAIL=0. For an integration proof this should fail if fewer than the expected nodes were checked, and query failures should be counted as failures rather than skipped.

Skipping unreachable nodes without incrementing FAIL lets this test exit successfully without testing anything. If auth tokens are missing, or every queried node returns an empty/unparseable response, the loop can produce PASS=0 FAIL=0 and line 66 exits 0. For a validation script, unreachable or absent nodes should fail the run, or at least fail when no nodes were checked.

[otReviewAgent]

🔴 Bug: This test can pass without testing any node: missing token files are silently skipped, unreachable nodes are skipped when DATA is empty, and line 66 exits success whenever FAIL remains 0. A stopped or misconfigured devnet can therefore report ALL AGENTS-META CHECKS PASSED with PASS=0. Track a tested-node count and fail on missing/unreachable expected nodes.

[otReviewAgent]

🟡 Issue: This adds the RFC49 catalog suite to the shared comprehensive run, but that suite is stateful: it rewrites .devnet/node4/config.json to set swmHostMode.stripCiphertext=false and restarts node4 without restoring it. The orchestrator then continues into rc11/rfc38/probes/soak with node4 no longer using the normal strip-ciphertext behavior, and a passing run leaves the devnet mutated for future runs. Either isolate this suite or add cleanup/trap logic in the RFC49 script to restore and restart node4 before returning.

The three new devnet proof scripts (devnet-test-agents-meta-bounded.sh, devnet-test-cg-registration-deposit.sh, and devnet-test-seal-decouple.sh) are not registered in the comprehensive orchestrator; only rfc49-catalog was added here. Unless CI invokes those scripts explicitly elsewhere, the primary validation run will never execute the new coverage this PR adds, so the evidence remains manual-only.

[otReviewAgent]

🟡 Issue: Only the RFC49 script is wired into the comprehensive run. The new devnet validation scripts for agents meta bounds, CG registration deposits, and seal decoupling are also absent from _devnet-full-sweep.sh's hard-coded list, so the default comprehensive validation still never exercises those claimed coverage gaps. Add them to an orchestrated suite or make the validation evidence explicitly manual-only.

[otReviewAgent]

🟡 Issue: MT is allowed to be empty and ${MT:-0} then treats a failed _meta query or malformed response as zero rows, producing a false agents/_meta BOUNDED pass. That masks exactly the storage/query failure this script is supposed to catch. Require a parsed numeric count for _meta and fail the node when the count cannot be read.

[otReviewAgent]

🟡 Issue: The section B setup calls discard both HTTP status and body, so create/write failures are only surfaced later as misleading share/finalize failures. The surrounding sections already preserve responses and assert status; extracting a small checked request helper would keep the test readable and make failures point at the operation that actually broke.

[otReviewAgent]

🟡 Issue: The memory governor is effectively disabled on Linux because it only reads macOS memory_pressure and otherwise returns a constant 50. On the Linux devnet/CI hosts this soak can keep publishing even when free memory is below MEM_FLOOR_PCT, contradicting the resource-governed behavior and risking host exhaustion during long runs. Use /proc/meminfo as the fallback instead of a fixed value.

The memory governor only checks macOS memory_pressure and otherwise returns a constant 50, which disables the advertised resource guard on Linux runners/dev machines. Since this script is positioned as a long-running resource-governed soak, the abstraction should either implement Linux /proc/meminfo/free support or clearly fail/disable with an explicit log instead of reporting synthetic free memory.

[otReviewAgent]

🟡 Issue: The 'update' activity does not exercise an update path: it creates a brand-new KA, writes one quad, finalizes, shares, and publishes it once. That increments UPDATED without calling /api/update or modifying an existing published asset, so the soak gives false coverage for update lifecycle behavior.

[otReviewAgent]

🟡 Issue: The timeout fallback is effectively dead code. If timeout is missing, the if compound runs the else branch, logs a failure, increments ERRORS, and returns success from the assignment, so the trailing || { ... } fallback never executes. This makes the invite flow look intentionally portable while silently failing on hosts without GNU timeout. Prefer an explicit command -v timeout branch or a small helper for timed best-effort commands.

[otReviewAgent]

💡 Suggestion: The new scripts copy several nontrivial shell helpers (node_token, node_port, api, field/jget, pass/fail counters) with slightly different timeouts, parsing behavior, and error handling. This PR adds enough duplicated devnet harness code that future API or response-format changes will require synchronized edits across multiple files. Consider extracting a small sourced scripts/devnet-lib.sh for HTTP calls, JSON field extraction, node auth, and result accounting before adding more scenarios.

[branarakic]

Review feedback addressed (2 commits)

Thanks for the thorough pass. Consolidated response to the comments across the review rounds (they dedupe to 11 distinct findings + the DRY suggestion). Commits: eaee0f14a (fixes) and dc79c626b (shared-lib extraction).

🔴 agents-meta could report green without checking any node — fixed. Missing token, unreachable node, and an unreadable _meta count are now failures (not skips); the run tracks a CHECKED count and fails if it checked fewer than NUM_NODES (or zero). An empty _meta graph returns COUNT "0" (a value, not ""), so a healthy-but-quiet fleet still passes — no false-red.

🟡 cg-deposit interrupt safety — added an EXIT/INT/TERM trap that restores the captured baseline P0 (not a hard-coded 0), armed right after the setter and disarmed after the verified teardown. A crash/timeout after arming can no longer leave the global param live.

🟡 seal-decouple skipSeal contract — now asserts both sealed=false AND publishReady=false; added a checked-request helper so the section-B setup surfaces create/write failures at the failing op.

🟡 soak governor / update / timeout — mem_free_pct now has a real /proc/meminfo branch (was a constant 50 off-macOS → inert on Linux CI); the "update" tick does a real POST /api/update on the published asset (was a first-time publish); the dead timeout fallback is replaced with an explicit command -v check (run_timed).

🟡 devnet.sh oxigraph sweep — no longer feeds the user-configurable DEVNET_DIR to pkill -f as a raw regex; matches the dir as a literal substring via a quoted case glob (regex/glob-metachar safe).

🟡 rfc49-catalog statefulness (newly wired into the comprehensive run) — backs up the baseline core's config and restores it + restarts on any exit, so a passing comprehensive run no longer leaves the devnet on stripCiphertext=false.

🟡 new scripts vs the sweep — documented them as manual standalone proofs in their headers (the "explicitly manual-only" option); the deposit one arms a global chain param, so it's deliberately kept out of the always-on sweep.

🟡/💡 DRY — extracted scripts/devnet-lib.sh (sourced): centralizes node_token/node_port/api/code_of/body_of/field. The four scripts source it and drop their local copies; per-suite curl timeouts preserved via DKG_API_MAXTIME; each keeps its own prefixed log/ok/bad.

Verification: bash -n + shellcheck clean across the lib and all touched scripts. These need a live 6-node devnet to exercise, so they're static-checked, not fleet-run this round (flagged in the commit messages too).

(v10-rc-validation.sh in the diff is intentional, unrelated rc19 work — NO_DATA_IN_SWM treated as expected post-RFC-49, and the SKILL.md check hardened.)