HyperBEAM Mainnet & Native AO Migration - #186
Open
jim-toth wants to merge 125 commits into
Open
Conversation
Migrate operator-registry off the dead AO legacynet onto self-hosted
HyperBEAM as a native lua@5.3a process. This is the validated pilot for
the D26 native contract shape; the reward contracts follow.
Runtime (runtime/native.lua): infallible compute trampoline (atomic
revert, outbox, runtime-owned ACL roles, Owner set-once, dispatch);
base-addressed state (now/state/<key>); computed views (now/as/<view>).
Migrate-on-spawn seeds both base.state and base.acl.roles from the
module's declared initial state/acl. Carries the A16 strip-on-write fix
(silent write-loss on string-valued state maps grown across slots).
Contract (src/contracts/native/operator-registry.lua): behavior frozen
from the legacynet contract, reshaped to { state, actions, views }.
Addresses now stored as EIP-55 (was 0x+ALLCAPS): on-chain keccak-256 +
EIP-55 validate-and-reject at every untrusted ingress (common/eip55.lua,
proven bit-identical to ethers.getAddress on luerl and the device VM).
common/bigint.lua replaces .bint for luerl-safe 256-bit math; the legacy
relay/staking contracts are updated to match (.common.bigint,
AnyoneUtils.parseInt).
Tests: 3-tier harness (spec/) — Tier-1 busted/Lua-5.3 92/0, Tier-2 luerl
1.3.0 141/0, Tier-3 live v0.9-FINAL node 66/0. Real-seed migrate-on-spawn
validated on-node against the 2026-07-09 dump (~1MB: 7932 verified / 2940
claimable / 1088 hardware): full seed-diff byte-identical, roles seeded,
EIP-55 lookups + post-seed writes hold at scale.
Tooling (scripts/): native HB client (hb-client.ts), by-id module
publish, seed builder (build-seed.ts), tier3 validators + A-series repro
probes, and the d6-conformance harness.
Second contract of the migration ported to the D26 native shape, with the frozen reward math proven byte-identical to legacynet all the way to Tier-3 on a live node. Contract (ao/src/contracts/native/relay-rewards.lua): - Complete-Round math copied verbatim from legacynet; only the wrapper reshaped (Handlers -> native actions, RelayRewards.X -> ctx.state.X, msg -> ctx). - EIP-55 at every untrusted address ingress; PreviousRound.Details off-persist (served from the Complete-Round output, never re-serialized into base.state) -> persistent state drops from ~4MB to ~740KB. - Staged Add-Scores + Complete-Round kept verbatim; delegate feature retained. Two on-device bugs the spike caught (invisible to Tier-1/2), both fixed: - A17 (luerl): a large positive-int table key (13-digit ms round timestamp) makes the device VM array-allocate and hang. PendingRounds is now keyed by tostring(timestamp). The legacy contract has this latent too. - A18 (runtime/native.lua): action handlers were handed metadata-laden base.state; a handler iterating a persisted map hit HB ao-types/commitments as data. Strip at compute start, symmetric with the view read path. Verification (all pass): - Tier-2 golden 12/0 (reproduces the exact bint golden under luerl). - Tier-3 on-node: 719KB real seed materializes byte-identical; a 300-fingerprint realistic round onto migrated balances -> 300/300 rewards + accumulation. - Legacy cross-check (tier2-relay-legacy-crosscheck.ts): the same realistic round through both the legacy Handlers contract and the native port yields 300/300 Details + Summary + TotalFingerprintReward byte-identical across every branch (hardware pool split, exit distribution, multi-fp proportional division, delegate split). Built-in negative control confirms the diff has teeth. - Durability measured: verdict GO for our sizes (moderate staged batches).
Per-behavior regression net (38 tests) on the native runtime under Lua 5.3, mirroring the legacynet WASM harness (test/spec/contracts/relay-rewards/*), re-expressed native: state read from base.state, a write's reply is the compute output, reads are native.view, an assert failure surfaces as output 'error: ...' and reverts atomically. Covers: ACL gating per action (owner/admin/named-role/denied), Update-Configuration + Modifiers/Multipliers/Delegates validation, Add-Scores per-field validation and validate-before-mutate atomicity, the A17 string-keyed PendingRounds, Set-Delegate set/clear, Cancel-Round, Complete-Round lifecycle (old-round pruning, summary persisted / Details off-persist on the output), reward RELATIONSHIPS (ratings, period, accumulation by fingerprint + EIP-55 address, delegate split identity), Claim-Rewards, all six views, and the D8 runtime-safety axes (unsigned/unknown action/atomic revert). Reward MAGNITUDES stay a Tier-2 concern by design: common/bigint is a native-integer wrapper, exact only where ints are arbitrary-precision (luerl/device); real Lua 5.3 is 64-bit and overflows at token scale. Tier-1 shrinks TokensPerSecond to assert relationships; the exact bint golden is reproduced at Tier-2 (spec/luerl/scenarios/native-relay-rewards.lua) and cross-checked vs legacy.
Recreate every legacynet WASM+TS harness case (test/spec/contracts/relay-rewards/*, 78 it() across 15 files) against the native port, split across tiers. Tier-1 busted (spec/native/relay-rewards_spec.lua): grown to 79 tests covering all validation (add-scores per-field, configuration, modifiers, multipliers, delegates), ACL (admin + named role per gated write), round lifecycle (cancel/complete pruning + last-round metadata), and the reward MAGNITUDE cases recreated at the harness's small TokensPerSecond so the exact expected integers (ratings 110/70, uptime 100/700/675, hardware 65-35 split 186/373, exit 660, delegate 2240/3360, accumulation 558/930/3631/3259/5808 -> 3918/6530/5871, claim tracking) hold in 64-bit. Init import/reimport -> migrate-on-spawn seeded-base view round-trips. Full spec/native/ = 171/171 (op-registry 92 + relay 79). Tier-2 luerl (scripts/tier2-relay-datasets.ts): the 3 data-driven cases run the real captured datasets at real token scale through the native contract under luerl (arbitrary-precision) -- staging1 330fp / staging2 353fp / test1 420fp, all process to OK with full Details. Reward MAGNITUDES stay a Tier-2 concern by design (common/bigint overflows 64-bit at real token scale); the exact bint golden (12/0) and the legacy cross-check (300/300) already cover full scale. Two intentional deviations, asserted as such: Set-Delegate trusts the node-verified committer (ctx.from) as operator, not re-validated in-contract; no Init action (state import is migrate-on-spawn). Minor luerl finding noted: its parser rejects 4-equals long brackets ([====[); use [==[.
Third and final contract ported to the D26 native shape, with full parity against the legacynet WASM harness (108 it() across 13 files). Runtime — ctx.timestamp (ms): Contracts that record message time (staking Set-Share) need a native clock. The scheduler stamps the assignment with two DIFFERENT units: `timestamp` = erlang:system_time(millisecond), and `block-timestamp` = Arweave block time in unix SECONDS (literally 0 when the node runs mode: debug). hyper-aos maps os.time to the latter. We take `timestamp` (ms), matching legacynet msg.Timestamp exactly, and document the trap at the source. Verified on a live node with scripts/probe/timestamp-probe.lua: timestamp=1784989059945 (inside the host's own Date.now() window), block-timestamp=0. This resolves the open "os.time — verify" item in D8; the answer was that it is NOT fine. Contract (src/contracts/native/staking-rewards.lua): - Frozen reward math copied verbatim; only the wrapper reshaped. - Two-level maps throughout (Rewarded/Claimed/Details[hodler][operator]); the operator's own cut lives at the self-key Rewarded[operator][operator]. - Details ARE persisted here (state is ~322KB total vs relay's 4MB), so Last-Snapshot/Last-Round-Data stay plain views. - A17: PendingRounds keyed by the string timestamp. - EIP-55 at every untrusted ingress, wrapped so legacynet error strings survive. Share-change delay unit fix (deliberate, not a math change): legacynet compared `RequestedTimestamp + ChangeDelaySeconds <= roundTimestamp` where both timestamps are MILLISECONDS but the delay is denominated in SECONDS, so the 7-day (604800s) default elapsed in 604800ms ~= 10 minutes -- with hourly rounds, at the very next round. Fixed by converting the delay to ms. Safe to fix in place: the feature is dormant (live config SetSharesEnabled=false, Shares and PendingShareChanges both empty, no Set-Share in the live message tail, and no Set-Share caller in any consumer repo). Tests: - Tier-1 busted spec/native/staking-rewards_spec.lua: 114 tests, all 108 WASM cases recreated (the suite's largest asserted value is 7.5e13, inside 64-bit, so unlike relay nothing needs Tier-2 for magnitude) + D8 safety axes. Full spec/native/ = 285/285, stable over 10 consecutive runs. - Tier-2 luerl spec/luerl/scenarios/native-staking-rewards.lua: 29/29 on the real device VM using REALISTIC 13-digit ms timestamps, which is what exercises A17. - New realistic-millisecond delay cases at both tiers. The legacynet harness only ever used toy integers (request@1000, round@2000, delay 1000) which pass under BOTH the buggy and fixed arithmetic -- that is precisely how the unit bug survived. The new cases fail on the old arithmetic and pin the fix. Also fixes a latent flaky assertion in both reward specs: comparing json.encode(map) strings is order-dependent because Lua pairs() order is not stable; compare deep copies structurally instead.
Units were only half the question. Traced where req.timestamp actually comes
from: it is the SCHEDULER'S LOCAL WALL CLOCK -- HyperBEAM's own comment reads
"Note: Local time on the SU, not Arweave" (dev_scheduler_server.erl:231). It is
not the data item's timestamp (ANS-104 items carry none, so a sender cannot set
or forge it) and not Arweave block time.
Guaranteed: committed into the signed assignment, so it is fixed at assignment
time and identical for every CU replaying the schedule -- execution stays
deterministic.
NOT guaranteed: monotonicity. erlang:system_time follows the OS clock and an NTP
correction can step it backwards, and the scheduler performs no ordering
validation on it. So message timestamps do not reliably order messages.
This matters because staking's share-change delay gates RequestedTimestamp (SU
wall clock) against Round-Timestamp (the controller's clock) -- two independent
clocks, so host skew shifts the effective delay. That is inherited from legacynet,
which compared msg.Timestamp against Round-Timestamp the same way, and is kept
deliberately to keep the port faithful. Round-Timestamp is by contrast
contract-enforced monotonic via the backdating assert, and the alternative design
(start the delay at the first Complete-Round after the request and measure purely
in round time) is documented for revisit before SetSharesEnabled is ever turned on.
Documented at every point of use: runtime/native.lua where ctx.timestamp is
derived, the staking contract header ("ACCEPTED LIMITATION") and its Set-Share
site, and the D8 port-safety checklist.
No behaviour change: comments only. Tier-1 285/285, staking Tier-2 29/29.
The staking counterpart of tier2-relay-legacy-crosscheck.ts, closing the same
weak link. The native reward math is a verbatim frozen copy of legacynet
Complete-Round, so the residual risk is a transcription typo -- and the busted
and luerl specs cannot see it, because they assert the port against itself.
This drives the same round through BOTH contracts under luerl (legacy in `run`
mode against the StakingRewards global, native in `native` mode against
base.state) and diffs the parsed results.
Inputs are real, from the 2026-07-09 live dump: the deployed Configuration, 300
(hodler, operator) pairs drawn from Rewarded, and the full 562-hodler Rewarded
and 400-hodler Claimed maps as priors -- so the cumulative bigint add lands on
migrated balances rather than zeros, and every restaked branch fires from real
data (Rewarded-Claimed, Rewarded alone, and no prior at all).
Three runs:
A live configuration verbatim. 373 pairs -- 92 self-pairs (the operator
own-cut key), 92 multi-operator hodlers, 215 with a real Claimed prior, 73
fresh, 34 below the Running gate and 34 exactly at it.
373/373 pair Details identical, Summary/Period/Rewarded/Claimed identical,
309 hodlers accumulated off their seeded prior.
B SetSharesEnabled with per-operator Shares {0, 0.1, 0.5, 1.0}. The live
config has share-setting disabled, so run A cannot reach this branch at
all. 373/373 identical; shares 0, 0.05 (the Default fallback), 0.1, 0.5 and
1.0 all exercised in-round, including both edges (139 pairs at a 100%
operator cut, 65 at 0%).
C the one INTENDED deviation, asserted rather than assumed: legacy compares
ms + SECONDS against ms, so a configured delay elapses ~1000x early; the
port converts to ms. Legacy applies a 1-hour delay after 60 seconds of
round time, native holds. C fails in BOTH directions -- if native applies
early the fix has regressed, and if legacy holds the fixture has stopped
reproducing the bug and proves nothing.
Negative control (PERTURB=1, native TokensPerSecond +1 wei) catches 114
mismatches across Details, Summary and Rewarded, so the diff has teeth.
Address encoding is the other deliberate deviation (legacy 0x+ALLCAPS, native
EIP-55): priors are seeded to each side in its own encoding and keys are
lowercased before diffing, so the math is compared apples-to-apples. Encoding
fidelity is covered separately by validate-address-migration.ts.
Timestamps stay small (roundLength 60) to keep the legacy contract clear of its
own A17 large-int-table-key hang; roundLength is a scalar multiplier and changes
no branch, and intermediates still exceed 64-bit by ~20 orders of magnitude.
Run C picks a 1-hour delay for the same reason -- the live 7-day value would
need 13-digit ms timestamps to straddle the two readings. The realistic-ms cases
live in the native Tier-1/Tier-2 specs, where no legacy contract is present to
hang.
Compares parsed structures, never json bytes: Lua pairs() order is not stable.
Closes the last gap for the third contract: the native staking-rewards port now
materializes the real 322KB legacynet state on a live HyperBEAM node and settles
a realistic round byte-identically against an independent luerl oracle.
build-staking-seed.ts -- live dump to native seed (migrate-on-spawn). EIP-55 at
BOTH levels of the two-level maps (Rewarded/Claimed/PreviousRound.Details) and
in Shares/PendingShareChanges; 3511 addresses converted, none rejected. Unlike
relay, PreviousRound.Details is KEPT: staking's whole state is 322KB, so there
is no size pressure and seeding the final legacynet round preserves
Last-Snapshot/Last-Round-Data across the migration. The transform asserts it is
total -- a dropped key would silently shrink the migration, so hodler and pair
counts must match the dump exactly on the way out.
build-staking-oracle.ts -- the parity oracle. Relay needed a minimal config-only
oracle because its 800KB bundle times out in luerl; the staking bundle is 402KB
and parses in under 2s, so the oracle runs the SAME full seed, with the SAME
562-hodler Rewarded and 400-hodler Claimed priors, that the node spawns from.
The comparison is whole-state rather than "fresh addresses accrue the same". It
also refuses to emit a degenerate round (wrong Period, zero Rewards, or a Details
pair count that does not match the input).
util/staking-round.ts -- the single definition of the round, imported by both the
oracle and the node driver, with a timestamp equality guard so the two provably
cannot drift. 250 real pairs drawn from the seed plus 25 fresh ones, so restaked
is computed against real migrated balances (182 pairs with a Claimed prior ->
Rewarded-Claimed, the rest -> Rewarded alone, fresh -> 0), 74 self-pairs, and 56
pairs at or below the Running gate. Timestamps are realistic 13-digit ms
(seeded PreviousRound.Timestamp + 1h): a large integer round timestamp is what
hung the device VM under A17, so Tier-3 must use one.
tier3-staking-validate.ts -- ALL PASS on hb-tier3:
* seed materializes: 562/400 hodler counts, config, roles, PreviousRound
* full seed-diff pair-by-pair -- Rewarded 865 pairs, Claimed 594, and the
persisted Details 636, all identical
* the legacynet read surface answers off migrated state: last_round,
last_snapshot (carrying all 636 Details pairs), last_round_data, rewards,
claimed, plus a base-addressed point read
* the round settles identical to the oracle: Period, Summary, all 275
per-pair Score/Rating/Reward, and the whole 587-hodler/915-pair cumulative
Rewarded map; Claimed untouched; PendingRounds cleared; 215 hodlers moved
off their migrated priors, and Rewarded grew by exactly the 25 fresh
operators' own-cut self-keys
* Add-Scores 7.1s / Complete-Round 4.0s for a 33KB payload
PERTURB=1 shifts the oracle by one wei and is caught on all three surfaces, so
the whole-map comparison is not a no-op.
Comparisons are structural, never JSON strings. The node re-encodes state from
Lua tables and pairs() order is not stable, so byte-different encodings
routinely carry identical data -- the first run of this script reported four
false failures for exactly that reason. Same trap as the reward specs and the
relay cross-check; noted at the helper so it does not return a fourth time.
Tier-1 spec/native/ still 285/285.
scripts/spawn.ts was written for the genesis-wasm era and carried exactly the defaults that caused this migration: HB_URL fell back to https://push.forward.computer (FR-operated), MODULE to a hardcoded genesis-wasm id, and DEPLOYER_PRIVATE_KEY to a hardhat test key. It spawned, slept 5s, then Eval'd the contract source as aos. Moved to scripts/legacy/ and kept runnable as `deploy:legacy-genesis-wasm`; `npm run deploy` is now the native path. scripts/deploy.ts has no fallbacks anywhere. HB_URL, DEPLOYER_PRIVATE_KEY, MODULE_ID and --seed are each required, and --seed has no default because "which state does this process start from" is not a question to guess at. It also refuses outright to deploy against a forward.computer / ao-testnet / arweave.dev host rather than trusting whatever HB_URL says -- the no-FR-infra constraint checked in code, not just written down. Two phases, because they need different credentials. --publish-cmd emits the node-host command that commits the module signed by the NODE wallet: that writes it into the node's local cache so the spawn resolves immediately, rather than waiting on Arweave gateway propagation as a deployer-wallet Turbo upload would. The spawn phase then runs anywhere, signed by the deployer wallet that must be in the node's faff allow-list, with scheduler-location and authority explicit (H1/H2/H3 from the blast-radius audit are all "a default nobody set"). Seeds come from the existing build-{,relay-,staking-}seed.ts builders, invoked rather than reimplemented -- they are what the Tier-3 validations ran against and they assert their own totality. The deploy VERIFIES rather than trusting a 200. Migrate-on-spawn happens on first compute, so a broken seed cannot surface any earlier: it polls the status view and compares the materialized counts against what the seed builder emitted. Writing that caught a gap -- each contract's status view names its counts differently (op-registry claimable/verified/blocked vs the reward contracts' addresses/fingerprints/rewardedHodlers), and the first version would have run ZERO checks on a seeded op-registry deploy and reported success, since undefined === undefined passes. The mapping is now explicit per contract, and a deploy that can verify no count at all fails rather than passing quietly. Verified end-to-end on hb-tier3, all three contracts from the real live dump: operator-registry claimable 2940, verified 7932, hardware 1088, blocked 0, credits 0 -- all 5 counts matched relay-rewards addresses 763, fingerprints 9750, claimed 530 staking-rewards rewarded 562, claimed 400 Owner came back as the deployer wallet in every case. Guards confirmed firing: missing HB_URL, an FR host, and a missing --seed each abort with an actionable line rather than a stack trace. Transport is @anyone-protocol/ao-client (git dep, v0.1.1), its first real consumer -- and the only one that exercises spawnProcess.
scripts/verify-migration.ts checks the chain from manifest hash to live state: dump files still hash to the recorded sha256, the seed transform is deterministic, on-node state deep-equals the transform output, and roles are restored. D12 asks for a hash of post-init state against the manifest. That predates the EIP-55 decision and is unsatisfiable now -- native state is a deliberate transform of the dump, so the bytes cannot match. Checking every link instead; rationale is in the script header. D13: "no replay required" was a human determination in a doc. Now machine- checked -- every tail message must predate the dump, and none may have failed except known-benign ones. Also wires D10's negative fixtures: the staking tail's three "No rewards for 0xFB5BA08D..." failures, replayed against the live process, must still fail closed. All three processes, 49 checks, ALL PASS on hb-tier3.
The score generator varies hardware/exit/uptime-tier, but only the delegate
branch was ever asserted to have fired. "Both sides agree" stays true even if a
branch stops firing entirely -- a config change (Modifiers.Hardware.Enabled=
false), a different tier table, or an edit to the generator would silently shrink
what the test proves while still reporting 300/300.
Same trap that made the staking share run look green while using only half its
share values.
Now reports and requires: hardware 100, exit 60, uptime 300, tiers {0,3,14,45}.
Verified it fails when a branch cannot fire (K=3 yields 3 of 4 tiers -> exit 1
with zero math mismatches).
Reads back the policy the node actually loaded rather than reviewing the jobspec, probes the nginx edge from outside, and asserts faff denies a non-allow-listed signer. 74/74 across all three; PERTURB=1 self-test.
Closes D3's rate-limiting/body-cap clause. --dos is opt-in and dev-only by default since it generates real load. Asserts the probe actually exceeded 30 req/s first: an earlier form reached only ~17 req/s against a large payload and would have 'passed' without exercising the limiter.
An arbitrary (non-allow-listed) wallet is admitted past faff for the whitelisted ids and refused for a foreign one — by faff on dev, at the edge on stage/live. 80/80.
Legacy asserted 'timestamp is integer'; the native specs only fed non-numeric values, so parseInt's integer-only strictness (the A12 workaround) was untested. Relay also lacked the missing-tag and empty cases staking had. Verified to fail when fed a valid integer.
Tier-2 invocations are non-uniform (three runner modes, different arities) and scenarios/ also holds a Tier-3 oracle that is not a test and spins for hours under A17 if run wrong. Encode all of it in spec/run-tier2.sh rather than leaving it tribal; a scenario producing no result line is an error, never a pass. ao-test.yml runs Tier-1 and Tier-2 as separate jobs so a failure names the layer. The legacy mocha suite runs non-blocking: it is a coverage reference for the native specs, not a gate.
Fork PRs execute contributed code (two image builds + a bun install that runs their lifecycle scripts). The read-only token and absent secrets mean that is not a credential risk, but it is free compute now and RCE if this ever moves to a self-hosted runner — so gate every job on the PR originating from this repo. Also state permissions explicitly rather than inheriting them, and pin bun instead of tracking latest.
The aos-WASM harness is obsolete under the Lua device and the legacynet contracts now carry luerl workarounds their specs never got. Keep the specs in the repo as a coverage reference; stop executing them.
The E2E assertions already existed across the tier3-* verticals; what was
missing was a way to run them as a suite. Each vertical needed a different
module id from a manual publish, in an order written down nowhere — the same
problem run-tier2.sh solved for Tier-2, with the same failure mode of a
mis-invoked or skipped stage reading as success.
scripts/run-e2e.ts: preflight -> artifacts -> module registration -> the four
verticals -> verify-migration x3 -> aggregate. Skipped stages are reported as
skipped rather than counted, failing stages dump full output to dist/e2e-logs/,
and a vertical that exits 0 without spawning is an ERROR, not a pass.
19/19 green from a clean tree against a local v0.9-FINAL node.
scripts/build-relay-probe.ts: dist/relay-oracle-probe.json was read by
tier3-relay-validate.ts and written by nothing, so a fresh checkout could not
run the relay vertical at all. Staking's equivalent already self-automated.
Guards that the oracle actually computed its three fingerprints with non-zero
rewards — otherwise the parity check passes vacuously against a node that also
did nothing.
hb-client.ts: correct the moduleIdFor comment. It claimed an inline spawn
leaves the module child in the node's cache so a later by-id spawn resolves it.
It does not, and it is expensive to believe: the by-id spawn is ACCEPTED and
only fails at compute time with {case_clause,{error,not_found}}. Verified
refuted on v0.9-FINAL at both ~200B and 806KB, including after forcing a
compute on the inline process first. A spawnable module must be a signed ans104
module message written to the node cache via bin/hb eval.
verify-migration gets a FRESHLY spawned process rather than its vertical's:
the verticals mutate state by design (relay and staking drive a full round),
so reusing one fails looking like a migration defect — extra reward keys, a
moved PreviousRound.Timestamp — which is the kind of red that gets "fixed" by
weakening an assertion.
CI: an e2e job on push + workflow_dispatch, not pull_request. It needs
bun install, which re-introduces the lifecycle-script surface that dropping
legacy mocha removed from the PR path, and Tier-1/Tier-2 already gate contract
logic there.
The stage-node run stays re-homed to D14.
publish-native-module.ts's bin/hb eval flow cannot target a self-hosted bundler: `bin/hb eval` starts a separate VM that never boots the hb application, so it loads NO config and always resolves bundler-ans104 to the compiled-in default. Verified — with config set to http://127.0.0.1:9999 an eval still reported <<"https://up.arweave.net:443">> and uploaded there. So publishing moves client-side over HTTP: sign the ans104 item here, POST to <bundler>/~bundler@1.0/tx. The bundler becomes a URL rather than a deployment assumption, and the same script works against up.arweave.net or our own node. The verification is the point. A bundler answers 200 with a signed receipt as soon as it has QUEUED an item, and settlement takes hours — so a receipt proves nothing, and the gateway DATA endpoint is actively misleading (it serves 200 from the optimistic cache before settlement). Only the GraphQL index distinguishes queued from persisted, which is also exactly what node-side resolution needs: hb_store_gateway -> hb_gateway_client:read/2 goes via /graphql. --verify-spawn additionally proves a cold node can resolve the id, which settlement alone does not establish. Three ids are in play and confusing them turns a successful publish into an apparent silent failure: - signed ans104 item id = what is uploaded, indexed, and spawnable - hb_util:id(Msg) = local-cache id, NEVER uploaded - bundle tx id = the L1 tx carrying the item, not the module This script tracks the first and says so in its output. BUNDLER has no default: publishing to up.arweave.net should be a deliberate choice, not the consequence of an unset variable.
… probe
Settlement takes hours, so blocking on it is the wrong workflow. --manifest
records {file, id, bytes, publishedAt} and --recheck re-reads it later,
reporting largest-settled vs smallest-pending so a free-tier boundary shows up
as the size at which settlement stops. The manifest is written BEFORE the wait:
if the wait is interrupted the ids must still be recoverable, or the upload is
unverifiable and effectively lost. --wait 0 publishes without waiting at all.
build-size-ladder.ts emits inert, self-describing lua probes at 100KB..10MB.
They are permanent if they settle, so each carries a header saying it is a
capacity probe and not a contract.
Already answered by publishing the ladder: up.arweave.net caps requests at
EXACTLY 5 MiB (5,242,880 accepted, 5,242,881 -> 413, nginx client_max_body_size
5M), measured across the whole ans104 item. That is a TRANSPORT limit, not a
billing one. Bisected with deliberately invalid payloads so nothing was
persisted to find it: over the cap nginx 413s on size before parsing, under it
the bundler rejects the malformed item.
Our largest module (operator-registry-seed, 1,077,072 B) is at 20% of the cap.
Whether items settle for FREE at these sizes is still open and needs the
recheck; the ids are recorded outside dist/ because dist/ is gitignored.
GraphQL indexing only means a gateway saw the item. It does not mean anything was mined, and the gateway data endpoint returns 200 from the bundler's optimistic cache the whole time, so neither signal distinguishes queued from persisted. ANS-104 data items are L2: they live inside a bundle and L1 knows nothing about them, so /tx/<dataItemId>/status returning Not Found is CORRECT and says nothing either way. Bundles also nest -- bundledIn can point at an intermediate bundle that is itself a data item. Only the ROOT of that chain is an L1 transaction, and only its confirmation count is evidence of mining. finality() walks bundledIn to the root and reports the root's L1 confirmations. Validated against two references: legacynet item -> depth 2, root oYcUSTh3..., 151,436 confirmations our 66KB module -> depth 1, root O9K653PJ..., 96 confirmations Caveat kept in the code: a confirmed L1 tx carries a data_root, but chunks are uploaded separately (dev_bundler's third phase, post_proof). A tx can be mined while its chunks were never seeded, so confirmations plus repeated successful retrieval over days is the real bar -- a single check right after publishing proves neither. Hence --recheck against a manifest rather than a one-shot wait. Baseline at 0.2h after publishing: 0/8 indexed, all showing the misleading data-200.
The publish phase had a worse problem than publishing to the wrong place.
MODULE_ID came from hb_util:id(Msg) after a node-host hb_cache:write — a
NODE-LOCAL id. The id Arweave indexes is the signed ans104 item id, a different
value. So a process spawned this way had a module living in exactly one node's
cache: a rebuilt node, a fresh alloc, or the second node of the redundancy pair
could never resolve it, and the process could never compute another slot.
Nothing about the process reveals this; it surfaces at cold start, which is
precisely what D5 is meant to guarantee against.
The eval also ran hb_client:upload "for durability". It did not: `bin/hb eval`
starts a VM that never boots the hb application, so it loads no config and
always resolves bundler-ans104 to the compiled-in default regardless of what
the node is configured with (verified 2026-07-27). Removed rather than left as
a false comfort.
Now:
--publish delegates to publish-module.ts — signed client-side, posted to
BUNDLER, settlement verified. One implementation of publishing
rather than a second copy that can drift. BUNDLER has no
default.
--publish-cmd still available for fast local iteration, but prints the
node-local caveat instead of implying durability.
And the spawn phase now refuses to deploy against a module that is not indexed
on Arweave, since that is the last point where the mistake is cheap to fix.
--allow-unpublished-module opts out for throwaway test deploys.
Verified: node-local id refused with actionable guidance; settled id passes
("module durability on Arweave (indexed)"); escape hatch warns and proceeds.
A long-lived branch with an open PR fires both `push` and `pull_request` for the same commit, so tier1 and tier2 were running twice on every push to dev-hyper-aos-migration. Deduped with a concurrency group keyed on the commit rather than the event: github.sha on a push and pull_request.head.sha on a PR are the same commit, so both runs land in one group and the second cancels the first. That dedup only works if every job behaves the same for both event types, so e2e loses its blanket `!= 'pull_request'` and takes the same fork guard as the tiers. Otherwise coverage would depend on which run won the race — if the PR run survived, e2e would silently never run, which is worse than running twice. Dropping the blanket exclusion is also the better policy on its own. The concern was that `bun install` re-introduces a contributor-controlled lifecycle-script surface, but that is a FORK problem and the fork guard addresses it precisely; excluding all PRs additionally meant same-repo PRs got no integration coverage at all.
Simpler than the concurrency-by-commit trick in 899c960, and Jim's call. Listing the working branch under `push` meant every commit fired both `push` and `pull_request` once a PR was open. Deduping that let both runs start and then cancelled one -- burning runner minutes and leaving cancelled runs in the UI that read like failures. Not firing the second event at all is strictly better. Branch coverage now comes from the PR, which is where the result is needed anyway. Keeping e2e's fork guard from 899c960, which this makes necessary rather than merely tidier: with `push` main-only, a PR is the only event a working branch fires, so the old blanket `!= 'pull_request'` would have meant e2e never ran until merge. Concurrency stays, retargeted to its actual job: superseding an in-flight run when new commits land on the same PR. It is no longer what prevents the double-run.
48d4cb5 carries the strengthened assertModuleIsDurable: a module id must be indexed, its containing bundle mined, at least 50 confirmations deep, and its bytes must sha256-match the bundle the image builds. Safe because 8e02154..48d4cb5 changed zero Lua, only TypeScript tooling, docs and jobspecs. The image therefore builds byte-identical bundles to the one that published the modules these jobs spawn, so the new sha256 check still passes. Diff for .lua changes before any future repin: a Lua-changing repin fails this check closed. Stage stays on 0e1566b, the record of what actually ran.
The fixture dated from 575df19, captured before the flattening landed later the same day. Six contract commits have changed source since. Re-captured against the production node image, sample held fixed: 182 of 186 view responses byte-identical, 2 differing only by an added Network key (4eaeda7 per-operator relay counts, purely additive), and 2 status views differing only by the already-approved drop of 2 empty legacynet rows. No regressions. The sampling guard was on the wrong path. sampleAddresses() is index based over the seed and drifts when the seed shape changes, and the reuse-the-golden's-addresses guard applied to --check only. So a plain re-capture silently rotated 13 of 21 addresses, and a whole-file diff then reported 212 differences that were only different addresses being compared. Re-baselining is exactly when the sample must be held fixed, because continuity is what the fixture is for. Reuse now applies in both modes; --resample opts out deliberately. Network is captured as [], since buildRound supplies no counts. The golden records that the key exists, not what it carries.
WS-6 runs publishing on our own bundling, which points bundler_httpsig at our own node. dev_bundler_task signs and pays a real L1 transaction from the node's wallet, so an accepted item is a charge against it. Dev's wallet holds 28.5 AR and dev's edge is deliberately public, which makes faff the only thing between that wallet and anyone who can POST. Three checks: a never-allow-listed ephemeral signer must not get an item accepted, an unsigned item must be refused by verify_message, and no p4-non-chargable carve-out may exist for the bundler path, since a carve-out would ungate spending entirely. The signer check is one-directional on purpose. A disabled or unrouted bundler refuses for its own reason and still passes, because the property under test is that strangers cannot make us pay, not that the bundler is on. Acceptance is the only failure. Passing on dev, stage and live.
D21 made assignments reach Arweave, but the published chain is anchored to
nothing: its earliest predecessor hashpath names an assignment that predates
the fix and is absent from the network. A snapshot carrying its slot and that
slot's assignment id is what anchors it.
Four tools, all self-tested against stage and live:
snapshot-state bracketed capture (slot/current, dump, slot/current again)
because a view CANNOT be pinned to a slot - both spellings
fail silently, one returning an empty state
verify-snapshot gzip, sha256, shape, anchor, chain contiguity, hash links;
PERTURB=1 self-test; 0-link chains report warn, never pass
publish-snapshot direct L1, dry-run by default, refuses UNANCHORED snapshots,
idempotent on (process, slot), loud on a sha mismatch
recover-from-arweave newest anchored snapshot, forward chain walk, message
bodies, and a provablyComplete verdict that gates the exit
Publishing runs as a periodic batch job per environment rather than a sidecar
in the hyperbeam group: a sidecar shares the node's lifecycle, so a snapshot
failure could restart the scheduler.
Two things this does not do, both stated in the docs: the node snapshots but
does not upload on its own, and stock HyperBEAM still cannot cold-boot a
process from Arweave.
arweave moves from a transitive dependency to a declared one. The Dockerfile
allowlist names every file, so the four scripts are added there too.
Every piece of evidence for "can this image run stage and live?" already existed and nothing assembled them, and nothing recorded what a good result looks like - so "did this version regress?" was not mechanically answerable. A VERSION bump is now on the table for D24, which makes that the blocker. Qualifies an IMAGE, not a deployment: boots a container from the image under test, runs 12 phases, tears it down, compares to spec/fixtures/node-baseline.json. About four minutes. --url mode covers what only a real environment can answer. The load-bearing decision is what gates. Absolute latency does NOT: the probes' own headers say the numbers move with host load, and local figures came out at roughly half stage's. Only shape gates - growth ratios, scaling ratios, counts and identity - which is also where the failures we have actually seen land. The bands live in METRICS in the script, not in the fixture, so widening one is a reviewable diff. Two phases are new because nothing covered them. toolchain fingerprints the applied patches out of the image's own git tree (the Dockerfile leaves /app dirty, so `git diff` IS the patch set) and hard-fails if the image's luerl drifts from the version run-tier2.sh pins. config boots a second container on a production-shaped opt surface and asserts every key reads back, including the nested p4 entry - the standing regression test for opts that are silently not-found rather than an error. fingerprint answers the deployed-node question as far as it can be answered: the build-default opt surface and the device surface, both of which survive the locked edge. Verified against stage and live, which match the blessed baseline exactly. It proves the same upstream version family, not the same image, and says so rather than implying more. Baseline recorded against sha256:7e4f2d1e, the image stage and live run: tier-2 265/0, verticals 13/0, golden 0 diffs, gc.growth ~1.0x, trie scaling ~0.9x, restore byte-identical. Also fixes gc-restore-fidelity, which was reading now/state/claimable/<fp> - a pre-D32 address that resolves to nothing. It failed warm and reported "9 writes LOST" when every write had landed. Now reads the as/fingerprints view. Four traps this caught while being built, all guarded now: a .flat config has no comment syntax, so a `#` line containing a colon becomes a real opt named after the comment text; opts must be read via serialize~json@1.0, since under an accept header list opts are +link pointers and a healthy node looks like it dropped its access policy; serialize~json overwrites `device`, so the p4 entry is found by pricing-device; and the qualification node was publishing test assignments to the real up.arweave.net, 260 of them in one run.
Self-bundling puts the node's own 43-char Arweave address on faff-allow-list: p4 evaluates the OUTER request when the node POSTs its own upload to ~bundler@1.0/tx, and the node signs that as itself, so without the entry it 400s its own bundle. Carved out by identity rather than by loosening the EIP-55 check, so a stray 43-char string that is not this node still fails. The entry is asserted to appear at most once. dev_faff has no per-route scoping, so the entry admits the node on every chargeable route. That is acceptable for our own scheduler but it is a real widening, which is why it is asserted explicitly rather than tolerated.
Qualifying vanilla edge surfaced three problems, all in the reporting rather than the measurement. edge has no `preloaded-devices` opt at all: upstream moved preloaded devices into a prebuilt LMDB store, which shows up as the new `preloaded-store`. The device enumeration is now best-effort and reports why it is unavailable instead of throwing, the surface records "(not enumerable)" rather than a sha of the empty list, and the device diff is suppressed when nothing could be enumerated. Without that last part the run reported all 61 baseline devices as GONE, which reads as a catastrophic removal rather than a changed opt. Whether a changed build surface FAILS now depends on the question. Qualifying a candidate image expects the surface to move, so it is reported like toolchain.* and the phase passes; asking whether a deployed node is the build you already blessed treats it as the answer no. Both print the same diff by name; only the verdict differs. A missing baseline metric no longer claims "phase passed but stopped emitting" when the phase actually failed. That is a consequence of the failure, not a second finding.
This question has been answered wrongly three times, so the probe is built around what made those answers wrong. The failure mode is that the node signs, prices and successfully MINES the bundle transaction, and only then every chunk POST is refused data_root_not_found. The transaction is real, paid for and indexed; its data is not there. So /tx/<id>/status and /tx/<id>/offset are useless, both looking healthy for a bundle whose data never landed. The probe reads GET /raw/<txid> and compares the served byte length to the transaction's declared data.size, because a short body is a failure too. Posting without `accept: application/json` returns the Hyperbuddy HTML page with HTTP 200, which has read as success twice before, so the response is asserted to be JSON. The bundle is found by watching the node's own Arweave address over GraphQL rather than its logs, which keeps the probe self-service and guarantees the transaction it judges is one this run caused. Transactions with data.size 0 are skipped: the node also signs plain AR transfers, and one of those returns 200 with an empty body, which would report a landing that never happened. A timeout is reported INCONCLUSIVE, not FAIL. "No bundle appeared" and "the bundle appeared and its data is missing" are entirely different findings and only the second says anything about the fix. Control, measured on dev before deploying the fix: of the ten transactions the dev node has ever signed, the two carrying data (2,000 B and 13,729 B) both 404 from /raw, while the eight plain transfers correctly return 200 with an empty body. The probe therefore demonstrably distinguishes a landed bundle from a lost one, on the exact node and wallet under test.
…er by config, not a fixed goal
…ot-consuming bypass on any path
… across the image crossing
…before cutover Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, so a gated node can be deployed to Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…able Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ocally first Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.