Publish snapshots through the node's own bundler instead of direct L1

Author: jim-toth

ao/docs/durability-and-recovery.md

@@ -77,7 +77,7 @@ the dependency D22 exists to remove.
 | script | what it does |
 |---|---|
 | `scripts/snapshot-state.ts <env>` | capture a consistent, anchored snapshot per contract |
-| `scripts/publish-snapshot.ts <dir>` | publish snapshots as direct L1 transactions (dry run by default) |
+| `scripts/publish-snapshot.ts <dir>` | publish snapshots through the node's own `~bundler@1.0` (dry run by default) |
 | `scripts/verify-snapshot.ts <dir>` | verify payload, anchor, chain continuity and message retrievability |
 | `scripts/recover-from-arweave.ts <pid>` | reconstruct state + ordered history from Arweave alone |
 
@@ -121,29 +121,34 @@ and the snapshot never touches the write path.
 `periodic { cron = "@daily", prohibit_overlap = true }`. Overlap prohibition plus the
 (process, slot) dedupe below means a double-post is unreachable even if a run wedges.
 
-Snapshots go to Arweave as **direct L1 transactions**, not through a bundler - including not
-through our own `~bundler@1.0`. Three reasons, in order of weight:
-
-1. **Our `~bundler@1.0` is broken.** It signs, prices and mines the bundle transaction, then
-   every chunk POST returns `400 data_root_not_found`: `building_proofs` computes a `data_size`
-   that does not match the bundle it just posted a header for, so the merkle root describes a
-   payload that does not exist. Structural in the `SignedTX -> structured@1.0 -> tx@1.0` round
-   trip between `post_tx` and `build_proofs`, not payload-specific - snapshots would fail the
-   same way while still being paid for.
-2. **A bundled item is only queryable by tag if a gateway chooses to unbundle it.** A direct L1
-   transaction is indexed natively with its tags. Recovery finds snapshots by
-   `tag process=<pid>, type=state-snapshot`, so bundling would make durability depend on gateway
-   policy for no benefit.
-3. **The saving is 0.348%.** Measured 2026-08-25: three separate transactions cost 0.0205867 AR,
-   one bundle of the same bytes 0.0205150 AR. Arweave prices per byte; bundling amortises many
-   small items, and three ~1 MiB blobs are the opposite shape.
-
-If `~bundler@1.0` is fixed, revisit (2) before (3) - the unbundling dependency is the real
-objection, not the fee.
-
-Measured 2026-08-25:
-
-| contract | state | gzipped | L1 cost |
+Snapshots go to Arweave through **the node's own `~bundler@1.0`**, the same path every
+scheduled message and assignment already takes since D24. The point is to operate, fund and
+monitor one upload mechanism rather than two.
+
+This was decided against direct L1, which the tooling originally used, and the argument that had
+to be answered was real: recovery finds snapshots by GraphQL **tag query**
+(`tag process=<pid>, type=state-snapshot`), and a bundled data item is indexed only if a gateway
+chooses to unbundle it, whereas an L1 transaction is indexed natively. Verified against live on
+2026-08-28 and the concern does not hold in practice - items inside our own node-signed bundles
+are tag-discoverable on arweave.net, and `transaction(id:)` reports a block height for them,
+which is what the publisher's settlement wait depends on.
+
+Two consequences are worth stating plainly rather than discovering later:
+
+- **The node pays.** `PUBLISH_JWK` signs the data item and holds no AR. The wallet to keep
+  funded is the node's, which already pays for every assignment it publishes.
+- **Acceptance is not settlement.** The bundler batches on an idle flush, then mines, then the
+  gateway indexes - minutes, not seconds. A run that ends with items accepted but not yet
+  indexed is normal and exits 0. Confirming they actually landed is D25's job, and it has to
+  cover snapshots and not only assignments, because snapshot durability now shares a failure
+  domain with the node's upload queue.
+
+The fee was never the deciding factor either way: bundling the same bytes saves 0.348%
+(measured 2026-08-25, 0.0205867 AR as three transactions against 0.0205150 AR as one bundle).
+
+Measured 2026-08-25 - now an estimate of what the node pays, not a direct charge:
+
+| contract | state | gzipped | cost |
 |---|---|---|---|
 | operator-registry | 0.99 MiB | 0.36 MiB | 0.0059 AR |
 | relay-rewards | 4.02 MiB | 0.96 MiB | 0.0117 AR |
@@ -161,8 +166,10 @@ If a published snapshot exists for the same (process, slot) but its `state-sha25
 that is not a duplicate - one slot produced two different states, which is a correctness problem.
 The publisher reports it and exits non-zero rather than skipping silently. `--force` overrides.
 
-`PUBLISH_JWK` is an **Arweave JWK** that signs and pays. It is not the EVM key used by
-`publish-module.ts`, which signs ANS-104 items for a bundler and holds no AR.
+`PUBLISH_JWK` is an **Arweave JWK** that signs the data item and pays nothing. `BUNDLER` says
+which node takes the upload and defaults to `http://$SNAPSHOT_HOST`, the in-cluster address the
+periodic job resolves from Consul. That default matters: `~bundler@1.0` is refused at the edge
+on stage and live, so the upload only works from inside the cluster.
 
 ### Verifying
 

ao/scripts/publish-snapshot.ts

@@ -1,17 +1,26 @@
 /**
- * D22 - publish state snapshots to Arweave as DIRECT L1 transactions.
+ * D22 - publish state snapshots to Arweave through the node's own ~bundler@1.0.
  *
- * Why L1 and not a bundler
- * ------------------------
- * Bundling exists to amortise many small items into one transaction. A snapshot is a
- * single ~1 MiB blob, so bundling buys nothing and costs us three problems:
- *   - up.arweave.net is Forward Research infrastructure, and getting our durability
- *     path off it is the entire point of WS-6;
- *   - its 5 MiB ceiling is a POLICY limit we are already close to (live relay-rewards
- *     is 4.02 MiB uncompressed and grows with fingerprints);
- *   - our own ~bundler@1.0 is blocked on a proof/header size mismatch.
- * A direct L1 transaction has none of those. Measured 2026-08-25, a full round of all
- * three live contracts is ~1.54 MiB gzipped and costs 0.0205 AR.
+ * Why our own bundler and not direct L1
+ * -------------------------------------
+ * WS-6 exists to get our durability path off Forward Research infrastructure, and D24
+ * closed that: all three nodes sign and pay for their own bundles. Every scheduled message
+ * and assignment already reaches Arweave that way, so publishing snapshots through the same
+ * path leaves ONE upload mechanism to operate, fund and monitor instead of two.
+ *
+ * The objection this had to clear is that recovery finds snapshots by GraphQL TAG QUERY
+ * (recover-from-arweave.ts), and a bundled data item is indexed only if a gateway chooses to
+ * unbundle it. Verified against live 2026-08-28: items inside our own node-signed bundles ARE
+ * tag-discoverable on arweave.net, and `transaction(id:)` reports a block height for them,
+ * which is what the settlement wait below depends on.
+ *
+ * Consequences of the handoff, all real:
+ *   - the NODE pays, from its own wallet. This signer needs no AR at all.
+ *   - acceptance is not settlement. The bundler batches on an idle flush, then mines, then the
+ *     gateway indexes - minutes, not seconds. A run that ends PENDING is normal, not failure.
+ *   - snapshot durability now shares a failure domain with the node's own upload queue. That is
+ *     the trade taken for a single mechanism, and it is exactly why D25 must cover snapshots
+ *     too, not only assignments.
  *
  * Idempotency
  * -----------
@@ -27,7 +36,7 @@
  *
  * Safety
  * ------
- * Dry-run is the DEFAULT. Posting requires --confirm and spends real AR. Publishing an
+ * Dry-run is the DEFAULT. Posting requires --confirm and spends the NODE's AR. Publishing an
  * UNANCHORED snapshot is refused: a snapshot with no anchor assignment leaves the
  * published chain rootless, which is the exact defect D22 exists to close, so paying to
  * store one would buy a false sense of durability.
@@ -40,10 +49,16 @@
  *   PUBLISH_JWK=<json> bun run scripts/publish-snapshot.ts <dir> --confirm
  *
  * Env:
- *   PUBLISH_JWK   Arweave JWK (JSON, or a path to one). Signs AND PAYS. Required for --confirm.
- *   GATEWAY       gateway + peer for posting and verification (default https://arweave.net).
+ *   PUBLISH_JWK   Arweave JWK (JSON, or a path to one). SIGNS the data item and nothing else -
+ *                 it does not pay, so it needs no balance. Required for --confirm.
+ *   BUNDLER       base URL of the node whose ~bundler@1.0 takes the upload. Defaults to
+ *                 http://$SNAPSHOT_HOST, which is what the periodic job resolves from Consul.
+ *                 The route is p4-exempt and refused at the edge, so this must be an
+ *                 in-cluster address, never the public host.
+ *   GATEWAY       gateway for dedupe, size pricing and settlement checks.
  */
 import Arweave from 'arweave'
+import { createData, ArweaveSigner } from '@dha-team/arbundles'
 import { readFileSync, readdirSync } from 'node:fs'
 import { existsSync } from 'node:fs'
 import { join } from 'node:path'
@@ -57,6 +72,8 @@ const FORCE = has('--force')
 const ALLOW_UNANCHORED = has('--allow-unanchored')
 const WAIT_S = Number(flag('--wait') ?? 900)
 const GATEWAY = (process.env.GATEWAY || 'https://arweave.net').replace(/\/$/, '')
+const BUNDLER = (process.env.BUNDLER
+  || (process.env.SNAPSHOT_HOST ? `http://${process.env.SNAPSHOT_HOST}` : '')).replace(/\/$/, '')
 
 if (!DIR) {
   console.error('usage: bun run scripts/publish-snapshot.ts <snapshotDir> [--confirm] [--wait 900]')
@@ -75,7 +92,7 @@ const ar = (winston: string) => (Number(winston) / 1e12).toFixed(8)
 
 function loadJwk (): any {
   const raw = process.env.PUBLISH_JWK
-  if (!raw) throw new Error('PUBLISH_JWK is not set - required to sign and pay for an L1 transaction')
+  if (!raw) throw new Error('PUBLISH_JWK is not set - required to sign the data item')
   const text = existsSync(raw) ? readFileSync(raw, 'utf8') : raw
   const jwk = JSON.parse(text)
   if (!jwk.n || !jwk.d) throw new Error('PUBLISH_JWK does not look like an Arweave JWK')
@@ -171,38 +188,53 @@ async function main () {
     console.log('\nnothing to publish - every snapshot is already on chain at its slot')
     return
   }
-  console.log(`\n  TOTAL ${ar(totalWinston.toString())} AR for ${priced.length} transaction(s)`)
+  console.log(`\n  TOTAL ${ar(totalWinston.toString())} AR for ${priced.length} item(s) - ESTIMATE of what the NODE pays`)
 
   if (!CONFIRM) {
     console.log('\nDRY RUN - nothing posted. Re-run with --confirm (and PUBLISH_JWK set) to publish.')
     return
   }
 
+  if (!BUNDLER) {
+    throw new Error('BUNDLER (or SNAPSHOT_HOST) is not set - required to know where to upload')
+  }
   const jwk = loadJwk()
+  const signer = new ArweaveSigner(jwk)
   const addr = await arweave.wallets.jwkToAddress(jwk)
-  const balance = await arweave.wallets.getBalance(addr)
-  console.log(`\n  signer  ${addr}`)
-  console.log(`  balance ${ar(balance)} AR`)
-  if (BigInt(balance) < totalWinston) {
-    throw new Error(`insufficient balance: need ${ar(totalWinston.toString())} AR, have ${ar(balance)} AR`)
-  }
+  console.log(`\n  signer  ${addr} (signs the item; the node pays for the bundle)`)
+  console.log(`  bundler ${BUNDLER}`)
 
   const published: { contract: string, slot: string, id: string }[] = []
   for (const { item } of priced) {
-    const tx = await arweave.createTransaction({ data: item.data }, jwk)
-    for (const [k, v] of Object.entries(item.meta.tags)) tx.addTag(k, String(v))
-    await arweave.transactions.sign(tx, jwk)
-
-    const uploader = await arweave.transactions.getUploader(tx)
-    while (!uploader.isComplete) {
-      await uploader.uploadChunk()
-      process.stdout.write(`\r  ${item.meta.tags.contract} upload ${uploader.pctComplete}% (${uploader.uploadedChunks}/${uploader.totalChunks})   `)
+    // The snapshot's tags are already lowercase and unique, which is what lets dev_codec_ans104
+    // re-encode the stored item bit-exact for signature verification on later reads.
+    const di = createData(item.data, signer, {
+      tags: Object.entries(item.meta.tags).map(([name, value]) => ({ name, value: String(value) })),
+    })
+    await di.sign(signer)
+
+    // `Accept: application/json` is REQUIRED. Without it the node answers the POST with the
+    // Hyperbuddy HTML UI and HTTP 200, which reads as success and is not.
+    const res = await fetch(`${BUNDLER}/~bundler@1.0/tx`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/ans104',
+        'codec-device': 'ans104@1.0',
+        'Accept': 'application/json',
+      },
+      body: di.getRaw(),
+      signal: AbortSignal.timeout(300_000),
+    })
+    const body = (await res.text()).replace(/\s+/g, ' ')
+    if (!res.ok || !body.includes('"id"')) {
+      throw new Error(`bundler REFUSED ${item.meta.tags.contract} - HTTP ${res.status}: ${body.slice(0, 200)}`)
     }
-    console.log(`\n  ${item.meta.tags.contract} posted ${tx.id}`)
-    published.push({ contract: item.meta.tags.contract, slot: item.meta.tags.slot, id: tx.id })
+    console.log(`  ${item.meta.tags.contract} accepted ${di.id} (queued - NOT yet on chain)`)
+    published.push({ contract: item.meta.tags.contract, slot: item.meta.tags.slot, id: di.id })
   }
 
   console.log(`\nwaiting up to ${WAIT_S}s for GraphQL settlement (a 200 from the data endpoint is NOT settlement)`)
+  console.log('the bundler batches on an idle flush, then mines, then the gateway indexes - minutes')
   const deadline = Date.now() + WAIT_S * 1000
   const pending = new Set(published.map(p => p.id))
   while (pending.size && Date.now() < deadline) {
@@ -215,9 +247,14 @@ async function main () {
     console.log(`  ${p.contract.padEnd(18)} slot=${String(p.slot).padEnd(6)} ${p.id}  ${pending.has(p.id) ? 'PENDING' : 'settled'}`)
   }
   if (pending.size) {
-    console.log(`\n${pending.size} transaction(s) not yet indexed. Re-check with:`)
+    // Every item here was ACCEPTED by the bundler; the node owns landing it from this point.
+    // Exiting non-zero would mark a routine slow flush as a failed job and train the operator
+    // to ignore it. Settlement is D25's job, not this one's - it has to watch for the case
+    // where an accepted item never lands, which is the 2026-08-27 failure class.
+    console.log(`\n${pending.size} item(s) accepted but not yet indexed - NORMAL for a bundled upload.`)
+    console.log('Confirm later with:')
     for (const id of pending) console.log(`  bun run scripts/verify-snapshot.ts --published ${id}`)
-    process.exit(1)
+    return
   }
   console.log('\nVerify each with:  bun run scripts/verify-snapshot.ts --published <id>')
 }

operations/ao/publish-snapshot-live.hcl

@@ -11,29 +11,26 @@
 # both on p4's non-chargable routes, so this signer needs NO faff allow-list entry, and the
 # snapshot never touches the write path.
 #
-# --- Why direct L1 and not the node's ~bundler@1.0 ------------------------------------------
-# Three reasons, in order of weight:
+# --- Why the node's own ~bundler@1.0 and not direct L1 --------------------------------------
+# One upload mechanism, not two. D24 closed self-bundling on all three nodes, so every scheduled
+# message and assignment already reaches Arweave through ~bundler@1.0. Publishing snapshots the
+# same way leaves a single path to operate, fund and monitor.
 #
-#   1. Our own ~bundler@1.0 is BROKEN. It signs, prices and mines the bundle transaction, then
-#      every chunk POST returns 400 data_root_not_found: `building_proofs` computes a data_size
-#      that does not match the bundle it just posted a header for, so the merkle root describes
-#      a payload that does not exist. That is structural in the SignedTX -> structured@1.0 ->
-#      tx@1.0 round trip between post_tx and build_proofs, not payload-specific, so snapshots
-#      would fail exactly the same way - while still being PAID FOR.
-#   2. A bundled item is only queryable by tag if a gateway chooses to UNBUNDLE it. A direct L1
-#      transaction is indexed natively with its tags. Recovery finds snapshots by
-#      `tag process=<pid>, type=state-snapshot`, so bundling would make the durability
-#      mechanism depend on gateway policy for no benefit.
-#   3. The saving is 0.348%. Measured 2026-08-25: three separate transactions cost 0.0205867 AR,
-#      one bundle of the same bytes costs 0.0205150 AR. Arweave prices per byte; bundling exists
-#      to amortise MANY SMALL items, and three ~1 MiB blobs are the opposite shape.
+# The objection this had to clear: recovery finds snapshots by GraphQL TAG QUERY, and a bundled
+# data item is indexed only if a gateway chooses to UNBUNDLE it. Verified against live on
+# 2026-08-28 - items inside our own node-signed bundles ARE tag-discoverable on arweave.net, and
+# `transaction(id:)` reports a block height for them, which is what the settlement wait uses.
 #
-# If ~bundler@1.0 is ever fixed, revisit (2) before (3) - the unbundling dependency is the real
-# objection, not the fee.
+# What the handoff costs, and it is not nothing: snapshot durability now shares a failure domain
+# with the node's own upload queue, and acceptance by the bundler is NOT settlement on chain. A
+# run can legitimately end with items accepted but not yet indexed, and it exits 0 when it does.
+# That is exactly why D25's publishing-reliability monitoring has to cover snapshots too.
 #
 # --- Cost and safety ------------------------------------------------------------------------
-# WARNING: this SPENDS AR from PUBLISH_JWK - a real L1 transaction per contract. Keep it funded.
-# Measured 2026-08-25: 0.0206 AR for all three live contracts.
+# WARNING: this spends the NODE's AR, not PUBLISH_JWK's - the node signs and pays for the bundle
+# that carries the snapshot. PUBLISH_JWK signs the data item only and needs NO balance. The
+# thing to keep funded is the NODE wallet, which also pays for every assignment it publishes.
+# Estimated at 2026-08-25 prices: 0.0206 AR for all three live contracts.
 #
 # Publishing is IDEMPOTENT on (process, slot). A contract that has not advanced a slot since its
 # last published snapshot is SKIPPED, not re-posted - live operator-registry sits at slot 8 for
@@ -115,9 +112,14 @@ job "publish-snapshot-live" {
     # SNAPSHOT_HOST and PUBLISH_JWK are here, not in the env block: an env block does not run
     # through consul-template, so neither a service lookup nor a Vault read works there.
     #
-    # PUBLISH_JWK is a DEDICATED Arweave JWK that signs and pays. Not the node's wallet - the
-    # node's identity key stays in its own Vault path and is never used to publish - and not the
-    # EVM key publish-module.ts uses, which signs ANS-104 items for a bundler and holds no AR.
+    # PUBLISH_JWK is a DEDICATED Arweave JWK that SIGNS the snapshot data item and nothing else.
+    # It holds no AR and needs none: the node pays for the bundle. It is deliberately not the
+    # node's own key - the node's identity stays in its own Vault path and never signs a payload
+    # - so a snapshot stays attributable to the publisher rather than to the scheduler.
+    #
+    # BUNDLER is not set here: publish-snapshot.ts defaults it to http://$SNAPSHOT_HOST, which is
+    # the in-cluster address resolved just below. That matters - `~bundler@1.0` is refused at the
+    # edge on this env, so the upload only works from inside the cluster.
     template {
       destination = "secrets/keys.env"
       env         = true