function_clause in dev_process_worker:compute_cached/3 — hb_util:int/1 called with the {ao-result, body} envelope

[codex-curator]

First — thanks for the v0.9 work. Field report from a production HyperBEAM node serving a custom (non-aos) Lua AO process — our Aeternum Registrar at PID Dwnuy4MbuQkgwxw4-P08wxeny2KcwCh8Kd22mehacTc, behind Golden Codex's Aeternum Asset class (fine-art provenance on AO + Arweave). After v0.9-FINAL rolled through, every /compute request crashes deterministically. Diagnosis + a small defensive patch below; PR companion linked.

Symptom

100% of GET /<PID>/compute?slot=N requests crash the worker with function_clause. OTP restarts the worker, the next request crashes the same way, ad infinitum. Schedule writes still succeed; reads are dead.

Stacktrace (from journalctl -u hyperbeam)

returning_500_error, method: GET, path: Dwnuy..hacTc/compute,
Termination type: 'error'

Stacktrace:
  hb_util:int/[#{<<"ao-result">> => <<"body">>,<<"body">> => <<"14040">>}]
    at src/core/util/hb_util.erl:39
  dev_process_worker:compute_cached/3
    at src/preloaded/process/dev_process_worker.erl:51
  dev_process_worker:compute_group/3
    at src/preloaded/process/dev_process_worker.erl:30
  hb_persistent:await/4
    at src/preloaded/persistent/hb_persistent.erl:186
  hb_ao:resolve_stage/4
  hb_ao:do_resolve_many/2
  hb_ao:resolve_many/2
  dev_meta:handle_resolve/3
  hb_http_server:handle_request/3
  cowboy_handler:execute/2
  cowboy_stream_h:execute/3

Error details:
  function_clause

(Stacktrace captured against our local pre-reorganization tree where dev_process_worker.erl lived directly under src/; on current main the paths are src/preloaded/process/dev_process_worker.erl and src/core/util/hb_util.erl. Function signatures unchanged.)

Root cause

compute_cached/3 (around line 50 of src/preloaded/process/dev_process_worker.erl) expects RawSlot as a bare binary like <<"14040">>. The v0.9 HTTP resolve path is delivering it as the AO-Core envelope:

#{<<"ao-result">> => <<"body">>, <<"body">> => <<"14040">>}

hb_util:int/1 in src/core/util/hb_util.erl has three clauses — is_binary, is_list, is_integer. No map clause. BEAM raises function_clause. The data is fine — the slot id is at the body key; the consumer just isn't unwrapping it.

Proposed patch

Add a defensive unwrap as the first clause of compute_cached/3:

compute_cached(ProcID, RawSlot, Opts) when is_map(RawSlot) ->
    %% Defensively unwrap the AO-Core message envelope when the v0.9 HTTP
    %% resolve path delivers RawSlot wrapped instead of as a bare binary.
    %% The slot id lives at the `body' key; everything else is unchanged.
    case maps:find(<<"body">>, RawSlot) of
        {ok, Body} -> compute_cached(ProcID, Body, Opts);
        error -> false
    end;

Locally validated against production traffic: the function_clause crash stops entirely. Companion PR: codex-curator/HyperBEAM @ fix/compute-cached-envelope-unwrap.

The unwrap might belong upstream in dev_process:target_slot/2 (the caller that constructs RawSlot) — happy to follow whichever seam you prefer.

Environment

HyperBEAM	v0.9 banner (VERSION: v0.9.), at HEAD of permaweb/HyperBEAM/main at the moment of clone
Erlang/OTP	erts-15.2.7.8
Build profile	rocksdb+genesis_wasm
Store	configured hb_store_fs at /opt/gcx/node/data/cache; on-disk cache observed at _build/rocksdb+genesis_wasm/rel/hb/cache-mainnet/, per-PID checkpoints under genesis-wasm/checkpoints/state-<PID>.dat
Process	custom (non-aos) Lua, PID Dwnuy4MbuQkgwxw4-P08wxeny2KcwCh8Kd22mehacTc
OS	Ubuntu 22.04, GCE e2-standard-4, us-west1-a

Related v0.9 bugs we also hit (out of scope for the companion PR)

In recovery we also exposed:

1. LMDB mdb_page_dirty assertion against the existing cache after applying the unwrap patch. Next /compute exits with Assertion 'rc == 0' failed in mdb_page_dirty() at lmdb-sys-0.8.0/.../mdb:2121, status=6/ABRT. Possibly a v0.9 upgrade-migration gap on cached entries written under the old envelope.
2. loadMessages "Body is not valid" 500s after a full cache wipe — body "Body is not valid: would attempt to fetch from scheduler in loadMessages" from the genesis-wasm Node helper. Looks like the same envelope mismatch one layer up.

We're not proposing patches for these here — flagging only. The picture: nodes that ran pre-v0.9 long enough to accumulate cache hit (1); nodes that wipe to recover hit (2). The unwrap fixes neither alone.

Offer

Happy to follow up with full traces, reproduce against a freshly-provisioned node, pair on the right seam for the unwrap, or test patches against our Registrar before they ship. Reach us at curator@golden-codex.com · @codex-curator.

---

— Tad MacPherson, Metavolve Labs / Golden Codex
curator@golden-codex.com · @codex-curator · golden-codex.com

• Claude (Anthropic) — Jedi Code Master, paired on diagnosis + patch

Aeternum Registrar PID: Dwnuy4MbuQkgwxw4-P08wxeny2KcwCh8Kd22mehacTc
Operator: jFJbKYmhZVnmGk-ZTain8B3rV4S0aalTx6AEIKItNTM

[samcamwilliams]

Hey Tad / @codex-curator !

Is there any chance that your build/deployment:

1. Uses force-message in node options in a slightly non-standard way?
2. Experimented with delegating the lookup of cached compute results to a secondary node?

Those are the two ways I can think that an ao-result envelope would end up being passed around here. A defensive patch as you have in #943 covers this particular case, but if there is real smoke here there is likely a fire elsewhere. My hunch is to think that this is a deployment issue with some of our (apologies...) badly nodes Opts flags.

[chatgpt-codex-connector]

Codex couldn't complete this request. Try again later.

[codex-curator]

Hi Sam, thanks for the fast read and the pointers.

Honest answer: not deliberately. Our deployment uses the stock make compile → rebar3 as rocksdb,genesis_wasm release against HEAD of main. I'm not aware of any custom Opts flags being passed. Our config file is just port, priv_key_location, operator, host, hb_store, hb_store_prefix, and node_history_endpoint — nothing exotic. No force-message, and no explicit delegate-compute setup.

But your second hypothesis lights up perfectly. We are seeing exactly that pattern on a freshly-built shadow node we stood up overnight (built from current main + the #943 unwrap baked in).

The first /compute?slot=N request for our PID goes through _hb_device_delegated_compute → dev_genesis_wasm:41 → dev_process:compute_to_slot and returns a 500 with undef after ~40s. No cache dir is created. Our process's Scheduler is n_XZJhUnmldNFo4dhajoPZWhBXuJk-OcQr5JQ49c4Zo (Scheduler-Location → schedule.forward.computer), so we are presumably trying to delegate to your team's mainnet scheduler infrastructure, but something in the lookup is coming back undefined.

If there is a node Opts flag we should be setting (or unsetting) to make a fresh node behave as primary-compute rather than delegate-cached-lookup, I'd love a pointer. Happy to help dig further from this end — we have a clean reproduce on the shadow VM.

(And no apology needed — the docs gap is a real opening to help close. If a "running a fresh node serving a custom-Lua mainnet process" walkthrough would be useful, we'd be glad to write up our deployment story once we have the bootstrap working.)

— Tad

[codex-curator]

Hi Sam, your second hypothesis about the delegated cached-compute lookup was exactly the thread to pull. We dug in, and it looks like a two-part issue: a minor config gap on our end, and an upstream packaging regression on main.

1. The Config Gap (HB_URL)
Our systemd unit wasn't setting HB_URL. The genesis-wasm Node.js sidecar didn't know how to call back to the Erlang VM, so the locateProcess lookup was returning undefined. We added Environment=HB_URL=http://localhost:10000 to the service, and compute response times dropped instantly from ~41s to ~11s.

2. The Upstream Regression (The undef root cause)
Even with the lookup fixed, we were still hitting a 500 error. Curling with Accept: application/json bypassed the HTML wrap and revealed the true undef site: dev_scheduler_formats:assignments_to_aos2/4.

It looks like the ad285cd5 refactor (split core forge and preloaded devices) introduced a cross-package aliasing bug in the hb_forge_preload plugin.

Our store builds perfectly, and dev_scheduler_formats is inside the binary blob. But because the packager only rewrites within-package calls, cross-package helper calls break. dev_delegated_compute (in src/preloaded/vm/) tries to call dev_scheduler_formats (in src/preloaded/process/) by its bare module name rather than its aliased name, and the call goes nowhere.

Next Steps:
To unblock our deployment this weekend, we are going to pin our local build to ccd49590 (pre-refactor, where everything flat-compiles to BEAMs).

I will write up a clean repro and open a formal Issue for the hb_forge_preload cross-package bug so your team can track it for the next patch.

Thanks again for the diagnostic hint—it pointed us right at the module boundaries!

— Tad