feat: fetch ERC-20 balances via Multicall3 (per-chain address for Tron)

[cranycrane]

Closes #1683.

Routes ERC-20 balance lookups through Multicall3 aggregate3: one billable eth_call per chunk instead of N (a JSON-RPC batch is 1 HTTP request but N metered method calls), with every balance pinned to the same block. Falls back per-chunk to the existing JSON-RPC batch path on any failure (chain without Multicall3, transient error, decode/count mismatch), preserving the same-length/order and nil-on-failure contract callers rely on.

Adds a code-set per-chain override (EthereumRPC.Multicall3AddressOverride, empty = canonical) for chains that deploy Multicall3 at a non-canonical address, and wires Tron mainnet's (0x32a4f47a…) in NewTronRPC — code-driven rather than chain-id-detected because Tron's runtime net_version is not the real chain id.

Observability

• eth_call_requests{mode="multicall"} — new: aggregate3 call volume (the fast path), so the credit savings are measurable. aggregate3 calls were previously counted as mode="single".
• rpc_fallback_calls{component="erc20_multicall", reason="error"} — unexpected fallbacks to the JSON-RPC batch (the expected "Multicall3 not deployed" case is silent).
• The deployment probe logs once per process whether Multicall3 was found on the chain.
• No metrics.yaml change (mode is a dynamic label). Note: the mode="multicall" reclassification also moves the existing erc4626 enrichment's aggregate3 calls from single → multicall.

Commits

• feat(eth): add per-chain Multicall3 address override
• feat(eth): fetch ERC-20 balances via Multicall3 aggregate3
• feat(tron): fetch ERC-20 balances via Tron's Multicall3
• feat(eth): label Multicall3 aggregate3 calls as mode="multicall"

Testing: go build ./..., go vet, and unit tests for the resolver/override, aggregate3 decode + fallback, and NewTronRPC wiring all pass. Not yet smoke-tested against a live RPC.

🤖 Generated with Claude Code

[pragmaxim]

Reviewed 138b9c32. Verified locally: go build ./..., go vet, and go test ./bchain/coins/eth/... ./bchain/coins/tron/... (150 tests) all pass. Also confirmed the Tron constant is correct — base58 TEazPvZwDjDtFeJupyo7QunvnrnUjPH8ED decodes with a valid checksum to 0x41 + 32a4f47a74a6810bd0bf861cabab99656a75de9e.

The design is sound and the risky part is handled well: because api/worker.go:1298-1305 treats map presence as "already covered, do not single-call", a gas-starved Success=false would otherwise surface as a wrong balance rather than a missing one — the per-element re-resolve is exactly the right guard. Test coverage is targeted at the actual claims (call-count assertions, whole-chunk fallback, length mismatch, chunk bounding, genuine-revert-stays-nil).

Five things worth addressing, left as inline comments. I'd consider (1) and (3) blocking: (1) is a silent per-request cost regression on a hot path, and (3) undercuts the PR's own observability story. (2) is the one to watch in production — most relevant for Tron, where the constant-call energy cap for a 100-element aggregate3 is still unverified against a live node.

Nits not worth inline comments: min(erc20BatchSize, multicall3MaxCallsPerAggregate) couples two limits that bound different things (request count vs gas), so lowering erc20_batch_size for a provider's batch limit needlessly shrinks aggregate3 chunks too (latent — both default to 100); probeMulticall3's doc comment still says "deployed at the canonical address" though it is now override-aware; mode == "" as a "don't instrument" sentinel in ethCallAtBlock would read better as a bool; Multicall3AddressOverride is unvalidated, so a typo degrades to "not deployed" instead of failing loudly; and the balances path probes, then EthereumTypeMulticallAggregate3 probes again (harmless atomic load).

Test gap: no coverage for Success=false without a batcher — the one branch where a user-visible wrong balance is possible by design.

[pragmaxim]

1. Probe failure is not cached, and this path is now hot.

probeMulticall3 deliberately does not cache a transient eth_getCode error ("will retry on next call"). That was fine when only the erc4626 enrichment probed. Now every account-info request with more than one token pays an extra eth_getCode before falling back to the batch — forever — on any chain whose provider errors on or restricts eth_getCode. It is also silent, since the error is discarded here (deployed, _ :=).

Net effect on such a chain: +1 metered call per balances request, permanently, with nothing in the metrics to show it.

Suggestion: a consecutive-failure counter that latches multicall3NotDeployed after K errors, or at least backoff plus a ObserveChainDataFallback("erc20_multicall", "probe_error") so it is visible.

[cranycrane]

Fixed in 05436d95. The probe now latches to not-deployed after multicall3MaxProbeFailures (5) consecutive transient errors, so an eth_getCode-restricting provider stops paying a probe per request; and the transient case now emits ObserveChainDataFallback("erc20_multicall", "probe_error") so it is visible.

[pragmaxim]

2. Re-resolve cost on revert-heavy addresses may erase the savings.

Every Success=false element is re-resolved on every request, and the code cannot distinguish gas starvation from a deterministic revert. But ErrInvalidErc20Balance's own comment (contract.go:508-512) says reverting/dead/non-conforming tokens are "benign and common for dead/non-conforming/rebasing tokens" — and those are precisely the tokens that linger in holders' contract lists.

So for an address with, say, 30 such tokens, this path now costs 1 aggregate3 plus a 30-element batch — more metered calls than the old path, on a request that repeats for every account view, with no caching between requests.

Cheap partial mitigation: skip the re-resolve when len(results[i].Data) > 2. A revert carrying reason data or a Solidity Panic(uint256) cannot be an out-of-gas sub-call, so those are known-genuine reverts. Bare revert() / require(false) still returns empty returndata, so this is a filter rather than a proof — but it should remove most of the repeated work while keeping the gas-starvation guard intact.

[cranycrane]

Fixed in 05436d95 via your returndata heuristic: only Success=false elements with empty returndata are re-resolved (possible gas starvation). A failure carrying revert returndata (Error/Panic) is a genuine revert and stays nil, so revert-heavy addresses no longer pay a re-resolve batch every request. Covered by TestEthereumTypeGetErc20ContractBalancesGenuineRevertNotReresolved.

[pragmaxim]

4. elem_fallback counts requests, not elements.

This increments once per request regardless of len(failed), so the metric cannot distinguish "one dead token re-resolved" from "80 of 100 re-resolved". That element count is precisely the number that tells you whether the credit savings this PR is measuring are actually being realized, or being eaten by the re-resolve path (see the comment on the surrounding block).

ChainDataFallbacks.With(...).Add(float64(len(failed))) — or a separate histogram of the failed-subset size — would make it actionable.

[cranycrane]

Fixed in 05436d95. Added observeChainDataFallback(component, reason, count); the call site now records Add(len(failed)), so the metric counts elements re-resolved rather than requests. (After the comment-2 change failed is only the empty-returndata subset, so this now tracks the potential-gas-starvation volume specifically.)

[pragmaxim]

3. Metric continuity gap in the multicall path.

The batch path emits observeEthCallError("batch", "invalid") when parseSimpleNumericProperty returns nil (contract.go:752) and observeEthCallError("batch", "elem") on per-element failure (contract.go:730). The multicall path emits neither: a nil parse is silent, and Success=false is silent.

Consequence: eth_call_errors{type="invalid"} volume drops off the dashboard the moment a chain flips onto multicall, and there is no signal at all for per-element aggregate3 failures. For a PR whose explicit goal is that "the credit savings are measurable" and that per-token read volume "stays continuous", this is the one place where continuity breaks.

Suggestion: observeEthCallError("multicall", "elem") on !results[i].Success and observeEthCallError("multicall", "invalid") when the parse yields nil, mirroring the batch path.

[cranycrane]

Fixed in 05436d95. The multicall path now emits observeEthCallError("multicall", "elem") on !Success and observeEthCallError("multicall", "invalid") on a nil parse, mirroring the batch path so eth_call_errors continuity holds when a chain moves onto aggregate3.

[pragmaxim]

5. Probe is at latest, but aggregate3 runs at blockNumber.

For a blockNumber below the chain's Multicall3 deployment, eth_call to a then-codeless address returns "0x", which decodeAggregate3Result rejects as "response too short". That surfaces as an error, so it is safe — the caller logs a warning, bumps rpc_fallback_calls{reason="error"}, and falls back to the batch — but it happens on every such request, and it pollutes the "unexpected fallback" metric that is supposed to be quiet.

Impact today is essentially nil: bchain/erc20_batch_integration.go:103 is the only caller passing a non-nil block, and it uses the best block height. But it is a trap for the next caller of ...AtBlock. Worth either a comment here, or skipping the multicall path when blockNumber is non-nil and older than the probe.

[cranycrane]

Documented in 05436d95 rather than adding deployment-block tracking: the only non-nil-block caller uses the best block height, and a historical block below deployment already falls back safely (aggregate3 returns 0x → decode error → batch). Added a comment at the probe explaining the trap for future ...AtBlock callers, and fixed the stale "canonical address" wording in the probe doc.

[pragmaxim]

Re-reviewed 05436d95. Verified: go build ./..., go vet ./bchain/... ./api/..., and go test ./bchain/coins/eth/... ./bchain/coins/tron/... ./api/... — 191 tests pass.

All five are resolved, and correctly:

• (2) revert filter — len(results[i].Data) <= 2 is exactly the right predicate. decodeAggregate3Result always emits "0x"+hex, so length 2 is precisely empty returndata; out-of-gas and invalid-opcode produce empty returndata, while an explicit REVERT carrying Error/Panic data cannot. The Panic(0x11) test asserting a size-1 re-resolve batch pins the distinction well.
• (3) metrics — observeEthCallError("multicall", "elem"/"invalid") now mirrors the batch path exactly, including counting benign reverts under "elem" the way erc20BalancesBatchAtBlock does.
• (4) element count — clean; keeping ObserveChainDataFallback as a count=1 wrapper avoids touching other callers.
• (5) historical blocks — the comment is accurate and sufficient given only the integration helper passes a non-nil block today.
• Plus the probeMulticall3 doc fix and the no-batcher Success=false test I mentioned as gaps.

One new issue introduced by the fix for (1), inline below. It is not a correctness problem — the code always degrades safely to the batch path — but the failure mode is a silent, permanent loss of the savings this PR exists for, which I think is a worse trade than the extra probe it removes.

Nit: rpc_fallback_calls_total{component="erc20_multicall", reason="elem_fallback"} now counts elements while every other series in that family counts requests, so sum by (component) mixes units. A reason name that signals it, or a separate metric, would avoid a confusing dashboard later.

Otherwise this looks good to merge.

[pragmaxim]

The latch is permanent, and its blast radius is wider than balances.

multicall3ProbeFailures never resets and the K-th failure stores multicall3NotDeployed for the process lifetime. Three things make that a harsher trade than the problem it fixes:

1. Five consecutive probe rounds is a low bar. A backend restart, or a brief provider 5xx window while the API keeps serving address requests, reaches it easily. Blockbook processes then run for weeks with multicall off — the entire saving this PR exists for, gone until someone restarts the process.
2. It disables more than balances. probeMulticall3 is shared, so the latch also permanently kills ERC-4626 enrichment: api/erc4626.go:184-187 silently continues on the resulting errMulticall3NotDeployed.
3. The decisive event is invisible in metrics. See the separate comment on contract.go:573.

Suggestion: re-arm instead of latching. An atomic.Int64 holding a nextProbeAt deadline keeps the "no probe on every request" property while recovering after the outage — on the K-th failure set the deadline ~10 minutes out and return (false, nil) without storing multicall3NotDeployed, then allow one re-probe once it passes. Alternatively, reset multicall3ProbeFailures whenever any other RPC call succeeds.

TestProbeMulticall3_LatchesNotDeployedAfterRepeatedFailures would need adjusting; nothing else in the suite depends on this.

[pragmaxim]

Follow-on to the latch comment on multicall.go:126: probe_error fires only while probeErr != nil, but the call that actually trips the latch returns (false, nil), so the transition to permanently-disabled emits no metric at all — just the one glog.Warningf. That is the one event you would want to alert on, and it is the only one not counted, which sits a little awkwardly next to the mode="multicall" error metrics added in this same commit.

A distinct reason="probe_disabled" (or "probe_latched") on the latching branch would close it.