lwwallet recovery holds the wallet writer across Esplora network I/O

[bhandras]

Description

The lightweight wallet can hold btcwallet's single bbolt write transaction
while performing remote Esplora HTTP requests during startup recovery. Any
concurrent wallet operation that needs key derivation then blocks behind that
writer. On a physical iPhone this makes invoice creation, balance refresh, and
activity refresh hang until their outer RPC deadlines expire; restarting the
embedded daemon only changes the timing.

Reproduction

This was reproduced with current Wavelength main embedded through
wavelength-mobile on an iPhone 12 Pro, using the signet lwwallet/Esplora
configuration:

1. Start and unlock an existing lightweight wallet.
2. While btcwallet startup recovery is scanning, create a Lightning invoice.
3. The receive path reaches receive_auth_key and remains there until its
   20-second mobile deadline expires.

Representative trace:

[MOBILE] Receive started amount_sat=2000 timeout=20s
[INF] WRPC: Wallet receive stage completed stage=wallet_ready elapsed=47.583µs
[INF] WRPC: Wallet receive stage started stage=receive_auth_key
[MOBILE] Receive failed amount_sat=2000 elapsed=20.002s err="create receive invoice: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"

The same build succeeds in the simulator because the relevant Esplora requests
usually finish quickly enough there to hide the lock inversion.

Goroutine evidence

An untruncated goroutine dump taken at the receive deadline shows the blocked
invoice goroutine waiting to begin a bbolt write transaction:

sync.(*Mutex).Lock
go.etcd.io/bbolt.(*DB).beginRWTx
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).Update
github.com/lightningnetwork/lnd/lnwallet/btcwallet/keychain.(*BtcWalletKeyRing).DerivePrivKey
github.com/lightningnetwork/lnd/lnwallet/btcwallet/keychain.(*BtcWalletKeyRing).ECDH
github.com/lightninglabs/wavelength/waved.(*RPCServer).deriveReceiveAuthKey
github.com/lightninglabs/wavelength/swapwallet.(*Service).receiveAuthPrivateKey
github.com/lightninglabs/wavelength/swapwallet.(*Service).ReceiveAuthKey

The writer is owned by btcwallet recovery, which is waiting on an Esplora
HTTP/2 request inside the transaction callback:

net/http.(*http2ClientConn).roundTrip
github.com/lightninglabs/wavelength/lwwallet.(*EsploraClient).GetScriptChainTxs
github.com/lightninglabs/wavelength/lwwallet.(*EsploraChainService).scriptHistoryToHeight
github.com/lightninglabs/wavelength/lwwallet.(*EsploraChainService).FilterBlocks
github.com/btcsuite/btcwallet/wallet.(*Wallet).recoverScopedAddresses
github.com/btcsuite/btcwallet/wallet.(*Wallet).recovery.func2
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).Update
github.com/btcsuite/btcwallet/wallet.(*Wallet).recovery

btcwallet v0.18.0 calls chainClient.FilterBlocks from
recoverScopedAddresses while its outer walletdb.Update callback is active.
Wavelength's FilterBlocks implementation is necessarily network-backed: it
queries script history and may fetch transactions or whole blocks. A slow or
stalled mobile HTTP/2 connection therefore holds the wallet-wide writer for
the duration of network I/O.

The RST_STREAM CANCEL is a consequence, not the cause: it is the local
embedded bufconn RPC being cancelled after the mobile deadline.

Expected behavior

No remote chain request should execute while a wallet database write
transaction is held. Startup recovery must not make unrelated wallet key
operations wait indefinitely, and RPC cancellation should not leave callers
blocked behind a non-context-aware database writer.

Fix direction

The durable fix should split btcwallet recovery into phases:

1. derive/expand the recovery horizon and construct the filter request in a
   short database transaction;
2. execute FilterBlocks outside every database transaction;
3. apply the response and advance recovery state in another short transaction.

A Wavelength-only mitigation must preserve seed recovery. Globally setting the
recovery window to zero or treating an interrupted scan as empty would be a
funds-discovery regression and is not an acceptable fix.

Related

• GetInfo can block behind wallet database writes #1132 / waved: reuse cached identity in GetInfo #1133 remove an unnecessary GetInfo identity-key derivation from
  the competing-writer side. Physical-device timing confirms that change makes
  the wallet_ready stage complete in tens of microseconds, but the receive
  auth key (and any other write-backed derivation) still blocks behind the
  recovery writer described here.

[bhandras]

The transaction-boundary fix is now proposed upstream in btcsuite/btcwallet#1318, with the root cause tracked in btcsuite/btcwallet#1317. The PR preserves the recovery batch's atomic persistent commit while running all FilterBlocks calls outside wallet transactions. I am integrating the exact branch into the iOS diagnostic build for physical-device validation.

[bhandras]

The upstream transaction-boundary fix in btcsuite/btcwallet#1318 has now been
validated on the affected physical iPhone through a Wavelength gomobile build.

Two consecutive 2,000-sat invoices completed successfully in 984 ms and 882 ms.
The previously blocked receive-auth ECDH stage completed in 69 ms and 2.6 ms,
respectively, instead of exhausting the 20-second deadline. No RST_STREAM,
deadline error, or goroutine dump occurred.

This confirms the root cause reported here: moving FilterBlocks outside the
wallet database transaction removes the real-device writer starvation without
disabling or weakening recovery.