tbc: tx index intermittently loses entries during IBD

[marcopeereboom]

Summary

The tx index (TransactionsDB) intermittently loses 't' entries during IBD. The ordinal indexer is the first consumer that reads every tx index entry during wind and surfaces this as a crash.

Evidence

Three different txids missing across three independent fresh-sync runs on testnet4:

1. f621f096... at block 65569 — "tx not in block" (tx index pointed to wrong block, later resolved itself)
2. da220b73... at block 81528 — "tx not found" (0 entries in raw LevelDB prefix scan, 1 missing out of 413 txs in the block, 412 are present)
3. 2a2b5ede... at block 114125 — "tx not found" (entry missing despite all verify checks passing)

Investigation performed

• Synthetic processTxs simulation: Replayed block 81528's 413 txs through processTxs logic. All 413 entries correctly added to the Go map cache, including the missing txid. No skip, no collision, no duplicate.
• Key collision check: No TxKey collision between da220b73... and any other entry. 't' prefix (65 bytes) and 's' prefix (69 bytes) cannot collide.
• Duplicate txid check: No duplicate txids within block 81528.
• Reorg check: Single block header at each affected height. No reorg evidence. Fresh sync.
• Post-commit verify: Added a verify step that reads back every 't' txid immediately after BlockTxUpdate commits. The verify PASSES — the entry is readable right after commit. But the entry is missing when the ordinal indexer reads it later (minutes to hours after).
• Slice vs copy test: Replaced key = k[0:65] slice aliasing in BlockTxUpdate with explicit make + copy. The original da220b73... entry survived with copies, but a DIFFERENT entry (2a2b5ede...) went missing on the same run. The copy fix is inconclusive.
• Wide-range audit: Scanned blocks 60000-82000 via direct LevelDB read. Found exactly 1 missing tx out of ~22000 blocks.
• Cross-indexer dependency: The ordinal indexer is the only indexer that reads BlockHashByTxId during wind. The UTXO indexer reads its own OutputsDB via ScriptHashByOutpoint during wind. The tx index data loss may have existed unnoticed before the ordinal indexer was added.

Key observation

The post-commit verify reads back from the same LevelDB handle immediately after BlockTxUpdate returns. This reads from the LevelDB memtable (in-memory), NOT from persisted SSTables. The verify proves the write reached the memtable. It does NOT prove the data survives memtable flush and compaction to disk.

Reproduction

Testnet4 IBD with TBC_ORDINAL_INDEX=true. Hits within the first 120K blocks. Different txid each run. Approximately 1 in 1M+ entries affected.

Batch sizes

The tx indexer cache holds up to 960K entries (96% of 1M capacity) before flushing. Each flush produces a single LevelDB batch with ~960K operations (mix of 't' and 's' entries) committed inside a LevelDB transaction.

Affected code

• database/tbcd/level/level.go — BlockTxUpdate: batch write path
• service/tbc/txindex.go — processTxs: cache population

Potential fix

Storing the tx byte offset (TxLoc) in the currently-nil 't' value may surface the issue differently or eliminate it — the current 't' value is nil, and a non-nil value changes the LevelDB write path characteristics. This is speculative and being built as a separate PR regardless (for performance).

[marcopeereboom]

Update: concurrency + goleveldb transaction layer

New finding

The "immediate read-back verify passes" result is invalid: BlockHashByTxId reads the same LevelDB handle right after commit, which serves from the in-memory memtable, not persisted SSTables. The verify proves the write reached the memtable. It proves nothing about survival through memtable to SSTable compaction.

The batch-corruption panic is the smoking gun

A separate run produced:

panic: runtime error: slice bounds out of range [:1099533940524] with capacity 142021011
  goleveldb/leveldb.batchIndex.v(...) batch.go:57
  goleveldb/leveldb.(*Batch).replayInternal(...) batch.go:171
  goleveldb/leveldb.(*Transaction).Write(...) db_transaction.go:177
  level.(*ldb).BlockUtxoUpdate(...) level.go:1903

The length field 1099533940524 (~0x10000000EC) is garbage. This is memory corruption inside goleveldb's batch buffer (b.data/b.index), surfacing during Transaction.Write -> replayInternal. Single-threaded batch construction cannot produce this; appendRec copies immediately. It requires concurrent mutation of shared goleveldb internal state.

Concurrency present during indexing

utxoindex.go fetchOPParallel fans out up to 128 goroutines doing concurrent ScriptHashByOutpoint (iterators) on the OutputsDB pool handle, while the same handle is used for OpenTransaction/Commit in BlockUtxoUpdate. The tx index uses the same OpenTransaction+tx.Write(batch)+Commit pattern on TransactionsDB.

goleveldb OpenTransaction freezes the frontend memdb and rotates it on commit. A concurrent iterator/read holding a reference into the rotating memdb is a known class of race in this library. Both symptoms (a missing 't' entry where the write lands in a frozen memdb that is dropped rather than compacted, and the batch-buffer corruption) are consistent with this single root in the transaction/memdb layer, not heminetwork code.

Why ordinals surfaced it

The ordinal indexer is the first consumer that reads every 't' entry during wind. The UTXO indexer's own occasional loss is masked by the same-block-spend fallback path, so it went unnoticed.

goleveldb maintenance status

github.com/syndtr/goleveldb last commit is 2022-07-21; the module is effectively unmaintained. The heminetwork pin is v1.0.1-0.20210819022825-2ae1ddf74ef7 (Aug 2021), predating even the final master commit.

Proposed fixes (root, not symptom)

1. Replace OpenTransaction+tx.Write(batch)+Commit with a direct atomic db.Write(batch, &opt.WriteOptions{Sync: true}) in BlockTxUpdate and BlockUtxoUpdate. This eliminates the transaction memdb freeze/rotate window entirely; db.Write is atomic and does not expose the frozen-memdb race.
2. Evaluate migrating off goleveldb given it is unmaintained (see discussion).

Either fix should be validated by a full testnet4 IBD with TBC_ORDINAL_INDEX=true and post-flush re-read verification (read after forcing compaction, not from memtable).

[marcopeereboom]

Root cause: BlockTxUpdate aliases the batch iterator key with key = k[0:65] (line 1918 on main). LevelDB batch iterators reuse the underlying byte buffer across calls, so key points into memory that gets overwritten on the next iteration. In large batches this causes silent data loss — the entry is written to the memtable with the correct key, passes post-commit verification, but the aliased slice is later mutated before or during flush.

Fix: PR #1052 replaces the alias with an explicit copy(keyBuf[:], k[0:65]) into a stack-allocated buffer. This will close the issue when #1052 merges.