Error in block freeze operation err="canonical hash missing, can't freeze block

[marschall-sh]

System information

Geth version: Version: 1.17.2-stable
CL client & version: Prysm version is v7.1.3.
OS & Version: Linux

Expected behaviour

Geth should get the missing hash and freeze the block

Actual behaviour

Geth throws error (see below) and the beacon chain validator dies after a while.

Steps to reproduce the behaviour

Around two weeks ago, might be related to the upgrade of geth to version 1.17.2-stable i started to see the ERROR you can see in belows backtrace. after discord consultation i ran a geth removedb (including ancientdir), after re-syncing i stumbled into the same problem again, the effect of this error is that after ~24h the prysm validator client dies with a "Context exceeded" error, only workaround currently is to restart geth every 24h.

Backtrace/Logs

Apr 15 18:18:09 blckchain geth[827839]: ERROR[04-15|18:18:09.338] Chain view: block hash unavailable       number=24,874,886 head=24,886,358
Apr 15 18:18:09 blckchain geth[827839]: ERROR[04-15|18:18:09.338] Log index head rendering failed, reverting to unindexed mode error="failed to advance log iterator at 24852735241 while rendering map 379222: receipts not found for block 24874886"
Apr 15 18:19:04 blckchain geth[827839]: ERROR[04-15|18:19:04.044] Error in block freeze operation          err="canonical hash missing, can't freeze block 24874886"

When submitting logs: please submit them as text and not screenshots.

[marschall-sh]

i run the geth data and ancient dir on two different disks. Not sure if this is related.
/dev/nvme0n1 1.8T 510G 1.3T 30% /media/ethdata
/dev/sdb1 1.8T 28K 1.7T 1% /media/ethfreezer

[rjl493456442]

Unfortunately the block 24874886 is missing in your local data. There is no easy way to recover except for resyncing.

Apologize for the inconvenience.

[marschall-sh]

I did re-sync already 2 times. Every time it's a different block missing. E.g. starting with this block arround 1 week ago: err="canonical hash missing, can't freeze block 24738874"

[ktfive]

I have the same issue.

WARN [05-10|22:04:55.660] Truncating freezer table database=d:\ethereum\geth\chaindata\ancient\chain table=bodies items=24,906,631 limit=24,906,584
WARN [05-10|22:04:55.662] Truncating freezer table database=d:\ethereum\geth\chaindata\ancient\chain table=receipts items=24,906,639 limit=24,906,584
ERROR[05-10|22:04:55.664] Error in block freeze operation err="canonical hash missing, can't freeze block 24906646"

[divanov13]

I have the same problem. After almost every restart of geth (successful), I started getting the errors (Error in block freeze operation. err="canonical hash missing, can't freeze block).

I deleted and rescynced geth databases (state, ancient, chain) but that kind if errors are popping up again.
I have another space on sdd & hdd, all disks healthy, the server with UPS.

May 12 08:54:01 geth[13134]: INFO [05-12|08:54:01.157] Imported new potential chain segment number=25,079,164 hash=2ba570..9d3bca blocks=1 txs=248 mgas=47.175 elapsed=378.622ms mgasps=124.596 triediffs=258.45MiB triedirty=0.00B
May 12 08:54:01 geth[13134]: INFO [05-12|08:54:01.262] Chain head was updated number=25,079,164 hash=2ba570..9d3bca root=c00611..4ef444 elapsed=4.291585ms
May 12 08:54:03 geth[13134]: WARN [05-12|08:54:03.338] Truncating freezer table database=/mnt/hdd2data/geth/chaindata/ancient/chain table=bodies items=25,077,494 limit=25,077,462
May 12 08:54:03 geth[13134]: WARN [05-12|08:54:03.367] Truncating freezer table database=/mnt/hdd2data/geth/chaindata/ancient/chain table=receipts items=25,077,514 limit=25,077,462
May 12 08:54:03 geth[13134]: ERROR[05-12|08:54:03.401] Error in block freeze operation err="canonical hash missing, can't freeze block 25077524"
May 12 08:54:06 geth[13134]: INFO [05-12|08:54:06.610] Generating snapshot root=0da6fc..e1c250 accounts=3,049,985 slots=15,876,683 storage=1.38GiB dangling=0 elapsed=3m28.026s

[ozpool]

A note on the "data is just missing, resync" reading: for at least some of these reports the missing block is well inside the immutable range, which makes it look less like a stray missing block and more like a kv/ancient consistency gap.

In the original log the freeze fails at block 24874886 while the head is 24886358, i.e. only ~11.5k blocks back. The freeze threshold is max(finalized, head - FullImmutabilityThreshold) (chainFreezer.freezeThreshold, core/rawdb/chain_freezer.go), and FullImmutabilityThreshold is 90000, so head - 90000 is ~24796358. The failing block is above that, which means the threshold here is being driven by the finalized number, and the block that has no canonical hash sits below finality. On a healthy canonical chain that mapping should always be present below the freeze threshold, so this is a genuine hole rather than a query past the indexed/ancient range.

The freezeRange loop bails on the first block whose ReadCanonicalHash returns the zero hash (core/rawdb/chain_freezer.go:314-317), so once such a hole exists the freezer can no longer make progress and re-logs the same error every recheck interval. That matches the reports where geth has to be restarted roughly daily and the CL eventually times out.

The other two logs point at how the hole forms. Both show Truncating freezer table ... table=bodies/receipts ... limit=N immediately before canonical hash missing, can't freeze block at a number just above N, i.e. a head truncation (reorg / setHead) followed by a freeze attempt over a range whose kv canonical hashes are gone.

setHead already orders this carefully: it deletes the kv canonical hashes and syncs the kv store before truncating the ancient store, specifically to avoid an inconsistent state on crash (core/headerchain.go:612-637). The comment there explicitly calls out the failure mode:

in the event of a panic, it's theoretically possible to lose recent key-value store writes while the ancient store deletions remain, leading to data inconsistency, e.g., the gap between the key value store and ancient can be created due to unclean shutdown.

So a gap of exactly this shape (freezer Ancients() ahead of the kv canonical hashes) is a known-possible state after an unclean shutdown around a rewind. The fact that several reporters hit it repeatedly, including after a full resync (divanov13, ktfive), suggests it is being re-created during normal operation rather than being a one-off corrupted disk.

A couple of things that would help pin it down:

• For an affected node, is the gap a single block or a contiguous run? ReadCanonicalHash for a window around the failing number, plus the freezer head/tail (Ancients() / Tail()), would show whether the freezer counter is simply ahead of the kv canonical hashes.
• Whether the affected nodes saw a crash / OOM / unclean shutdown shortly before the error first appeared (the original report mentions the validator dying, so a restart loop is plausible).

If the gap turns out to be the freezer being ahead of the kv hashes after an unclean shutdown, a startup-time reconciliation that truncates the freezer head down to the highest contiguous kv canonical hash would let the node self-heal instead of wedging the freeze loop. Happy to put that together with a regression test if that direction sounds right.