Flat is 17-20% slower per block than HalfPath on chiado and 2.3x slower at p99, the only pairing where Flat loses

[stdevMac]

Description

On chiado, the Flat state backend is consistently slower per block than HalfPath — 17 to 20% at p50 and p90, and up to 2.3x slower at p99. This is the only network where Flat loses; on every other pairing measured, Flat wins by 1.4 to 2.4x at p50 and up to 3.2x at p99.

What makes chiado different is that its blocks are trivially small: median block processing there is 2 to 3 ms. That points at a fixed per-block overhead in Flat that only becomes visible when there is almost no state work to amortise it against — which would also affect small or empty blocks on any chain.

Steps to Reproduce

1. Run two chiado nodes with identical configuration except the state backend, one Flat and one HalfPath, both following the head.
2. Collect the Processed ... | N ms | figure from each node's own logs over an hour.
3. Compare the percentile distributions.

Actual behavior

Two independent 60-minute windows, n = 637 and n = 650 blocks per node. HalfPath/Flat ratio:

percentile	window 1	window 2
p50	0.85	0.83
p90	0.84	0.84
p99	0.82	0.43

In window 2, p99 is 25.4 ms for Flat against 10.8 ms for HalfPath.

Expected behavior

Flat should be at least as fast as HalfPath on small blocks, as it is everywhere else.

Additional context

Confidence is high that the effect is real — it replicates across both windows with a consistent sign at every percentile — and zero as to cause.

One lead, unconfirmed: on the same pairing, Flat's RocksDB block cache was the smallest relative to its HalfPath twin of any pair measured, 123 MB against 471 MB, a ratio of 0.26. A smaller block cache means more reads per block. Whether that is cause, consequence, or coincidence was not established.

Desktop

• Operating System: Linux x64
• Version: 2.0.0-rc (e6d3d2f5ab)
• Installation Method: Docker

[stdevMac]

The block-cache lead in the description does not hold — the two numbers are not measuring the same thing, and the direction is the opposite of what the lead implies.

nethermind_db_block_cache_size is published from DbOnTheRocks.GetCacheSize(), which is rocksdb.block-cache-usage — live usage, not configured capacity — and which returns 0 whenever the DB uses the shared block cache:

private long GetCacheSize()
{
    if (_isUsingSharedBlockCache)
    {
        // returning 0 as we are using shared cache.
        return 0;
    }
    return FetchTotalPropertyValue("rocksdb.block-cache-usage");
}

A DB gets a dedicated cache only when its options string sets block_based_table_factory.block_cache explicitly (DbOnTheRocks.cs:559-580); otherwise it joins the shared cache.

That splits the two sides of the comparison:

• HalfPath. DbConfig.StateDbRocksDbOptions contains no block_cache entry, so the State DB is on the shared cache and reports 0. The 471 MB attributed to the HalfPath node is therefore not its state cache at all — it is the sum over the DBs that do set one explicitly (Blocks 32 MB, Headers 32 MB, BlobTransactions 32 MB, BlockNumbers 16 MB, BlockInfos 16 MB, Code 16 MB, Receipts 8 MB) and/or the separate Shared series.
• Flat. FlatRocksDbConfigAdjuster gives the Account and Storage columns their own HyperClockCacheWrapper, sized from FlatDb.BlockCacheSizeBudget (default 1 GiB) at 0.3 and 0.7 — so ~300 MB and ~700 MB of dedicated capacity, and usage is reported.

So Flat's state cache was configured at ~1000 MB and only 123 MB of it was in use — about 12% occupancy. It had a large dedicated cache it was not filling, which is the opposite of "a smaller block cache means more reads per block". Meanwhile the HalfPath figure it was compared against does not include that node's state cache in the first place.

Two consequences:

1. The block-cache ratio should be dropped as a lead; nothing about it points at the 17-20% gap.
2. The comparison itself is a trap worth knowing about generally: this metric is not comparable across a Flat/HalfPath pair, because the state backend determines whether the state cache is dedicated (reported) or shared (reported as 0). Comparing the per-DB series between the two layouts will always look like Flat has "less cache".

The core observation in the description — a fixed per-block cost that only becomes visible when median block processing is 2-3 ms — is untouched by this and still looks right. The Flat-side candidates for a per-block cost that does not scale with block size are the persist-pipeline knobs rather than the cache: MaxInFlightCompactJob (32), CompactSize (32), MinReorgDepth/MaxReorgDepth (128/256) and MaxInMemoryBaseSnapshotCount (160) all describe work proportional to snapshot count rather than to the amount of state touched. Confirming that needs a fresh chiado Flat/HalfPath pair — the ones that produced the original measurement have been torn down.