After etcd cluster upgraded from v3.4.34 -> v3.5.27, TotalDBSize has been increased and TotalDBSizeInUse remains stable.

[ishan16696]

We recently upgraded our etcd clusters from v3.4.34 -> v3.5.27 and after the upgrade it has been observed that there has been significantly increase in etcd's totalDBSize whereas totalDBSizeInUse remains stable.
The magnitude varies per cluster but the pattern is consistent across all of our etcd clusters.

Etcd v3.5.x has changed the default value of --backend-bbolt-freelist-type from array(persisted to disk) to map(in-memory). and because of this bolt-db is using hashmap to maintain the freelist, I suspect this to be the root cause of this issue.

Questions:

1. Is this expected behaviour after upgrading to v3.5.x ?

note: yes, We can run defragmentation to reclaim the free space, but running it too frequently is undesirable as it causes etcd members to become temporarily unavailable and can lead to trigger unnecessary leader elections.

2. On the claim that "map freelist type has much better performance than array.", Has this been validated via perf testing for clusters running with a quota of 8–10 GiB ? from this blog: https://www.cncf.io/blog/2019/05/09/performance-optimization-of-etcd-in-web-scale-data-scenario/ when this hashmap approach was introduced what I can understand that this hashmap based freelist was introduce for large-database scenarios. It is not clear that the performance benefit carries over to the small databases typical in most Kubernetes deployments.

3. Is there any other feature/changes can cause this issue ?

[ishan16696]

Another observation is that even though the total-databaseSize-inuse is going down but still totalDbSize is going up, may be because the freelist is allocating more pages instead of using fragmented available/free pages.

[dlanov]

Q1 — Is this expected after upgrading to v3.5.x?

Yes, this is expected and well-understood behavior. The v3.5 default change from --backend-bbolt-freelist-type=array to map changes how bbolt tracks free pages internally, and it has a direct effect on how the on-disk file size grows.

With array freelist (v3.4 default):

• The freelist is persisted to disk as a sorted array of free page IDs
• bbolt reuses free pages efficiently on the next write cycle because it can cheaply scan the sorted array
• totalDBSize stays close to totalDBSizeInUse

With map freelist (v3.5 default):

• The freelist is kept in memory only and rebuilt on startup
• bbolt no longer writes the freelist array to disk, which saves write amplification — but it means the allocator is more likely to append new pages rather than reuse existing free pages, especially across process restarts
• This causes totalDBSize to grow independently of totalDBSizeInUse, exactly the divergence you're seeing
  Your own follow-up observation is correct: the allocator is preferring to extend the file rather than reclaim fragmented free pages.

Q2 — Has the map freelist performance advantage been validated for small (8–10 GiB) databases?

The benchmarks that motivated the map default were designed for large, write-heavy databases (the web-scale scenario in the linked CNCF post). For Kubernetes-typical etcd clusters — which are small (usually under 2 GiB, rarely above 8 GiB) and have bursty rather than sustained write patterns — the performance benefit is marginal and the disk growth trade-off is more significant.

For clusters in the 8–10 GiB range you can safely switch back to array freelist:
--backend-bbolt-freelist-type=array

This restores v3.4 behavior: the freelist is persisted, free pages are reused aggressively, and totalDBSize tracks totalDBSizeInUse much more closely. The write amplification cost of persisting the freelist is negligible at your database sizes.

Q3 — Are there other changes that could contribute?

Two v3.5 changes are worth knowing about but are secondary to the freelist change:

• Snapshot catch-up — v3.5 changed how learner/follower snapshot transfers work; if any members recently caught up via snapshot, the file can be transiently larger until the next compaction+defrag cycle
• Increased default quota — if your quota was raised during the upgrade, bbolt pre-allocates more aggressively. Check that --quota-backend-bytes is unchanged

Defrag frequency concern

You noted that frequent defragmentation is undesirable due to leader election risk. With --backend-bbolt-freelist-type=array, you should need defrag far less often since the allocator reuses free pages naturally. A monthly or on-demand defrag triggered when totalDBSize / totalDBSizeInUse > 1.5 is a reasonable policy and avoids the availability concern.