Lift the etcd limit from 8GiB to 100GiB

[ronaldngounou]

As per performance improvements to etcd size limits have been evaluated to 100GB instead of 8GB.
https://www.cncf.io/blog/2019/05/09/performance-optimization-of-etcd-in-web-scale-data-scenario/

Contributes to issue #588

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ronaldngounou
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ronaldngounou]

Lint issues fixed:

content/en/docs/v3.4/faq.md:29:291 MD059/descriptive-link-text 
Link text should be descriptive [Context: "[here]"] 
(https://github.com/DavidAnson/markdownlint/blob/main/doc/md059.md)

[jberkus]

If you're doing this refactoring, I'd like to make it clear to users that the 100GB is a recommended maximum size, and not a hard limit. This would mean different text in a couple of places. I don't know what the actual hard limit is; probably need to look at the boltDB code.

[ronaldngounou]

Could you please suggest a wording that we should have in the meatime?

[wendy-ha18]

Suggested change

etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 100GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.

etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 8GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly. 100GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

[wendy-ha18]

Within the context of this doc: etcd has a relatively small memory ...... Typically 8GB is enough.... 100GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it is makes more sense for me.

[jberkus]

Do we actually have a warning at 100GB? I don't have a machine I can test that on.

[ronaldngounou]

Reverted this change

[jberkus]

Let's make this a limit, not a recommendation:

Suggested change

	etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 100GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
	etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 8GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly, up to a recommended maximum of 100GB.

[ronaldngounou]

Addressed

[jberkus]

For content/en/blog/2023/how_to_debug_large_db_size_issue.md let's take it out of this PR, and open a separate effort to convert the blog post into an Operations doc.

[ToSuperGod]

May I ask if there is any data in etcd that affects the cluster during data compression and fragmentation after storing 50GB of data? And how long does it take for large-scale insertion/query operations after completing the above operations

[ronaldngounou]

@ToSuperGod

May I ask if there is any data in etcd that affects the cluster during data compression and fragmentation after storing 50GB of data? And how long does it take for large-scale insertion/query operations after completing the above operations

When etcd stores large amounts of data (like 50GB, which is quite large for etcd), several things happen:
The main issue is fragmentation in the backend BoltDB database. As etcd performs updates and deletes, it creates "holes" in the database file. This fragmentation affects both disk usage and read performance since the database file becomes much larger than the actual data it contains.
Compression in etcd happens at the storage layer (BoltDB uses some compression), but the bigger concern is the growth of the database file itself. With heavy fragmentation, you might see a 50GB logical database consuming significantly more disk space, and reads having to traverse more pages.

Cluster Impact:
Performance degradation occurs in several ways: slower reads from the apiserver (affecting all kubectl commands and controller reconciliation loops), increased latency for watch operations (which controllers depend on), and potential leader election issues if followers fall too far behind during compaction.

The most critical impact is on write performance - if etcd is struggling, the entire Kubernetes control plane slows down because every resource change goes through etcd.

Compaction and Defragmentation:
After compaction (which removes old revisions) and defragmentation (which rebuilds the database file), you should see significant improvement. Defragmentation is particularly important - it rewrites the database to eliminate holes.

Timing for Large Operations:
For insertion/query timing after these operations, it really depends on your specific setup, but generally:

• After proper compaction and defrag, query latency should drop significantly (potentially 50-90% improvement if fragmentation was severe)

• Large-scale insertions should also improve, but etcd has hard limits - it's designed for storing configuration/state, not as a general-purpose database

• etcd recommends keeping databases under 8GB in practice

If you're consistently storing 50GB in etcd, that's a red flag - you might need to rethink what you're storing there. Consider if you're inadvertently storing large ConfigMaps/Secrets or have resource leaks.