etcd subnet watch never recovers from compaction (hot-loops, fills disk)

[phinze]

Expected Behavior

When the etcd subnet-lease watch falls behind etcd's compaction horizon (mvcc: required revision has been compacted), flanneld should recover on its own: re-list current leases at a fresh revision and resume watching from there, the same catch-up it does at startup. A transient compaction shouldn't require a restart.

Current Behavior

Once the watch's start revision has been compacted away, etcdSubnetRegistry.watchSubnets reconnects at the same revision indefinitely. The revision is never advanced, so every reconnect fails immediately with the same compaction error and the watch re-establishes in a tight loop, emitting a WARN+INFO pair per cycle (backoff caps at 5s). It never recovers without a restart. Captured loop (note the revision is always the original rev 1):

registry.go:293] registry: watching subnets starting from rev 1
registry.go:314] etcd watch channel for /coreos.com/network/subnets closed with error etcdserver: mvcc: required revision has been compacted, reconnecting...
registry.go:293] registry: watching subnets starting from rev 1
registry.go:314] etcd watch channel for /coreos.com/network/subnets closed with error etcdserver: mvcc: required revision has been compacted, reconnecting...

Root cause: watchSubnets (and watchSubnet) take a since revision and never update it across reconnects. Recovery logic does exist (leasesWatchReset, via the isIndexTooSmall case), but it lives inside the per-event parse switch, which a compaction never reaches: compaction arrives as a watch-response error (wresp.Err() / CompactRevision), handled by the earlier channel-closed branch that simply reconnects at the stale since.

Possible Solution

In watchSubnets/watchSubnet:

1. On a watch-response error, detect compaction (CompactRevision != 0 / rpctypes.ErrCompacted) and recover by re-listing (leasesWatchReset) at a current revision and resuming from there, instead of reconnecting at the stale revision.
2. Advance the resume revision as events arrive (since = resp.Header.Revision + 1) so an ordinary reconnect resumes forward rather than replaying from the original start rev (which is what lets it drift below the compaction horizon in the first place).

I have a patch plus a regression test and will open a PR referencing this issue.

Steps to Reproduce (for bugs)

1. Add the test below to pkg/subnet/etcd (it uses the existing integration harness).
2. It starts an in-process etcd, creates a subnet lease, advances and compacts the store past the watch's start revision, then watches from a now-compacted revision.
3. On master the watch never delivers the re-list snapshot; the test hangs to its 10s deadline while watchSubnets logs the loop shown above.
4. Real-world: run flanneld with the etcd (non-kube) subnet manager on a long-lived node; once etcd compacts past the watch's start revision and any reconnect occurs, the loop begins and never clears.

Reproduction test (also added by the PR)

func TestWatchSubnetsRecoversFromCompaction(t *testing.T) {
	integration.BeforeTestExternal(t)

	clus := integration.NewClusterV3(t, &integration.ClusterConfig{Size: 1})
	defer clus.Terminate(t)

	client := clus.RandClient()
	ctx := context.Background()

	r, kvApi := newTestEtcdRegistry(t, ctx, client)

	if _, err := kvApi.Put(ctx, "/coreos.com/network/config",
		`{ "Network": "10.1.0.0/16", "Backend": { "Type": "host-gw" } }`); err != nil {
		t.Fatal("Failed to put network config", err)
	}

	// An existing lease so the recovery re-list has something to snapshot.
	sn := ip.IP4Net{IP: ip.MustParseIP4("10.1.5.0"), PrefixLen: 24}
	attrs := &lease.LeaseAttrs{PublicIP: ip.MustParseIP4("1.2.3.4")}
	if _, err := r.createSubnet(ctx, sn, ip.IP6Net{}, attrs, 24*time.Hour); err != nil {
		t.Fatal("Failed to create subnet lease", err)
	}

	// Advance the store revision, then compact past it so that watching from an
	// old revision is guaranteed to hit "required revision has been compacted".
	var compactRev int64
	for i := 0; i < 5; i++ {
		resp, err := kvApi.Put(ctx, "/coreos.com/network/_bump", fmt.Sprintf("%d", i))
		if err != nil {
			t.Fatal("Failed to bump revision", err)
		}
		compactRev = resp.Header.Revision
	}
	if _, err := client.Compact(ctx, compactRev); err != nil {
		t.Fatal("Failed to compact etcd", err)
	}

	// Start the watch from rev 1, now below the compaction horizon.
	receiver := make(chan []lease.LeaseWatchResult, 16)
	go func() { _ = r.watchSubnets(ctx, receiver, 1) }()

	// Recovery must surface a re-list snapshot that includes the existing lease.
	if !waitForSnapshot(receiver, sn, 10*time.Second) {
		t.Fatal("watchSubnets did not recover from compaction with a re-list snapshot")
	}

	// A live event for a newly created lease proves the watch resumed at a current
	// revision instead of staying stuck on the compacted one.
	sn2 := ip.IP4Net{IP: ip.MustParseIP4("10.1.6.0"), PrefixLen: 24}
	if _, err := r.createSubnet(ctx, sn2, ip.IP6Net{}, attrs, 24*time.Hour); err != nil {
		t.Fatal("Failed to create second subnet lease", err)
	}
	if !waitForEvent(receiver, lease.EventAdded, sn2, 10*time.Second) {
		t.Fatal("watchSubnets did not resume delivering live events after recovery")
	}
}

func waitForSnapshot(receiver chan []lease.LeaseWatchResult, sn ip.IP4Net, timeout time.Duration) bool {
	deadline := time.After(timeout)
	for {
		select {
		case batch := <-receiver:
			for _, wr := range batch {
				for _, l := range wr.Snapshot {
					if l.Subnet.Equal(sn) {
						return true
					}
				}
			}
		case <-deadline:
			return false
		}
	}
}

func waitForEvent(receiver chan []lease.LeaseWatchResult, etype lease.EventType, sn ip.IP4Net, timeout time.Duration) bool {
	deadline := time.After(timeout)
	for {
		select {
		case batch := <-receiver:
			for _, wr := range batch {
				for _, ev := range wr.Events {
					if ev.Type == etype && ev.Lease.Subnet.Equal(sn) {
						return true
					}
				}
			}
		case <-deadline:
			return false
		}
	}
}

Context

We hit this on long-lived nodes using the etcd subnet manager (vxlan backend). A flanneld up for weeks fell into the loop after a watch reconnect and logged ~7.7 GB/day until the disk filled and the node went down. The kube subnet manager is unaffected (separate code path), which is likely why it hasn't surfaced widely; it specifically hits the legacy etcd datastore path on long-running nodes, where the start revision inevitably ages below the compaction horizon and then any reconnect wedges.

Your Environment

• Flannel version: observed on v0.26.7; code unchanged on master (a6ba76c5b0d3, 2026-06-30)
• Backend used (e.g. vxlan or udp): vxlan (bug is in the shared etcd subnet manager, so all backends on the etcd/local manager are affected)
• Etcd version: 3.5.x (reproduced against the integration harness, etcd 3.5.21)
• Kubernetes version (if used): N/A (etcd subnet manager, no Kubernetes)
• Operating System and version: Ubuntu 24.04
• Link to your project (optional): https://github.com/mirendev/runtime

[phinze]

Opened #2471 for the watchSubnets (plural) path, which is the one we hit in production.

The single-subnet watchSubnet has the identical hot-loop, but the right recovery is a genuine design question I'd rather get a maintainer's call on than guess: it should re-list just its own subnet, and when recovery finds the lease gone, should the watch resume or terminate? Today a normal delete event keeps the watch running, so "terminate on gone" would be a behavior change worth deciding deliberately. Happy to do that follow-up once there's a preference.

[phinze]

Thanks for merging #2471, @thomasferrandiz!

Any thoughts on the single-subnet watchSubnet question from my comment above? We don't actually use that path ourselves, but it's got the same hot-loop and I've got the bug paged in right now, so happy to send a PR to fix it up if you've got a read on the design call (resume vs. terminate when the re-list finds the lease gone).