Seed a master only where every Redis holds no data

[indiebrain]

The gap

CheckAndHeal promotes a node in two places when it finds no master:

• checker.go, the branch taken when CheckSentinelQuorum returns an error. It selected unconditionally.
• checker.go, the branch where CheckIfMasterLocalhost is true. It selected behind a guard that cannot do what it is being asked to do.

A failover whose master is gone and whose replicas are still up holding data reaches both. nMasters is 0, quorum is gone, and SetOldestAsMaster promotes by CreationTimestamp, which says nothing about how far a node's data is behind.

This is already decided

ADR-001 says the operator seeds an initial master and does not otherwise choose, and that it "does not choose among nodes that may hold data". The rule held in the decision record and not in the operator.

That ADR also named the signal it needed and did not have:

Rule 2 says the operator may seed an empty cluster but must not choose among nodes holding data, which requires telling those two apart. CheckIfMasterLocalhost cannot: a restarted pod reloads slaveof 127.0.0.1 while keeping its dataset. Something like "no node holds any keys" would serve, and is not implemented.

This adds it.

The change

• HoldsNoData on the Redis client reads the keyspace section of INFO, which lists a line per database holding keys and comes back empty when there are none. DBSIZE would only answer for the database the client selected.
• CheckIfAllRedisHoldNoData requires that of every running node.
• checkSeedingAllowed sits in front of both selecting paths. A node that holds keys, or that cannot be reached to be asked, stops the operator, which records MasterUnknown naming the cause and changes nothing.

A Redis reading its dataset from disk is an error, not an empty one. This is the race the check turns on. INFO is served while loading:1, and the keyspace section fills in as keys are inserted, so a node holding a full dataset on disk reports an empty keyspace until enough of it is in memory. It is reachable: GetRedisesIPs selects on the pod phase being Running, not on the container being ready, so a pod whose Redis is still loading is asked and answers. A whole cluster returning after an outage is precisely when every node is loading at once, and precisely when seeding would discard what is being loaded. async_loading counts the same, for a replica loading diskless.

A node that cannot be reached counts as holding data. The failure modes are not symmetric: treating an unreachable node as empty lets the operator promote over data it could not see, while treating it as holding data costs an availability stop a person can resolve.

The single-replica path is unchanged. One candidate is not a choice among nodes, and gating it would stop a standalone failover from ever getting a master.

What this costs

A no-quorum event on a cluster holding data now stops and stays stopped until someone looks, where it previously recovered by promoting a node that may have been behind. That is the trade ADR-001 accepts explicitly ("Recovery can stop and stay stopped… the deliberate price of not guessing"), extended to the path that was still guessing. Worth knowing for whoever operates these clusters: the failure is visible as the MasterUnknown condition in kubectl get redisfailover rather than silent.

On issue #100

This is the part of #100 that is still open and buildable. It does not implement that issue's request to select by replication offset: ADR-001 investigated and rejected that on measurement against redis:8.10.1, so the operator does not rank nodes at all. #100 stays open regardless, since the integration coverage it asks for is not here.

Also in this change

• ADR-001 is corrected. Its Consequences section claimed SetOldestAsMaster "is now reached only when seeding", which was not true of the code and is true after this. The paragraph about the missing cold-start signal is updated to describe the one that now exists.
• reportMasterUnknown's message template is generalized. It hardcoded "a Redis node could not be inspected, so one of them may still be the master", which was the only cause when it was written and is not now. Each caller supplies its own cause, so the condition names what actually happened rather than reading as a contradiction.
• CIR-004 records the behavior, the constraints, and the rejected alternatives.

Verification

go build, go vet, gofmt, and the full go test ./... pass.

Two test cases are added, covering the no-quorum and first-boot paths against a cluster that holds data, asserting that nothing is promoted and the condition is recorded.

The guard is mutation-tested rather than assumed. Replacing CheckIfAllRedisHoldNoData with an unconditional true fails the suite:

--- FAIL: TestCheckAndHeal/No_masters,No_sentinel_quorum_set_random
    FAIL: CheckIfAllRedisHoldNoData(*v1.RedisFailover)
    FAIL: 17 out of 18 expectation(s) were met.

Two existing test cases were changed. Both drive the seeding paths and now set the new expectation. They describe empty clusters, which is what seeding means after this change, so they assert the same outcome for a narrower case rather than a different outcome. Changing tests alongside the code they cover deserves scrutiny, so it is called out here rather than left to be noticed in review.

Not covered

No integration coverage. The harness cannot induce the states this turns on: a node that stays Running while holding data with no master anywhere, and Sentinel losing quorum on demand. This is the same limitation recorded on #106.

[powerhome-portal]

A change to documentation files was detected in your PR. Please visit this link to preview changes: https://portal-staging.powerapp.cloud/docs?filters[kind]=all&filters[user]=all&filters[namespaceFilter]=2026-09-04--seed-only-an-empty-cluster