RFC: Introduce ValkeyNode CRD as an internal infrastructure abstraction

[jdheyburn]

Summary

This proposal introduces a ValkeyNode CRD — an internal, operator-managed abstraction that represents a single Valkey pod. Users never create ValkeyNodes directly; they are created by parent controllers (e.g. ValkeyCluster, Valkey).

📄 Full proposal: VALKEY_NODE_CRD_PROPOSAL.md
🔧 Proof of concept: feat/valkeynode-crd

Why?

Today, ValkeyCluster directly manages Deployments/StatefulSets, PVCs, and pod lifecycle. As we add more CRDs (Valkey standalone/replicated, ValkeyPool, Sentinel), each would need to duplicate this infrastructure logic.

ValkeyNode provides a single place to manage:

• Singleton Deployment or StatefulSet (user's choice)
• PVC lifecycle (if persistence enabled)
• Observed replication state via INFO replication
• Status reporting (podIP, podName, ready, observedRole)

Design principles

• Deliberately "dumb" — ValkeyNode creates infrastructure and reports status. It does not make topology decisions, run failovers, or manage cluster slots.
• Generic — unaware of its parent type. Behaves identically whether owned by ValkeyCluster, Valkey, or a future CRD.
• Parent controllers own topology — they execute Valkey commands (REPLICAOF, CLUSTER MEET, etc.) directly against pods via status.podIP.

What it looks like

apiVersion: valkey.io/v1alpha1
kind: ValkeyNode
metadata:
  name: cache-0-0
spec:
  workloadType: StatefulSet
  podTemplate:
    spec:
      containers:
        - name: valkey
          image: valkey/valkey:8.0
  persistence:
    mode: enabled
    size: 10Gi
  config: {}

kubectl get valkeynode
NAME        READY   ROLE      POD-IP       REPLICA-OF     AGE
cache-0-0   True    primary   10.0.1.5                    5m
cache-0-1   True    replica   10.0.1.6     cache-0-0      5m
cache-1-0   True    primary   10.0.1.7                    5m
cache-1-1   True    replica   10.0.1.8     cache-1-0      5m

Trade-offs

Pros

• Eliminates duplicated infrastructure logic across CRDs
• Clean separation: infrastructure vs topology
• Gives operators visibility into individual pod state
• Enables future CRDs without re-implementing pod management

Cons

• Adds an extra layer of abstraction and indirection
• Additional CRD to maintain and version
• More resources in the cluster (one ValkeyNode per pod)
• Increases reconciliation chain depth

Questions for voters

Please vote in the poll below, and feel free to leave comments with:

• Concerns about the approach
• Alternative suggestions
• Fields you'd add/remove from the spec or status
• Naming preferences

[sandeepkunusoth]

is this planned for 0.2.0?

[jdheyburn]

I think we need a discussion as to when we are completely happy with releasing something that works. I am not married to whichever version this goes into.

[ysqyang]

I'm onboard with the overall design. Just one question: the "ROLE" column in the output of kubectl get valkeynode is actual rather than initial right? I think we've agreed upon not changing valkey node names after a failoever (so cache-0-0 could be a replica), but I suppose the ROLE value is always based on live topology and it's the controller's responsibility to keep it up-to-date?

[jdheyburn]

The status.observedRole exposes the output of the ROLE command. The valkeynode controller will ensure it is kept-up-to-date.

I'm sure the semantics will change in future, but I wanted to highlight how we can expose server internals in the K8s API and reuse them for troubleshooting.

[melancholictheory]

We explored a very similar "one workload primitive per Valkey process" idea and landed on a StatefulSet-per-shard shape rather than a per-pod CRD; sharing the trade-offs we measured in case they're useful input on the abstraction boundary.

• Why per-shard granularity helped most: a single cluster-wide StatefulSet can only carry one pod template, so it cannot express shard-scoped required anti-affinity (primary and replica of the same shard must not co-locate). Splitting workloads per shard gave us per-shard pod templates + a shard label, which made required anti-affinity and clean shard removal both expressible. A per-pod ValkeyNode is even finer-grained and would also solve this — just flagging that the placement win is the concrete payoff, so it's worth making sure the parent controller can stamp shard/role labels onto ValkeyNodes (this also feeds directly into [Design] Shard-aware placement policy for primaries and replicas #146).

• On the open ROLE question (initial vs live): agree it must be live. We keep workload/pod names stable across failover and treat role as observed topology only, read from INFO replication — never inferred from the ordinal/name. We have a separate finding (mirrors [Enhancement] Do not assume node role from node index during sequential rolls #123) where assuming role-from-index broke sequential rolls after a failover, so +1 to "ROLE is always controller-maintained from live topology."

• Clean scale-down was the other payoff: reshard slots away → del-node primary+replica → delete the shard's workload. No "which ordinal is actually the replica" guesswork, which is the trap with a single STS + ordinal removal (highest ordinal isn't necessarily the replica after a failover).

The main cost we paid for per-shard workloads is more objects to reconcile and a slightly more complex create/scale path — a per-pod ValkeyNode CRD is a cleaner way to centralize that pod/PVC/status logic, so the direction here makes sense to us.