Feature - Pod inplace upgrades using resize feature

[Sasidharan3094]

Summary

Adds in-place resize support for Redis master pods, so a CPU/memory-only change to a RedisFailover's spec.redis.resources no longer forces a master restart on nodes that are already packed tight. Slaves are untouched — they continue through the existing delete-based OnDelete rollout exactly as before.

Uses Kubernetes' in-place pod resize (resize subresource, stable since v1.35, beta and enabled by default since v1.33 — confirmed working against an EKS 1.34 test cluster).

Scope: master-only. Slave resource changes still restart normally; this PR does not change that.

Why

Redis endpoints run 1 master + N slaves, memory-bound, on nodes packed close to capacity. Any resource bump today restarts the master via delete-and-recreate. This PR lets a resource-only change resize the master's live container in place when possible, and safely falls back to the existing restart-based path whenever it can't.

How it works

1. Detecting a resource-only change (service/k8s/statefulset.go)
CreateOrUpdateStatefulSet now dry-runs the update first (UpdateOptions{DryRun: DryRunAll}) and compares the stored StatefulSet against the dry-run result — both API-server-defaulted, so the diff isn't polluted by implicit fields the server fills in on write (an early version of this compared the stored object against a raw, never-submitted one, which produced false positives on every change; fixed during testing). The comparison itself neutralizes only Resources on both container specs before running apiequality.Semantic.DeepEqual — provably exhaustive (any other field difference, current or future, correctly fails the check) rather than an enumerated field list. Result is stashed as the redis-failover.freshworks.com/resize-only annotation on the StatefulSet.

2. Master-branch decision (operator/redisfailover/checker.go)
In UpdateRedisesPods, the master branch now reads that annotation:

• Not resource-only → unchanged delete-based path.
• Resource-only → attemptMasterResize, which:
  • Tracks attempt state via a resize-started-at pod annotation (no Status subresource or requeue mechanism exists in this operator, so this is evaluated fresh against time.Now() each ~30s reconcile).
  • Calls the resize subresource (ResizePod, new method on the Pod interface, using client-go's UpdateResize).
  • Reads the pod's PodResizePending condition to see the outcome: absent (with resources actually verified to match, see below) → success; Infeasible → fail fast to delete; Deferred → try to free room (step 3), re-issuing the resize call on every retry so a mid-flight CR edit gets picked up rather than staying pinned to the original ask.
  • 10-minute timeout (hardcoded constant, not yet CR-configurable) bounds the whole attempt before falling back to delete.
  • On confirmed success, patches the pod's controller-revision-hash label to match the StatefulSet's status.updateRevision — required because a resize never updates that label on its own, and without it the next reconcile would treat the pod as stale again.
  • Verifies success via actual resource comparison (PodResourcesMatchDesired), not just the absence of a pending condition — found during live testing that an RBAC-rejected resize call also shows no pending condition, which had been silently treated as success.

3. Freeing node headroom (operator/redisfailover/service/resize_headroom.go, service/k8s/eviction.go)
When a resize is Deferred, evicts co-located slave pods belonging to other RedisFailovers on the same node (never the same endpoint's own slaves) to free room:

• ComputeRequiredHeadroom: Node.Status.Allocatable minus everything else on the node minus the master's desired ask, computed independently for CPU and memory.
• selectEvictionSet: exhaustive subset search over candidate slave pods (redisfailovers-role=slave label, already maintained by the operator) for the minimal-pod-count, minimal-waste subset covering whichever of CPU/memory is actually short — a zero/negative requirement drops out of the constraint automatically, so "CPU only," "memory only," and "both" all fall out of the same code path. Greedy fallback above 24 candidates bounds runtime.
• Evicts via the real Eviction API (policy/v1 Eviction, not a raw delete), so PDBs are honored — a rejected eviction triggers recomputing the best remaining subset against the still-outstanding deficit, not a full restart.

RBAC

Cross-endpoint eviction means this operator's ServiceAccount now needs to list/evict pods across all namespaces, not just its own managed ones. Added to both charts/redisoperator/templates/service-account.yaml and manifests/kustomize/components/rbac/clusterrole.yaml:

• pods/resize (patch/update)
• pods/eviction (create)
• nodes (get/list, for Status.Allocatable)

Deploy the updated RBAC before rolling out this operator image — without it, resize calls fail with a 403 that (pre-fix) was silently misread as success. Confirmed and fixed during live testing.

Testing

Tested live against an EKS 1.34 dev cluster with two co-located RedisFailover tenants on a single node:

• Happy path (node has room): resize applies with zero restart, zero eviction needed.
• Eviction path (node doesn't have room): correctly evicts the other tenant's slave (not the resizing endpoint's own), confirmed via operator logs.
• Genuinely infeasible (ask exceeds even full eviction of every candidate): retries every ~30s as designed, times out at 10 minutes, falls back to delete — no stuck state.
• Mid-flight CR edit while stuck: confirmed the pre-fix gap (resize target stayed pinned to the original ask, ignoring a subsequent CR edit until the full timeout elapsed) and fixed it by re-issuing the resize on every Deferred retry.
• Unit tests added for the subset-sum selection (CPU-only, memory-only, and combined cases where the optimal subset differs per-dimension) and the resource-only diff detection (service/k8s/resize_diff_test.go, operator/redisfailover/service/resize_headroom_test.go).

Files changed

Area	Files
Resource-only diff	service/k8s/statefulset.go, service/k8s/resize_diff_test.go
Resize execution	service/k8s/pod.go
Eviction primitive	service/k8s/eviction.go, service/k8s/k8s.go
Headroom + selection	operator/redisfailover/service/resize_headroom.go, resize_headroom_test.go
Checker/healer wiring	operator/redisfailover/checker.go, checker_test.go, operator/redisfailover/service/check.go, heal.go, constants.go
Container spec	operator/redisfailover/service/generator.go (explicit ResizePolicy)
RBAC	charts/redisoperator/templates/service-account.yaml, manifests/kustomize/components/rbac/clusterrole.yaml
Mocks	mocks/operator/redisfailover/service/RedisFailoverCheck.go, RedisFailoverHeal.go, mocks/service/k8s/Services.go