Skip to content

Commit 43b9ae6

Browse files
committed
Add timestamp (age-based) algorithm
1 parent 2deca29 commit 43b9ae6

10 files changed

Lines changed: 508 additions & 14 deletions

File tree

CONTRIBUTING.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,17 +30,17 @@ The controller uses a plugin-based architecture for extensibility:
3030
3131
┌─────────────┼─────────────┐
3232
▼ ▼ ▼
33-
┌─────────┐ ┌─────────┐ ┌─────────┐
34-
│ Zone │ │ Future │ │ Future │
35-
│ Handler │ │ Algo │ │ Algo │
36-
└─────────┘ └─────────┘ └─────────┘
33+
┌─────────┐ ┌───────────┐ ┌─────────┐
34+
│ Zone │ │ Timestamp │ │ Future │
35+
│ Handler │ │ Handler │ │ Algo │
36+
└─────────┘ └───────────┘ └─────────┘
3737
```
3838

3939
### Key Components
4040

4141
- **PodReconciler**: Main controller that watches Kubernetes resources and triggers reconciliation
4242
- **ModuleManager**: Routes pods to the appropriate algorithm handler based on Deployment annotations
43-
- **Handler**: Algorithm implementations (currently `zone`)
43+
- **Handler**: Algorithm implementations (currently `zone` and `timestamp`)
4444
- **Expectations Cache**: Thread-safe cache for handling async reconciliation
4545

4646
### Project Structure
@@ -59,6 +59,10 @@ The controller uses a plugin-based architecture for extensibility:
5959
│ │ ├── handler.go # Zone distribution handler
6060
│ │ ├── controller_utils.go # DeletionCostPool
6161
│ │ └── module.go # Module registration
62+
│ ├── timestamp/ # Timestamp (age-based) algorithm
63+
│ │ ├── handler.go # Age-based deletion cost handler
64+
│ │ ├── handler_test.go # Unit tests
65+
│ │ └── module.go # Module registration
6266
│ ├── module/ # Module interface definitions
6367
│ │ └── handler.go # Handler interface
6468
│ └── expectations/ # Caching layer
@@ -357,7 +361,7 @@ controller.GetType(dep *v1.Deployment) string
357361

358362
### Using the Expectations Cache
359363

360-
The `expectations.Cache` helps handle async reconciliation by tracking pending assignments:
364+
The `expectations.Cache` helps handle async reconciliation by tracking pending assignments. It is **only needed when a pod's cost depends on the state of sibling pods** (as the `zone` algorithm does). If your algorithm derives the cost purely from the pod itself (as the `timestamp` algorithm does from `.metadata.creationTimestamp`), the calculation is deterministic and idempotent, so you can skip the cache entirely and simply guard with `controller.HasPodDeletionCost` / `controller.IsDeleting`.
361365

362366
```go
363367
cache := expectations.NewCache[types.UID, int]()
@@ -458,7 +462,7 @@ Aim for at least 80% code coverage for new algorithms.
458462

459463
Looking for inspiration? Here are some algorithm ideas that could be valuable:
460464

461-
- **Age-based**: Prioritize deletion of newer/older pods
465+
- **Age-based**: Prioritize deletion of newer/older pods (implemented as the `timestamp` algorithm)
462466
- **Resource-based**: Consider pod resource usage for deletion priority
463467
- **Node-type-based**: Spread deletions across different node types (spot vs on-demand)
464468
- ...

README.md

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A Kubernetes controller that automatically manages the [`controller.kubernetes.io/pod-deletion-cost`](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost) annotation on pods. This annotation influences which pods are terminated first during scale-down operations, enabling smarter and more resilient downscaling behavior.
44

5-
The controller is designed to be **extensible** with a plugin-based architecture, allowing multiple algorithms for calculating pod deletion costs. Currently, it includes a **zone-aware distribution algorithm** that ensures even pod termination across availability zones.
5+
The controller is designed to be **extensible** with a plugin-based architecture, allowing multiple algorithms for calculating pod deletion costs. It currently includes a **zone-aware distribution algorithm** that ensures even pod termination across availability zones, and a **timestamp (age-based) algorithm** that prefers deleting the oldest pods to enable natural fleet refresh.
66

77
## The Problem: Default Kubernetes Scale-Down Behavior
88

@@ -84,6 +84,47 @@ Result: **Even distribution maintained** across all zones.
8484

8585
![Pod Deletion Cost Controller Flow](./docs/images/pod-deletion-cost-controller-flow.gif)
8686

87+
## Algorithm: Timestamp (Age-Based)
88+
89+
The `timestamp` algorithm assigns deletion costs based on pod creation time, so that **among otherwise equivalent eligible pods, older pods are preferred for deletion** during scale-down. Combined with everyday autoscaling, this drives a natural, continuous refresh of the fleet — long-lived pods are gradually recycled without a dedicated rollout.
90+
91+
> **Note:** `pod-deletion-cost` is not the first criterion ReplicaSet uses when choosing which pod to delete. Unassigned, pending, or not-ready pods are still selected before deletion cost is considered. The timestamp algorithm only influences the choice among pods that are otherwise equivalent (assigned, running, ready).
92+
93+
### How It Works
94+
95+
1. **Pod Detection** - Controller watches for pods belonging to enabled Deployments
96+
2. **Cost Calculation** - Assigns cost = **creation unix timestamp bucketed by the minute** (`creationTimestamp / 60`); older pod → smaller timestamp → lower cost → deleted first
97+
3. **Annotation** - Applies `controller.kubernetes.io/pod-deletion-cost` to the pod, once, on first reconciliation
98+
99+
### Design Notes
100+
101+
- **Ordering is reconciliation-time-independent** - The cost is a pure function of `.metadata.creationTimestamp`, not the pod's age at the moment it is reconciled. This is essential because the annotation is written once and never recalculated: deriving cost from age at reconcile time would let a delayed or late-discovered pod (e.g. after a controller restart) receive a stale value that inverts the intended oldest-first order.
102+
- **Deterministic & idempotent** - Because cost depends only on the pod's own creation time, it does not require the expectations cache used by the zone algorithm. Each pod is annotated exactly once (subsequent reconciliations are skipped), respecting the API server load guidance in KEP-2255.
103+
- **int32-safe** - Bucketing the timestamp by the minute keeps the value well within the API server's `int32` range; overflow would not occur until ~year 6053, avoiding the Year-2038 problem a raw `creationTimestamp.Unix()` would hit (a raw timestamp crosses `MaxInt32` on 2038-01-19). The value is additionally clamped to `[-2147483648, 2147483647]`.
104+
- **Minute-granularity ties** - Pods created within the same minute receive the same cost; Kubernetes then falls back to its next tiebreaker (newer-creation-first), which is acceptable for fleet refresh. Note that a large simultaneous scale-up can place many pods in one minute bucket, so strict oldest-first ordering is not guaranteed *within* such a batch.
105+
- **Algorithm ownership** - When the timestamp handler sets a cost, it also stamps the pod with `pod-deletion-cost.lablabs.io/managed-by: timestamp`. If a Deployment switches from another algorithm (e.g. `zone`) to `timestamp`, existing pods still carry the previous algorithm's cost (zone values are near `MaxInt32`). The handler detects that it does not own those values and **recalculates** them, so switching algorithms does not leave pods with stale, incompatible costs that would invert the deletion order.
106+
107+
### Example Scenario
108+
109+
**Initial state:** 3 pods of the same Deployment
110+
111+
```
112+
Pod1 (created 10:00 → cost: 29732280) # oldest
113+
Pod2 (created 11:00 → cost: 29732340)
114+
Pod3 (created 12:00 → cost: 29732400) # newest
115+
```
116+
117+
**After scaling down by 1:**
118+
119+
Kubernetes prefers deleting the pod with the lowest cost — `Pod1`, the oldest.
120+
121+
```
122+
Pod2 (cost: 29732340)
123+
Pod3 (cost: 29732400)
124+
```
125+
126+
Result: **oldest pod preferred for deletion**, enabling gradual fleet refresh.
127+
87128
## Installation
88129

89130
### Helm
@@ -111,6 +152,7 @@ Key configuration options in `values.yaml`:
111152
# Algorithms to enable
112153
algorithms:
113154
- "zone"
155+
- "timestamp"
114156

115157
# Logging configuration
116158
log:
@@ -177,7 +219,7 @@ spec:
177219
| Annotation | Required | Default | Description |
178220
|-----------|----------|---------|-------------|
179221
| `pod-deletion-cost.lablabs.io/enabled` | Yes | - | Set to `"true"` to enable the controller |
180-
| `pod-deletion-cost.lablabs.io/type` | No | `zone` | Algorithm type to use |
222+
| `pod-deletion-cost.lablabs.io/type` | No | `zone` | Algorithm type to use (`zone` or `timestamp`) |
181223
| `pod-deletion-cost.lablabs.io/spread-by` | No | `topology.kubernetes.io/zone` | Node label key for topology spreading |
182224

183225
### Custom Topology Label
@@ -228,6 +270,22 @@ metadata:
228270
pod-deletion-cost.lablabs.io/type: "zone"
229271
```
230272

273+
### Age-Based Deletion (Timestamp Algorithm)
274+
275+
To delete the oldest pods first during scale-down (enabling natural fleet refresh), select the `timestamp` algorithm:
276+
277+
```yaml
278+
apiVersion: apps/v1
279+
kind: Deployment
280+
metadata:
281+
name: my-app
282+
annotations:
283+
pod-deletion-cost.lablabs.io/enabled: "true"
284+
pod-deletion-cost.lablabs.io/type: "timestamp"
285+
```
286+
287+
The `timestamp` algorithm does not use topology labels, so `spread-by` has no effect on it.
288+
231289
## Contributing
232290

233291
The controller uses an extensible plugin-based architecture, making it easy to add new algorithms for different use cases. We welcome contributions!

charts/pod-deletion-cost-controller/Chart.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@ name: pod-deletion-cost-controller
33
description: K8s controller for managing pod-deletion-cost
44

55
type: application
6-
version: 0.0.1
7-
appVersion: "0.0.1"
6+
version: 0.1.0
7+
appVersion: "0.1.0"
88

99
maintainers:
1010
- name: lablabs

charts/pod-deletion-cost-controller/templates/deployment.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,9 @@ spec:
6666
- "{{ .Values.log.stackTraceLevel }}"
6767
- "-zap-time-encoding"
6868
- "{{ .Values.log.timeEncoding }}"
69-
{{- if .Values.algorithms }}
69+
{{- range .Values.algorithms }}
7070
- "-algorithm-type"
71-
- "{{ .Values.algorithms | join "," }}"
71+
- {{ . | quote }}
7272
{{- end }}
7373
ports:
7474
{{- if .Values.metrics.enabled }}

charts/pod-deletion-cost-controller/values.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,8 +77,13 @@ log:
7777
# Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
7878
timeEncoding: epoch
7979

80+
# Algorithms to enable (which handlers are registered).
81+
# Supported values: "zone", "timestamp". An empty list registers all algorithms.
82+
# The per-Deployment `pod-deletion-cost.lablabs.io/type` annotation selects which
83+
# enabled algorithm applies to that workload (defaults to "zone").
8084
algorithms:
8185
- "zone"
86+
- "timestamp"
8287

8388
metrics:
8489
## @param metrics.enabled Enable exposing prometheus metrics

cmd/main.go

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121
"os"
2222
"strings"
2323

24+
"github.com/lablabs/pod-deletion-cost-controller/internal/timestamp"
2425
"github.com/lablabs/pod-deletion-cost-controller/internal/zone"
2526
v1 "k8s.io/api/apps/v1"
2627
corev1 "k8s.io/api/core/v1"
@@ -53,7 +54,15 @@ func (s *sliceFlag) String() string {
5354
}
5455

5556
func (s *sliceFlag) Set(value string) error {
56-
*s = append(*s, value)
57+
// Accept both repeated flags (-algorithm-type zone -algorithm-type timestamp)
58+
// and a single comma-separated value (-algorithm-type zone,timestamp), so a
59+
// misconfigured chart or manual invocation still registers each algorithm.
60+
for _, item := range strings.Split(value, ",") {
61+
item = strings.TrimSpace(item)
62+
if item != "" {
63+
*s = append(*s, item)
64+
}
65+
}
5766
return nil
5867
}
5968

@@ -120,6 +129,11 @@ func main() {
120129
logger.Error(err, "unable to register zone")
121130
os.Exit(1)
122131
}
132+
err = timestamp.Register(logger, moduleMng, mgr.GetClient(), algoType)
133+
if err != nil {
134+
logger.Error(err, "unable to register timestamp")
135+
os.Exit(1)
136+
}
123137
if err := (&controller.PodReconciler{
124138
Client: mgr.GetClient(),
125139
Scheme: mgr.GetScheme(),

cmd/main_test.go

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
package main
2+
3+
import (
4+
"slices"
5+
"testing"
6+
)
7+
8+
func TestSliceFlag_RepeatedFlags(t *testing.T) {
9+
var f sliceFlag
10+
if err := f.Set("zone"); err != nil {
11+
t.Fatalf("unexpected error: %v", err)
12+
}
13+
if err := f.Set("timestamp"); err != nil {
14+
t.Fatalf("unexpected error: %v", err)
15+
}
16+
17+
want := []string{"zone", "timestamp"}
18+
if !slices.Equal([]string(f), want) {
19+
t.Fatalf("got %v, want %v", []string(f), want)
20+
}
21+
if !slices.Contains(f, "zone") || !slices.Contains(f, "timestamp") {
22+
t.Fatalf("both algorithms must be present: %v", []string(f))
23+
}
24+
}
25+
26+
func TestSliceFlag_CommaSeparated(t *testing.T) {
27+
var f sliceFlag
28+
if err := f.Set("zone, timestamp ,"); err != nil {
29+
t.Fatalf("unexpected error: %v", err)
30+
}
31+
32+
// Comma-separated values are split and trimmed; empty entries dropped.
33+
want := []string{"zone", "timestamp"}
34+
if !slices.Equal([]string(f), want) {
35+
t.Fatalf("got %v, want %v", []string(f), want)
36+
}
37+
}
38+
39+
func TestSliceFlag_Mixed(t *testing.T) {
40+
var f sliceFlag
41+
_ = f.Set("zone,timestamp")
42+
_ = f.Set("extra")
43+
44+
want := []string{"zone", "timestamp", "extra"}
45+
if !slices.Equal([]string(f), want) {
46+
t.Fatalf("got %v, want %v", []string(f), want)
47+
}
48+
}

internal/timestamp/handler.go

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
package timestamp
2+
3+
import (
4+
"context"
5+
"math"
6+
7+
"github.com/go-logr/logr"
8+
"github.com/lablabs/pod-deletion-cost-controller/internal/controller"
9+
appv1 "k8s.io/api/apps/v1"
10+
corev1 "k8s.io/api/core/v1"
11+
"sigs.k8s.io/controller-runtime/pkg/client"
12+
)
13+
14+
const (
15+
// TypeAnnotation is the algorithm type identifier for the timestamp handler.
16+
TypeAnnotation = "timestamp"
17+
18+
// ManagedByAnnotation records which algorithm produced the current
19+
// pod-deletion-cost value. It lets the timestamp handler distinguish a value
20+
// it owns (safe to skip) from a stale value written by another algorithm
21+
// (e.g. "zone"), which it must recalculate when a Deployment switches type.
22+
ManagedByAnnotation = "pod-deletion-cost.lablabs.io/managed-by"
23+
24+
// costBucketSeconds is the time granularity (in seconds) used to derive the
25+
// deletion cost from the pod creation timestamp. Bucketing by the minute keeps
26+
// the resulting int32 value far below the API server limit (overflow ~year 6053),
27+
// avoiding the Year-2038 problem a raw unix timestamp would hit.
28+
costBucketSeconds = 60
29+
)
30+
31+
// NewHandler creates a new timestamp-based Handler.
32+
func NewHandler(client client.Client) *Handler {
33+
return &Handler{client: client}
34+
}
35+
36+
// Handler assigns pod-deletion-cost based on pod creation time.
37+
// Older pods get lower cost values and are deleted first during scale-down,
38+
// enabling natural fleet refresh through everyday scaling operations.
39+
type Handler struct {
40+
client client.Client
41+
}
42+
43+
// AcceptType returns the algorithm type identifiers this handler responds to.
44+
func (h *Handler) AcceptType() []string {
45+
return []string{TypeAnnotation}
46+
}
47+
48+
// Handle sets the pod-deletion-cost annotation from the pod's creation timestamp.
49+
// The cost is a pure function of CreationTimestamp (older pod → smaller timestamp →
50+
// lower cost → deleted first), so the resulting ordering is independent of when the
51+
// controller reconciles the pod. This matters because the annotation is written once
52+
// and never recalculated: deriving the cost from the pod age at reconcile time would
53+
// make a delayed or late-discovered pod receive a stale value and could invert the
54+
// intended oldest-first order after a controller restart or reconciliation delay.
55+
//
56+
// A pod is skipped only when the timestamp handler already owns its cost. If the pod
57+
// carries a cost written by a different algorithm (e.g. after a Deployment switches
58+
// from "zone" to "timestamp"), the value is recalculated so the pod is not left with
59+
// a stale, incompatible cost that would invert the deletion order.
60+
func (h *Handler) Handle(ctx context.Context, log logr.Logger, pod *corev1.Pod, _ *appv1.Deployment) error {
61+
if controller.IsDeleting(pod) {
62+
return nil
63+
}
64+
65+
if controller.HasPodDeletionCost(pod) && managedByTimestamp(pod) {
66+
return nil
67+
}
68+
69+
cost := deletionCost(pod.CreationTimestamp.Unix())
70+
71+
patch := client.MergeFrom(pod.DeepCopy())
72+
controller.ApplyPodDeletionCost(pod, cost)
73+
setManagedBy(pod)
74+
if err := h.client.Patch(ctx, pod, patch); err != nil {
75+
return err
76+
}
77+
78+
log.WithValues(controller.PodDeletionCostAnnotation, cost).Info("updated")
79+
return nil
80+
}
81+
82+
// deletionCost maps a pod creation unix timestamp to a pod-deletion-cost value.
83+
// The timestamp is bucketed by costBucketSeconds so the value stays within the
84+
// int32 range the Kubernetes API server enforces, then clamped defensively.
85+
// The mapping is monotonic: an earlier creation time always yields a lower or
86+
// equal cost, so older pods are never ranked above newer ones.
87+
func deletionCost(creationUnix int64) int {
88+
cost := creationUnix / costBucketSeconds
89+
90+
if cost > math.MaxInt32 {
91+
return math.MaxInt32
92+
}
93+
if cost < math.MinInt32 {
94+
return math.MinInt32
95+
}
96+
return int(cost)
97+
}
98+
99+
// managedByTimestamp reports whether the pod's current cost was set by this handler.
100+
func managedByTimestamp(pod *corev1.Pod) bool {
101+
if pod.Annotations == nil {
102+
return false
103+
}
104+
return pod.Annotations[ManagedByAnnotation] == TypeAnnotation
105+
}
106+
107+
// setManagedBy stamps the pod as owned by the timestamp algorithm.
108+
func setManagedBy(pod *corev1.Pod) {
109+
if pod.Annotations == nil {
110+
pod.Annotations = make(map[string]string)
111+
}
112+
pod.Annotations[ManagedByAnnotation] = TypeAnnotation
113+
}

0 commit comments

Comments
 (0)