Skip to content

Commit a9f8dfd

Browse files
committed
feat(k8s): rolling deployment with rollback annotations and docs
- Add rollback annotations to backend and frontend deployments - maxUnavailable: 0 ensures zero-downtime rollouts - Readiness probes gate traffic to new pods - Document deploy, rollback, and migration compatibility steps Closes Nova-reward#938
1 parent c21de8f commit a9f8dfd

3 files changed

Lines changed: 97 additions & 0 deletions

File tree

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Rolling Deployment & Rollback Guide
2+
3+
Closes #938
4+
5+
## Strategy
6+
7+
Both `backend` and `frontend` deployments use `RollingUpdate` with:
8+
9+
```yaml
10+
strategy:
11+
type: RollingUpdate
12+
rollingUpdate:
13+
maxSurge: 1
14+
maxUnavailable: 0
15+
```
16+
17+
`maxUnavailable: 0` guarantees zero downtime — old pods stay up until new pods pass readiness.
18+
19+
## Health Checks
20+
21+
New pods must pass the readiness probe before receiving traffic:
22+
23+
| Service | Probe path | Period | Failure threshold |
24+
|----------|-----------------|--------|-------------------|
25+
| Backend | `GET /health` | 5s | 3 (15s total) |
26+
| Frontend | `GET /api/healthz` | 5s | 3 (15s total) |
27+
28+
A startup probe allows up to 60 s for the container to initialise before liveness kicks in.
29+
30+
## Deploying a New Version
31+
32+
```bash
33+
# Update the image tag
34+
kubectl set image deployment/nova-rewards-backend \
35+
backend=nova-rewards-backend:<NEW_TAG> \
36+
-n nova-rewards
37+
38+
kubectl set image deployment/nova-rewards-frontend \
39+
frontend=nova-rewards-frontend:<NEW_TAG> \
40+
-n nova-rewards
41+
42+
# Watch rollout progress
43+
kubectl rollout status deployment/nova-rewards-backend -n nova-rewards
44+
kubectl rollout status deployment/nova-rewards-frontend -n nova-rewards
45+
```
46+
47+
## Automated Rollback (within 5 minutes)
48+
49+
If new pods fail readiness probes the rollout stalls automatically — no traffic is shifted to broken pods. To revert:
50+
51+
```bash
52+
kubectl rollout undo deployment/nova-rewards-backend -n nova-rewards
53+
kubectl rollout undo deployment/nova-rewards-frontend -n nova-rewards
54+
```
55+
56+
Verify the rollback completed:
57+
58+
```bash
59+
kubectl rollout status deployment/nova-rewards-backend -n nova-rewards
60+
kubectl rollout status deployment/nova-rewards-frontend -n nova-rewards
61+
```
62+
63+
## Database Migrations
64+
65+
Migrations **must be backward-compatible** so old and new code can run simultaneously during the rolling window:
66+
67+
1. **Add columns as nullable** — never add a `NOT NULL` column without a default in the same migration.
68+
2. **Never rename or drop columns** in the same release as the code change. Use a two-phase approach:
69+
- Phase 1: add new column, deploy code that writes to both old and new.
70+
- Phase 2 (next release): remove old column.
71+
3. Run migrations before deploying new pods:
72+
```bash
73+
kubectl run migrate --rm -it --image=nova-rewards-backend:<NEW_TAG> \
74+
--env="DATABASE_URL=$DATABASE_URL" \
75+
-- node database/migrate.js
76+
```
77+
78+
## Rollback History
79+
80+
```bash
81+
# List revision history
82+
kubectl rollout history deployment/nova-rewards-backend -n nova-rewards
83+
84+
# Roll back to a specific revision
85+
kubectl rollout undo deployment/nova-rewards-backend \
86+
--to-revision=<REVISION> -n nova-rewards
87+
```

k8s/backend/deployment.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@ kind: Deployment
33
metadata:
44
name: nova-rewards-backend
55
namespace: nova-rewards
6+
annotations:
7+
# Automated rollback: kubectl rollout undo deployment/nova-rewards-backend -n nova-rewards
8+
# If new pods fail readiness within 5 minutes the deployment stalls; run the above to revert.
9+
deployment.kubernetes.io/revision: "1"
10+
nova-rewards/rollback-window: "5m"
611
spec:
712
replicas: 2
813
selector:

k8s/frontend/deployment.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@ kind: Deployment
33
metadata:
44
name: nova-rewards-frontend
55
namespace: nova-rewards
6+
annotations:
7+
# Automated rollback: kubectl rollout undo deployment/nova-rewards-frontend -n nova-rewards
8+
# If new pods fail readiness within 5 minutes the deployment stalls; run the above to revert.
9+
deployment.kubernetes.io/revision: "1"
10+
nova-rewards/rollback-window: "5m"
611
spec:
712
replicas: 2
813
selector:

0 commit comments

Comments
 (0)