|
| 1 | +# Rolling Deployment & Rollback Guide |
| 2 | + |
| 3 | +Closes #938 |
| 4 | + |
| 5 | +## Strategy |
| 6 | + |
| 7 | +Both `backend` and `frontend` deployments use `RollingUpdate` with: |
| 8 | + |
| 9 | +```yaml |
| 10 | +strategy: |
| 11 | + type: RollingUpdate |
| 12 | + rollingUpdate: |
| 13 | + maxSurge: 1 |
| 14 | + maxUnavailable: 0 |
| 15 | +``` |
| 16 | +
|
| 17 | +`maxUnavailable: 0` guarantees zero downtime — old pods stay up until new pods pass readiness. |
| 18 | + |
| 19 | +## Health Checks |
| 20 | + |
| 21 | +New pods must pass the readiness probe before receiving traffic: |
| 22 | + |
| 23 | +| Service | Probe path | Period | Failure threshold | |
| 24 | +|----------|-----------------|--------|-------------------| |
| 25 | +| Backend | `GET /health` | 5s | 3 (15s total) | |
| 26 | +| Frontend | `GET /api/healthz` | 5s | 3 (15s total) | |
| 27 | + |
| 28 | +A startup probe allows up to 60 s for the container to initialise before liveness kicks in. |
| 29 | + |
| 30 | +## Deploying a New Version |
| 31 | + |
| 32 | +```bash |
| 33 | +# Update the image tag |
| 34 | +kubectl set image deployment/nova-rewards-backend \ |
| 35 | + backend=nova-rewards-backend:<NEW_TAG> \ |
| 36 | + -n nova-rewards |
| 37 | +
|
| 38 | +kubectl set image deployment/nova-rewards-frontend \ |
| 39 | + frontend=nova-rewards-frontend:<NEW_TAG> \ |
| 40 | + -n nova-rewards |
| 41 | +
|
| 42 | +# Watch rollout progress |
| 43 | +kubectl rollout status deployment/nova-rewards-backend -n nova-rewards |
| 44 | +kubectl rollout status deployment/nova-rewards-frontend -n nova-rewards |
| 45 | +``` |
| 46 | + |
| 47 | +## Automated Rollback (within 5 minutes) |
| 48 | + |
| 49 | +If new pods fail readiness probes the rollout stalls automatically — no traffic is shifted to broken pods. To revert: |
| 50 | + |
| 51 | +```bash |
| 52 | +kubectl rollout undo deployment/nova-rewards-backend -n nova-rewards |
| 53 | +kubectl rollout undo deployment/nova-rewards-frontend -n nova-rewards |
| 54 | +``` |
| 55 | + |
| 56 | +Verify the rollback completed: |
| 57 | + |
| 58 | +```bash |
| 59 | +kubectl rollout status deployment/nova-rewards-backend -n nova-rewards |
| 60 | +kubectl rollout status deployment/nova-rewards-frontend -n nova-rewards |
| 61 | +``` |
| 62 | + |
| 63 | +## Database Migrations |
| 64 | + |
| 65 | +Migrations **must be backward-compatible** so old and new code can run simultaneously during the rolling window: |
| 66 | + |
| 67 | +1. **Add columns as nullable** — never add a `NOT NULL` column without a default in the same migration. |
| 68 | +2. **Never rename or drop columns** in the same release as the code change. Use a two-phase approach: |
| 69 | + - Phase 1: add new column, deploy code that writes to both old and new. |
| 70 | + - Phase 2 (next release): remove old column. |
| 71 | +3. Run migrations before deploying new pods: |
| 72 | + ```bash |
| 73 | + kubectl run migrate --rm -it --image=nova-rewards-backend:<NEW_TAG> \ |
| 74 | + --env="DATABASE_URL=$DATABASE_URL" \ |
| 75 | + -- node database/migrate.js |
| 76 | + ``` |
| 77 | + |
| 78 | +## Rollback History |
| 79 | + |
| 80 | +```bash |
| 81 | +# List revision history |
| 82 | +kubectl rollout history deployment/nova-rewards-backend -n nova-rewards |
| 83 | +
|
| 84 | +# Roll back to a specific revision |
| 85 | +kubectl rollout undo deployment/nova-rewards-backend \ |
| 86 | + --to-revision=<REVISION> -n nova-rewards |
| 87 | +``` |
0 commit comments