Skip to content

Commit 26a8cc0

Browse files
dbarkowskydbarkowsky
committed
documentation added/updated
1 parent 2436386 commit 26a8cc0

3 files changed

Lines changed: 235 additions & 26 deletions

File tree

‎docs-md/architecture/HIGH_AVAILABILITY.md‎

Lines changed: 27 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -27,24 +27,24 @@ HPAs automatically scale deployments based on CPU and memory utilization.
2727
### Backend Services HPA
2828

2929
- **Min Replicas:** 2 (for HA)
30-
- **Max Replicas:** 5 (limited by database connection pool capacity)
31-
- **Scale Trigger:** CPU 75% or Memory 80%
32-
- **Scale Up:** Add up to 100% of current pods or 2 pods (whichever is higher) every 30s
30+
- **Max Replicas:** 3 (namespace quota constraint — see [NAMESPACE_CAPACITY.md](NAMESPACE_CAPACITY.md))
31+
- **Scale Trigger:** CPU 80% or Memory 80%
32+
- **Scale Up:** Add up to 50% of current pods or 1 pod (whichever is higher) every 60s; 120 s stabilization window
3333
- **Scale Down:** Remove up to 50% of current pods every 60s, with 5 min stabilization window
3434

3535
**Connection Pool Limit:**
36-
Each backend-services pod uses `DB_POOL_MAX` connections (default **10**). With 5 pods at max HPA scale, that's 50 connections to PostgreSQL.
36+
Each backend-services pod uses `DB_POOL_MAX` connections (default **10**). With 3 pods at max HPA scale, that's 30 connections to PostgreSQL.
3737

3838
### Temporal Worker HPA
3939

4040
- **Min Replicas:** 2 (for HA)
41-
- **Max Replicas:** 4 (limited by database connection pool capacity)
42-
- **Scale Trigger:** CPU 70% or Memory 75%
43-
- **Scale Up:** Add up to 50% of current pods or 1 pod (whichever is higher) every 60s
41+
- **Max Replicas:** 3 (namespace quota constraint — see [NAMESPACE_CAPACITY.md](NAMESPACE_CAPACITY.md))
42+
- **Scale Trigger:** CPU 80% or Memory 80%
43+
- **Scale Up:** Add up to 50% of current pods or 1 pod (whichever is higher) every 60s; 180 s stabilization window
4444
- **Scale Down:** Remove up to 50% of current pods every 120s, with 10 min stabilization window
4545

4646
**Connection Pool Limit:**
47-
Each temporal-worker pod uses `DB_POOL_MAX=3` connections. With 4 pods max, that's 12 connections to PostgreSQL.
47+
Each temporal-worker pod uses `DB_POOL_MAX=3` connections. With 3 pods max, that's 9 connections to PostgreSQL.
4848

4949
**Scale Behavior:**
5050
Temporal workers scale more conservatively than backend-services because:
@@ -55,25 +55,25 @@ Temporal workers scale more conservatively than backend-services because:
5555
### Frontend HPA
5656

5757
- **Min Replicas:** 2 (for HA)
58-
- **Max Replicas:** 4
59-
- **Scale Trigger:** CPU 75% or Memory 80%
60-
- **Scale Up:** Add up to 100% of current pods or 1 pod (whichever is higher) every 30s
58+
- **Max Replicas:** 3 (namespace quota constraint — see [NAMESPACE_CAPACITY.md](NAMESPACE_CAPACITY.md))
59+
- **Scale Trigger:** CPU 80% or Memory 80%
60+
- **Scale Up:** Add up to 50% of current pods or 1 pod (whichever is higher) every 60s; 120 s stabilization window
6161
- **Scale Down:** Remove up to 50% of current pods every 60s, with 5 min stabilization window
6262

6363
**No Database Limit:**
64-
Frontend is nginx serving static files, so no database connections are used. Max replicas can be increased if needed.
64+
Frontend is nginx serving static files, so no database connections are used. Max replicas can be increased if the namespace quota is expanded.
6565

6666
## Database Connection Pool Capacity
6767

6868
PostgreSQL default `max_connections` is 100. Our configuration uses:
6969

7070
| Service | Pods (max) | Connections per Pod | Total Connections |
7171
|---------|------------|---------------------|-------------------|
72-
| backend-services | 5 | 10 | 50 |
73-
| temporal-worker | 4 | 3 | 12 |
74-
| **Total** | **9** | - | **62** |
72+
| backend-services | 3 | 10 | 30 |
73+
| temporal-worker | 3 | 3 | 9 |
74+
| **Total** | **6** | - | **39** |
7575

76-
At HPA max scale this stays below Crunchy Postgres default `max_connections` (100) with headroom for migrations and admin tools. For single-replica load testing, override `DB_POOL_MAX=20` in the instance env to remove the ~7 req/s read ceiling documented in [LOAD_TEST_REPORT_2026-05.md](../archive/LOAD_TEST_REPORT_2026-05.md).
76+
At HPA max scale this stays well below Crunchy Postgres default `max_connections` (100) with headroom for migrations and admin tools. For single-replica load testing, override `DB_POOL_MAX=20` in the instance env to remove the ~7 req/s read ceiling documented in [LOAD_TEST_REPORT_2026-05.md](../archive/LOAD_TEST_REPORT_2026-05.md).
7777

7878
This leaves headroom for:
7979
- Interactive queries (pgAdmin, psql)
@@ -105,11 +105,14 @@ All pods have resource requests and limits defined, which are required for HPA t
105105

106106
| Service | CPU Request | Memory Request | CPU Limit | Memory Limit |
107107
|---------|-------------|----------------|-----------|--------------|
108-
| backend-services | 100m | 256Mi | 500m | 512Mi |
109-
| temporal-worker | 100m | 256Mi | 500m | 512Mi |
108+
| backend-services | 100m | 1Gi | 500m | 1.5Gi |
109+
| temporal-worker | 100m | 768Mi | 500m | 1Gi |
110110
| frontend | 50m | 128Mi | 200m | 256Mi |
111111

112-
These values may need adjustment based on actual workload metrics.
112+
Memory requests are sized to production steady-state observations (backend ~740Mi, worker ~560Mi)
113+
so that HPA memory utilization sits at ~72% and ~73% respectively — below the 80% threshold.
114+
This prevents HPAs from pinning at max replicas under normal (non-peak) load.
115+
See [NAMESPACE_CAPACITY.md](NAMESPACE_CAPACITY.md) for the full capacity model.
113116

114117
## Monitoring HPA
115118

@@ -163,7 +166,7 @@ Each Temporal worker pod has concurrency limits to control parallel task executi
163166
### Activity Concurrency
164167

165168
**Per-pod limit:** 10 concurrent activities
166-
**Total capacity with HPA:** 2-4 pods × 10 = **20-40 concurrent activities**
169+
**Total capacity with HPA:** 2-3 pods × 10 = **20-30 concurrent activities**
167170

168171
**Rationale:**
169172
- Activities are I/O-bound (Azure AI API calls, database queries, blob storage)
@@ -173,13 +176,13 @@ Each Temporal worker pod has concurrency limits to control parallel task executi
173176

174177
**Scaling behavior:**
175178
- Low load: 2 pods × 10 = 20 concurrent activities
176-
- High load: HPA scales to 4 pods × 10 = 40 concurrent activities
177-
- If all 40 slots are busy, new activities queue in Temporal server until a slot opens
179+
- High load: HPA scales to 3 pods × 10 = 30 concurrent activities
180+
- If all 30 slots are busy, new activities queue in Temporal server until a slot opens
178181

179182
### Workflow Task Concurrency
180183

181184
**Per-pod limit:** 100 concurrent workflow decision tasks
182-
**Total capacity with HPA:** 2-4 pods × 100 = **200-400 concurrent decision tasks**
185+
**Total capacity with HPA:** 2-3 pods × 100 = **200-300 concurrent decision tasks**
183186

184187
**Rationale:**
185188
- Workflow tasks are lightweight in-memory operations (decision logic, conditional branching)
@@ -216,6 +219,6 @@ oc port-forward svc/temporal-ui 8080:8080
216219
```bash
217220
# Each activity may hold a DB connection for its duration
218221
# Ensure: (max worker pods * activity concurrency) * DB usage per activity < DB_POOL_MAX * max pods
219-
# Example: 4 pods * 10 activities = 40, with DB_POOL_MAX=3 per pod = 12 total connections
222+
# Example: 3 pods * 10 activities = 30, with DB_POOL_MAX=3 per pod = 9 total connections
220223
# This is safe because not all activities use the DB simultaneously
221224
```

‎docs-md/architecture/NAMESPACE_CAPACITY.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Namespace Capacity Model
2+
3+
This document records the per-instance resource footprint, the resulting namespace quota
4+
headroom, and the rationale for HPA `maxReplicas` and scale-up policy choices.
5+
6+
## Background
7+
8+
Shared OpenShift namespaces (e.g. `fd34fb-test`) apply a `ResourceQuota` that is
9+
shared across every instance deployed to that namespace. When HPAs across multiple
10+
instances scale up simultaneously they can collectively exhaust the quota ceiling,
11+
causing new pods to stay `Pending` (`FailedScheduling`) and rollouts to stall.
12+
13+
Observed quota on `fd34fb-test` (as of 2026-07-17):
14+
15+
| Resource | Used | Limit | % |
16+
|---|---|---|---|
17+
| `requests.cpu` | 3230m | 4000m | 80.75% |
18+
| `requests.memory` | 5736Mi | 16384Mi | 35% |
19+
| `requests.storage` | 64Gi | 64Gi | 100% |
20+
21+
CPU is the binding constraint. Storage exhaustion is addressed separately (PVC sizing).
22+
23+
---
24+
25+
## Per-pod resource footprint
26+
27+
Resources are **request** values, which count against the namespace `ResourceQuota`.
28+
Limits are not shown here; see individual deployment manifests.
29+
30+
### backend-services pod
31+
32+
| Container | CPU req | Memory req |
33+
|---|---|---|
34+
| `backend-services` | 100m | 1Gi (1024Mi) |
35+
| `logrotate` | 10m | 32Mi |
36+
| `promtail` | 50m | 64Mi |
37+
| **Pod total** | **160m** | **1120Mi** |
38+
39+
### temporal-worker pod
40+
41+
| Container | CPU req | Memory req |
42+
|---|---|---|
43+
| `temporal-worker` | 100m | 768Mi |
44+
| `logrotate` | 10m | 32Mi |
45+
| `promtail` | 50m | 64Mi |
46+
| **Pod total** | **160m** | **864Mi** |
47+
48+
### temporal-server pod (2 replicas, no HPA — fixed cost)
49+
50+
| Container | CPU req | Memory req |
51+
|---|---|---|
52+
| `temporal` | 250m | 512Mi |
53+
| `promtail` | 50m | 32Mi |
54+
| **Pod total** | **300m** | **544Mi** |
55+
56+
### temporal-ui pod (1 replica, no HPA — fixed cost)
57+
58+
| Container | CPU req | Memory req |
59+
|---|---|---|
60+
| `temporal-ui` | 50m | 128Mi |
61+
| **Pod total** | **50m** | **128Mi** |
62+
63+
### frontend pod
64+
65+
| Container | CPU req | Memory req |
66+
|---|---|---|
67+
| `frontend` | 50m | 128Mi |
68+
| `promtail` | 50m | 32Mi |
69+
| **Pod total** | **100m** | **160Mi** |
70+
71+
---
72+
73+
## Per-instance footprint
74+
75+
### At minimum replicas (steady-state floor)
76+
77+
| Deployment | Replicas | CPU | Memory |
78+
|---|---|---|---|
79+
| backend-services | 2 | 320m | 2240Mi |
80+
| temporal-worker | 2 | 320m | 1728Mi |
81+
| temporal-server | 2 (fixed) | 600m | 1088Mi |
82+
| temporal-ui | 1 (fixed) | 50m | 128Mi |
83+
| frontend | 2 | 200m | 320Mi |
84+
| **Instance total (min)** | **9** | **1490m** | **5504Mi** |
85+
86+
### At maximum replicas (HPA ceiling)
87+
88+
| Deployment | Replicas | CPU | Memory |
89+
|---|---|---|---|
90+
| backend-services | 3 | 480m | 3360Mi |
91+
| temporal-worker | 3 | 480m | 2592Mi |
92+
| temporal-server | 2 (fixed) | 600m | 1088Mi |
93+
| temporal-ui | 1 (fixed) | 50m | 128Mi |
94+
| frontend | 3 | 300m | 480Mi |
95+
| **Instance total (max)** | **12** | **1910m** | **7648Mi** |
96+
97+
---
98+
99+
## Namespace headroom model (fd34fb-test: 4000m CPU / 16384Mi memory)
100+
101+
| Scenario | CPU used | CPU % | Memory used | Memory % |
102+
|---|---|---|---|---|
103+
| 1 instance @ min | 1490m | 37% | 5504Mi | 34% |
104+
| 1 instance @ max | 1910m | 48% | 7648Mi | 47% |
105+
| **2 instances @ min** | **2980m** | **75%** | **11008Mi** | **67%** |
106+
| 2 instances @ max | 3820m | 96% | 15296Mi | 93% |
107+
| 3 instances @ min | 4470m | **112% — EXCEEDS CPU QUOTA** | 16512Mi | **101% — EXCEEDS MEM QUOTA** |
108+
109+
**Supported configuration: 2 instances per shared namespace.**
110+
111+
- At **min replicas** both instances consume **75% CPU and 67% memory**, leaving headroom
112+
for rolling-update surge pods during deployments.
113+
- At **max replicas** both instances consume **96% CPU and 93% memory**. This worst case
114+
requires both instances to be at full HPA ceiling simultaneously. Under correctly-sized
115+
requests (backend ~72% at steady state, worker ~73%), the memory HPA metric will not
116+
trigger scale-out at normal load — scale-out will only occur under genuine CPU load spikes.
117+
The max scenario is therefore only expected during benchmark or load-test runs.
118+
- **3 instances cannot fit** within either the CPU or memory quota even at minimum replicas.
119+
120+
> **Rolling-update surge budget:** each rolling deployment adds up to 1 extra pod
121+
> (`maxSurge: 1`). During a deploy, a worst-case surge across all 5 HPA-managed deployments
122+
> in 2 instances is 10 extra pods ≈ 1400m additional CPU requests. Because the base
123+
> is at 2980m (75%), there is only 1020m spare — enough for a controlled 2-at-a-time
124+
> deploy but not all 5 services simultaneously. Stagger rollouts when possible.
125+
126+
---
127+
128+
## HPA settings rationale
129+
130+
### maxReplicas reduction
131+
132+
Previous values (backend 5, worker 4, frontend 4) allowed a single instance to reach
133+
2490m CPU. With two instances at peak, aggregate demand was 4980m — 24% above the 4000m
134+
quota ceiling, which caused new pods to remain `Pending`.
135+
136+
New values (all capped at 3) reduce the per-instance peak to 1910m. Two-instance
137+
aggregate at peak: 3820m (96%).
138+
139+
### Target utilization raised to 80%
140+
141+
Previous targets (CPU 70–75%) caused HPAs to trigger scale-outs at relatively low actual
142+
load, consuming namespace headroom that neighbour instances needed. Raising to 80% means
143+
scale-out only when pods are genuinely busy.
144+
145+
### Scale-up stabilization windows extended
146+
147+
| HPA | Old window | New window | Rationale |
148+
|---|---|---|---|
149+
| backend-services | 60 s | 120 s | Prevents reacting to sub-minute CPU spikes (e.g. auth token validation bursts) |
150+
| temporal-worker | 120 s | 180 s | Workers handle long-running activities; short-lived load spikes do not warrant a new pod |
151+
| frontend | 60 s | 120 s | Static nginx; load spikes are almost always cache misses, not sustained |
152+
153+
### Scale-up rate limited to 1 pod / 60 s
154+
155+
Previous backend-services and frontend policies allowed doubling pod count every 30 s
156+
(100% per 30 s). Replacing with `max(50% per 60 s, 1 pod per 60 s)` makes scaling
157+
additive at one pod per minute, keeping aggregate namespace CPU growth predictable.
158+
159+
---
160+
161+
## Database connection headroom
162+
163+
With `maxReplicas: 3` for both backend-services and temporal-worker:
164+
165+
| Service | Max pods | DB_POOL_MAX | Max connections |
166+
|---|---|---|---|
167+
| backend-services | 3 | 10 | 30 |
168+
| temporal-worker | 3 | 3 | 9 |
169+
| **Total** | | | **39** |
170+
171+
This is well within the Crunchy Postgres default `max_connections = 100`, leaving
172+
headroom for migrations, monitoring tools, and pgAdmin.
173+
174+
---
175+
176+
## Storage quota (separate concern)
177+
178+
The `fd34fb-test` `storage-quota` of 64Gi is 100% consumed independently of compute
179+
scaling. Root causes include:
180+
181+
- CrunchyDB WAL and backup PVCs (default `PG_BACKUP_STORAGE_SIZE=10Gi`)
182+
- MinIO document storage PVCs
183+
184+
See `docs-md/operations/ENVIRONMENT_CONFIGURATION.md` for `PG_BACKUP_STORAGE_SIZE`
185+
guidance. Reduce to `2Gi` for short-lived test instances. Storage quota exhaustion
186+
is an operational concern and does not interact with HPA behaviour.
187+
188+
---
189+
190+
## Monitoring commands
191+
192+
```bash
193+
# Quota utilisation
194+
oc describe resourcequota -n <namespace>
195+
196+
# HPA status across all instances
197+
oc get hpa -n <namespace>
198+
199+
# Detailed HPA events (shows scale decisions)
200+
oc describe hpa <instance>-backend-services -n <namespace>
201+
oc describe hpa <instance>-temporal-worker -n <namespace>
202+
oc describe hpa <instance>-frontend -n <namespace>
203+
204+
# Current pod CPU/memory vs requests
205+
oc adm top pods -n <namespace> --containers
206+
```

‎docs-md/benchmarking/LOAD_TESTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -431,9 +431,9 @@ Assessment of **current** sample manifests (not a prescription for production wi
431431

432432
| Topic | Finding | References |
433433
|-------|---------|------------|
434-
| Backend replicas | `replicas: 2` plus HPA (`minReplicas: 2`, `maxReplicas: 5`) | [backend-services/deployment.yml](../../deployments/openshift/kustomize/base/backend-services/deployment.yml), [horizontalpodautoscaler.yml](../../deployments/openshift/kustomize/base/backend-services/horizontalpodautoscaler.yml) |
434+
| Backend replicas | `replicas: 2` plus HPA (`minReplicas: 2`, `maxReplicas: 3`) | [backend-services/deployment.yml](../../deployments/openshift/kustomize/base/backend-services/deployment.yml), [horizontalpodautoscaler.yml](../../deployments/openshift/kustomize/base/backend-services/horizontalpodautoscaler.yml) |
435435
| Backend rollout | `strategy: RollingUpdate` (`maxSurge: 1`) — no deploy downtime | Same deployment file |
436-
| Worker replicas | `replicas: 2`, `RollingUpdate`, HPA (`minReplicas: 2`, `maxReplicas: 4`) | [temporal-worker-deployment.yml](../../deployments/openshift/kustomize/base/temporal/temporal-worker-deployment.yml), [horizontalpodautoscaler-worker.yml](../../deployments/openshift/kustomize/base/temporal/horizontalpodautoscaler-worker.yml) |
436+
| Worker replicas | `replicas: 2`, `RollingUpdate`, HPA (`minReplicas: 2`, `maxReplicas: 3`) | [temporal-worker-deployment.yml](../../deployments/openshift/kustomize/base/temporal/temporal-worker-deployment.yml), [horizontalpodautoscaler-worker.yml](../../deployments/openshift/kustomize/base/temporal/horizontalpodautoscaler-worker.yml) |
437437
| Temporal server | `replicas: 2` | [temporal-server-deployment.yml](../../deployments/openshift/kustomize/base/temporal/temporal-server-deployment.yml) |
438438
| PodDisruptionBudget | Defined for backend, Temporal server, and worker (`minAvailable: 1`) | [backend-services/poddisruptionbudget.yml](../../deployments/openshift/kustomize/base/backend-services/poddisruptionbudget.yml), [poddisruptionbudget-server.yml](../../deployments/openshift/kustomize/base/temporal/poddisruptionbudget-server.yml), [poddisruptionbudget-worker.yml](../../deployments/openshift/kustomize/base/temporal/poddisruptionbudget-worker.yml) |
439439
| Postgres HA | Crunchy `instances[0].replicas: 1` in both the app and Temporal clusters — single database instance each | [crunchydb/postgrescluster.yml](../../deployments/openshift/kustomize/base/crunchydb/postgrescluster.yml), [temporal/postgrescluster.yml](../../deployments/openshift/kustomize/base/temporal/postgrescluster.yml) |

0 commit comments

Comments
 (0)