Skip to content

Commit 6dcc536

Browse files
authored
Merge pull request #296 from finos/feat/k8s-aws-parity
feat(k8s): bring the Kubernetes provider to parity with the AWS provider
2 parents 2161d7c + e4b9f10 commit 6dcc536

184 files changed

Lines changed: 18969 additions & 2006 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,13 @@ terraform.rc
180180

181181
# Application specific
182182
config/config.json
183+
# Runtime-generated per-provider-instance template files (e.g.
184+
# config/aws_<instance>_templates.json, config/k8s_<instance>_templates.json).
185+
# The canonical example files config/aws_templates.json and
186+
# config/k8s_templates.json are tracked and explicitly un-ignored below.
187+
config/*_*_templates.json
188+
!config/aws_templates.json
189+
!config/k8s_templates.json
183190
# Runtime-generated script copies (orb init)
184191
/scripts/
185192
data/

config/k8s_templates.json

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
{
55
"templateId": "k8s-pod-example",
66
"maxNumber": 1,
7-
"imageId": "busybox:latest",
7+
"imageId": "registry.k8s.io/pause:3.9",
88
"priceType": "ondemand",
99
"allocationStrategy": "lowestPrice",
1010
"name": "Kubernetes Pod example",
@@ -15,7 +15,7 @@
1515
{
1616
"templateId": "k8s-deployment-example",
1717
"maxNumber": 3,
18-
"imageId": "busybox:latest",
18+
"imageId": "registry.k8s.io/pause:3.9",
1919
"priceType": "ondemand",
2020
"allocationStrategy": "lowestPrice",
2121
"name": "Kubernetes Deployment example",
@@ -26,7 +26,7 @@
2626
{
2727
"templateId": "k8s-statefulset-example",
2828
"maxNumber": 3,
29-
"imageId": "busybox:latest",
29+
"imageId": "registry.k8s.io/pause:3.9",
3030
"priceType": "ondemand",
3131
"allocationStrategy": "lowestPrice",
3232
"name": "Kubernetes StatefulSet example",
@@ -37,13 +37,27 @@
3737
{
3838
"templateId": "k8s-job-example",
3939
"maxNumber": 3,
40-
"imageId": "busybox:latest",
40+
"imageId": "busybox:1.37",
4141
"priceType": "ondemand",
4242
"allocationStrategy": "lowestPrice",
4343
"name": "Kubernetes Job example",
4444
"providerApi": "Job",
4545
"providerType": "k8s",
46+
"command": ["sh", "-c", "exit 0"],
4647
"createdAt": "2026-07-06T22:07:11.808487"
48+
},
49+
{
50+
"templateId": "k8s-nginx-deployment-native-path",
51+
"maxNumber": 2,
52+
"imageId": "nginx:1.27",
53+
"priceType": "ondemand",
54+
"allocationStrategy": "lowestPrice",
55+
"name": "Kubernetes nginx Deployment via native_spec_path",
56+
"providerApi": "Deployment",
57+
"providerType": "k8s",
58+
"nativeSpecPath": "config/specs/k8s/nginx-deployment.yaml",
59+
"createdAt": "2026-07-12T00:00:00.000000",
60+
"_comment": "Requires native_spec_enabled=true in the k8s provider config. Set ORB_K8S_NATIVE_SPEC_ENABLED=true to use."
4761
}
4862
]
4963
}
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: "{{ resource_name }}"
5+
namespace: "{{ namespace }}"
6+
labels:
7+
app: nginx
8+
orb-request-id: "{{ request_id }}"
9+
spec:
10+
replicas: {{ replicas }}
11+
selector:
12+
matchLabels:
13+
app: nginx
14+
template:
15+
metadata:
16+
labels:
17+
app: nginx
18+
orb-request-id: "{{ request_id }}"
19+
spec:
20+
containers:
21+
- name: nginx
22+
image: "{{ image }}"
23+
ports:
24+
- containerPort: 80
25+
resources:
26+
requests:
27+
cpu: "100m"
28+
memory: "128Mi"
29+
limits:
30+
cpu: "500m"
31+
memory: "256Mi"

docs/root/providers/k8s/auth.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,117 @@ export ORB_K8S_CONTEXT="prod"
131131
| Auth works locally but fails when ORB runs as a system service | The service env does not inherit `$HOME` | Set `ORB_K8S_KUBECONFIG_PATH` to an absolute path. |
132132
| Auth works initially then 401s after some hours (EKS) | Federated token expired | Use `aws eks update-kubeconfig` with a long-lived identity, or run ORB in-cluster with IRSA. |
133133

134+
## Inbound TokenReview auth
135+
136+
ORB can optionally validate Bearer tokens on its own REST API by
137+
submitting them to the Kubernetes
138+
`authentication.k8s.io/v1 TokenReview` endpoint. This lets callers
139+
of ORB's REST API authenticate with a Kubernetes ServiceAccount token
140+
instead of a separate credential system.
141+
142+
### Enabling
143+
144+
Set `inbound_auth_enabled: true` in the provider config (or
145+
`ORB_K8S_INBOUND_AUTH_ENABLED=true`) to register the
146+
`KubeAuthStrategy` with ORB's auth registry.
147+
148+
```json
149+
{
150+
"providers": {
151+
"k8s": {
152+
"provider_type": "k8s",
153+
"namespace": "orb",
154+
"inbound_auth_enabled": true
155+
}
156+
}
157+
}
158+
```
159+
160+
### How it works
161+
162+
1. An inbound REST request arrives with an `Authorization: Bearer <token>` header.
163+
2. ORB extracts the token and submits it to `authentication.k8s.io/v1 TokenReview`
164+
on the cluster.
165+
3. If the API server reports the token is authenticated, the
166+
`system:serviceaccount:<namespace>:<name>` principal is extracted.
167+
4. The principal is mapped to an ORB role using the `sa_role_mapping`
168+
in the auth config. Principals not in the map receive the default
169+
`["user"]` role. Exact matches (`namespace:sa-name`) take priority;
170+
wildcard namespace matches (`*:sa-name`) are tried next.
171+
172+
TokenReview is preferred over local JWKS validation because the API
173+
server is the authoritative signer — there is no possibility of ORB
174+
accepting a token the cluster itself would reject.
175+
176+
### Required RBAC
177+
178+
The ORB pod's ServiceAccount must be able to create `TokenReview`
179+
objects. The recommended approach is to bind the
180+
`system:auth-delegator` ClusterRole (a standard Kubernetes role that
181+
grants precisely this permission):
182+
183+
```yaml
184+
apiVersion: rbac.authorization.k8s.io/v1
185+
kind: ClusterRoleBinding
186+
metadata:
187+
name: orb-auth-delegator
188+
subjects:
189+
- kind: ServiceAccount
190+
name: orb
191+
namespace: orb
192+
roleRef:
193+
apiGroup: rbac.authorization.k8s.io
194+
kind: ClusterRole
195+
name: system:auth-delegator
196+
```
197+
198+
Alternatively, grant only `tokenreviews: create` on the
199+
`authentication.k8s.io` API group via a targeted ClusterRole:
200+
201+
```yaml
202+
apiVersion: rbac.authorization.k8s.io/v1
203+
kind: ClusterRole
204+
metadata:
205+
name: orb-tokenreview
206+
rules:
207+
- apiGroups: ["authentication.k8s.io"]
208+
resources: ["tokenreviews"]
209+
verbs: ["create"]
210+
```
211+
212+
Both options are included (commented out) in [`rbac.yaml`](rbac.yaml).
213+
214+
### Role mapping
215+
216+
Configure which ServiceAccounts map to which ORB roles in the auth
217+
config section (not the provider config):
218+
219+
```json
220+
{
221+
"auth": {
222+
"provider_auth": {
223+
"kubernetes": {
224+
"sa_role_mapping": {
225+
"orb-system:orb-admin": ["admin"],
226+
"orb-system:orb-operator": ["operator"],
227+
"*:ci-runner": ["user"]
228+
}
229+
}
230+
}
231+
}
232+
}
233+
```
234+
235+
Principals not matched by any entry default to `["user"]`.
236+
237+
### Troubleshooting
238+
239+
| Symptom | Likely cause | Fix |
240+
|---|---|---|
241+
| `403 Forbidden` when ORB submits a `TokenReview` | ORB's ServiceAccount lacks `tokenreviews: create` | Apply the auth-delegator ClusterRoleBinding from [`rbac.yaml`](rbac.yaml). |
242+
| Callers get `Token rejected by Kubernetes API server` | Token has expired or is not a valid projected SA token | The caller must obtain a fresh projected token via `kubectl create token`. |
243+
| Auth succeeds but the caller gets the wrong role | SA principal not in `sa_role_mapping` | Add the `namespace:sa-name` entry to the mapping; or it matched a wildcard entry — check the order. |
244+
134245
## Why two wrappers?
135246

136247
ORB confines every `kubernetes` SDK import to

0 commit comments

Comments
 (0)