Skip to content

Commit e80744b

Browse files
fgogollifgogolli
committed
docs(k8s): correct field defaults, complete config reference, and document scaling, TokenReview auth, and disruption-watch features
Fix documented defaults that disagreed with the code (reject_high_risk_pod_fields defaults to True, namespace auto-detects rather than defaulting to "default"), correct the container image and image-pull-secret field names, and enumerate the full provider configuration surface with types, defaults, and environment variables. Add an AWS-to-Kubernetes concept guide and document configurable resource naming, START/STOP via replica scaling, inbound TokenReview authentication, the node-disruption events watcher, and the periodic LIST resync backstop, with matching RBAC grants.
1 parent a273f1f commit e80744b

8 files changed

Lines changed: 482 additions & 82 deletions

File tree

‎docs/root/providers/k8s/auth.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,117 @@ export ORB_K8S_CONTEXT="prod"
131131
| Auth works locally but fails when ORB runs as a system service | The service env does not inherit `$HOME` | Set `ORB_K8S_KUBECONFIG_PATH` to an absolute path. |
132132
| Auth works initially then 401s after some hours (EKS) | Federated token expired | Use `aws eks update-kubeconfig` with a long-lived identity, or run ORB in-cluster with IRSA. |
133133

134+
## Inbound TokenReview auth
135+
136+
ORB can optionally validate Bearer tokens on its own REST API by
137+
submitting them to the Kubernetes
138+
`authentication.k8s.io/v1 TokenReview` endpoint. This lets callers
139+
of ORB's REST API authenticate with a Kubernetes ServiceAccount token
140+
instead of a separate credential system.
141+
142+
### Enabling
143+
144+
Set `inbound_auth_enabled: true` in the provider config (or
145+
`ORB_K8S_INBOUND_AUTH_ENABLED=true`) to register the
146+
`KubeAuthStrategy` with ORB's auth registry.
147+
148+
```json
149+
{
150+
"providers": {
151+
"k8s": {
152+
"provider_type": "k8s",
153+
"namespace": "orb",
154+
"inbound_auth_enabled": true
155+
}
156+
}
157+
}
158+
```
159+
160+
### How it works
161+
162+
1. An inbound REST request arrives with an `Authorization: Bearer <token>` header.
163+
2. ORB extracts the token and submits it to `authentication.k8s.io/v1 TokenReview`
164+
on the cluster.
165+
3. If the API server reports the token is authenticated, the
166+
`system:serviceaccount:<namespace>:<name>` principal is extracted.
167+
4. The principal is mapped to an ORB role using the `sa_role_mapping`
168+
in the auth config. Principals not in the map receive the default
169+
`["user"]` role. Exact matches (`namespace:sa-name`) take priority;
170+
wildcard namespace matches (`*:sa-name`) are tried next.
171+
172+
TokenReview is preferred over local JWKS validation because the API
173+
server is the authoritative signer — there is no possibility of ORB
174+
accepting a token the cluster itself would reject.
175+
176+
### Required RBAC
177+
178+
The ORB pod's ServiceAccount must be able to create `TokenReview`
179+
objects. The recommended approach is to bind the
180+
`system:auth-delegator` ClusterRole (a standard Kubernetes role that
181+
grants precisely this permission):
182+
183+
```yaml
184+
apiVersion: rbac.authorization.k8s.io/v1
185+
kind: ClusterRoleBinding
186+
metadata:
187+
name: orb-auth-delegator
188+
subjects:
189+
- kind: ServiceAccount
190+
name: orb
191+
namespace: orb
192+
roleRef:
193+
apiGroup: rbac.authorization.k8s.io
194+
kind: ClusterRole
195+
name: system:auth-delegator
196+
```
197+
198+
Alternatively, grant only `tokenreviews: create` on the
199+
`authentication.k8s.io` API group via a targeted ClusterRole:
200+
201+
```yaml
202+
apiVersion: rbac.authorization.k8s.io/v1
203+
kind: ClusterRole
204+
metadata:
205+
name: orb-tokenreview
206+
rules:
207+
- apiGroups: ["authentication.k8s.io"]
208+
resources: ["tokenreviews"]
209+
verbs: ["create"]
210+
```
211+
212+
Both options are included (commented out) in [`rbac.yaml`](rbac.yaml).
213+
214+
### Role mapping
215+
216+
Configure which ServiceAccounts map to which ORB roles in the auth
217+
config section (not the provider config):
218+
219+
```json
220+
{
221+
"auth": {
222+
"provider_auth": {
223+
"kubernetes": {
224+
"sa_role_mapping": {
225+
"orb-system:orb-admin": ["admin"],
226+
"orb-system:orb-operator": ["operator"],
227+
"*:ci-runner": ["user"]
228+
}
229+
}
230+
}
231+
}
232+
}
233+
```
234+
235+
Principals not matched by any entry default to `["user"]`.
236+
237+
### Troubleshooting
238+
239+
| Symptom | Likely cause | Fix |
240+
|---|---|---|
241+
| `403 Forbidden` when ORB submits a `TokenReview` | ORB's ServiceAccount lacks `tokenreviews: create` | Apply the auth-delegator ClusterRoleBinding from [`rbac.yaml`](rbac.yaml). |
242+
| Callers get `Token rejected by Kubernetes API server` | Token has expired or is not a valid projected SA token | The caller must obtain a fresh projected token via `kubectl create token`. |
243+
| Auth succeeds but the caller gets the wrong role | SA principal not in `sa_role_mapping` | Add the `namespace:sa-name` entry to the mapping; or it matched a wildcard entry — check the order. |
244+
134245
## Why two wrappers?
135246

136247
ORB confines every `kubernetes` SDK import to

‎docs/root/providers/k8s/configuration.md‎

Lines changed: 163 additions & 33 deletions
Large diffs are not rendered by default.

‎docs/root/providers/k8s/handlers.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -39,20 +39,20 @@ it whenever you do not need controller-driven replica management.
3939

4040
### Template fields the Pod handler honours
4141

42-
| Template field | Meaning |
43-
|-------------------------|--------------------------------------------------------------------------|
44-
| `container_image` | Required. Mapped to `spec.containers[0].image`. |
45-
| `namespace` | Override the provider-level `namespace`. |
46-
| `resource_requests` | Container resource requests (e.g. `{"cpu":"1","memory":"2Gi"}`). |
47-
| `resource_limits` | Container resource limits. |
48-
| `node_selector` | `spec.nodeSelector`. |
49-
| `tolerations` | `spec.tolerations`. |
50-
| `service_account` | `spec.serviceAccountName`. |
51-
| `runtime_class` | `spec.runtimeClassName`. |
52-
| `environment_variables` | Injected into the container env. |
53-
| `image_pull_secrets` | Appended to `spec.imagePullSecrets`. |
54-
| `labels` | Merged into pod metadata labels (ORB labels always win). |
55-
| `annotations` | Merged into pod metadata annotations. |
42+
| Template field | Meaning |
43+
|---------------------|--------------------------------------------------------------------------|
44+
| `image_id` | Required. Container image reference (`registry/name:tag`). Mapped to `spec.containers[0].image`. |
45+
| `namespace` | Override the provider-level `namespace`. |
46+
| `resource_requests` | Container resource requests (e.g. `{"cpu":"1","memory":"2Gi"}`). |
47+
| `resource_limits` | Container resource limits. |
48+
| `node_selector` | `spec.nodeSelector`. |
49+
| `tolerations` | `spec.tolerations`. |
50+
| `service_account` | `spec.serviceAccountName`. |
51+
| `runtime_class` | `spec.runtimeClassName`. |
52+
| `env` | Container environment variables. Accepts a `dict[str,str]` or a list of `{"name":…,"value":…}` objects. |
53+
| `image_pull_secret` | Single image-pull secret name added to `spec.imagePullSecrets`. |
54+
| `labels` | Merged into pod metadata labels (ORB labels always win). |
55+
| `annotations` | Merged into pod metadata annotations. |
5656

5757
## `Deployment`
5858

‎docs/root/providers/k8s/index.md‎

Lines changed: 68 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ orb templates generate --provider-name kubernetes --provider-api Pod
7878
```
7979

8080
This emits a template that targets the Kubernetes provider's Pod handler.
81-
Tweak `container_image`, `resource_requests`, `resource_limits`, and any
81+
Tweak `image_id`, `resource_requests`, `resource_limits`, and any
8282
`node_selector` / `tolerations` to match your cluster, then save it.
8383

8484
### 4. Request capacity
@@ -102,6 +102,70 @@ Releases trigger a `delete_namespaced_pod` call (or the appropriate
102102
controller-driven replica reduction for Deployment / StatefulSet / Job
103103
workloads).
104104

105+
## AWS concepts mapped to Kubernetes
106+
107+
If you are familiar with the AWS provider, this table shows the
108+
equivalent concept in the Kubernetes world.
109+
110+
| AWS concept | Kubernetes equivalent |
111+
|-----------------------------------|---------------------------------------------------------------------------|
112+
| EC2 instance | Pod |
113+
| Auto Scaling Group / EC2 Fleet | Deployment or StatefulSet (controller manages the replica set) |
114+
| Amazon Machine Image (AMI) | Container image (`image_id` field, e.g. `registry/name:tag`) |
115+
| Instance type label | Node label (`node.kubernetes.io/instance-type` or custom node selector) |
116+
| Spot / On-Demand capacity type | Karpenter `karpenter.sh/capacity-type: spot` / `on-demand` node label |
117+
| `terminate` | Pod `delete`, or scale Deployment / StatefulSet to 0 |
118+
| `start` / `stop` | Scale Deployment / StatefulSet `spec.replicas` between 0 and N (see [START/STOP operations](#startstop-operations)) |
119+
| IAM instance profile | Kubernetes ServiceAccount (`service_account` template field) |
120+
| AWS region | Kubernetes namespace or cluster context |
121+
| EC2 security group | NetworkPolicy |
122+
| EBS volume | PersistentVolumeClaim (declare via `volumeClaimTemplates` on StatefulSet) |
123+
| `provider_api_spec` native body | `native_spec` field on `K8sTemplate` (see [Native spec escape hatch](native-spec.md)) |
124+
125+
Key differences to keep in mind:
126+
127+
* ORB identifies managed resources by the `orb.io/request-id` label,
128+
not by name. Names are cosmetic and generated according to the
129+
[naming policy](#configurable-resource-naming); do not rely on them in
130+
external scripts.
131+
* Kubernetes does not have a machine-level "stopped" state for bare
132+
Pods or Jobs. START and STOP operations are only meaningful for
133+
Deployment and StatefulSet workloads (scale to 0 / restore).
134+
135+
## START/STOP operations
136+
137+
For Deployment and StatefulSet workloads, ORB maps the
138+
`START_INSTANCES` and `STOP_INSTANCES` operations to `spec.replicas`
139+
scaling:
140+
141+
* **STOP** — patches `spec.replicas` to `0` and archives the original
142+
count in `provider_data["replicas_before_stop"]`. All pods are
143+
terminated by the controller.
144+
* **START** — patches `spec.replicas` back to the archived pre-stop
145+
count (or falls back to the acquire-time count).
146+
147+
Pod and Job workloads return an `UNSUPPORTED_OPERATION_FOR_KIND` error
148+
because pods and jobs cannot be meaningfully paused and resumed.
149+
150+
Required RBAC (`deployments/scale` and `statefulsets/scale` get/patch)
151+
is included in the baseline [`rbac.yaml`](rbac.yaml).
152+
153+
## Configurable resource naming
154+
155+
Every managed resource receives a name generated as
156+
`<prefix>-<uuid_segment>` for controller kinds (Deployment, StatefulSet,
157+
Job) and `<prefix>-<uuid_segment>-<seq:04d>` for individual Pods, where
158+
`uuid_segment` is the first `uuid_chars` hex characters of the
159+
hyphen-stripped request UUID.
160+
161+
Names are cosmetic — ORB recovers state via the `orb.io/request-id`
162+
label, not by parsing the name. The defaults reproduce the historical
163+
naming pattern so upgrades do not affect existing resources.
164+
165+
The naming policy is controlled by the `naming` field on
166+
`K8sProviderConfig`. See [Configuration reference](configuration.md#resource-naming)
167+
for all available fields.
168+
105169
## What is in this section
106170

107171
* [Infrastructure discovery](discovery.md) - interactive `orb init` flow, the
@@ -111,12 +175,12 @@ workloads).
111175
* [Handlers](handlers.md) - Pod, Deployment, StatefulSet, Job; when to pick each.
112176
* [Native spec escape hatch](native-spec.md) - submit a full kubernetes
113177
API body and bypass the typed builders for fields ORB does not model.
114-
* [Authentication](auth.md) - in-cluster vs kubeconfig, troubleshooting.
115-
* [RBAC example](rbac.yaml) - minimum ServiceAccount + Role + RoleBinding.
178+
* [Authentication](auth.md) - in-cluster vs kubeconfig, inbound TokenReview auth, troubleshooting.
179+
* [RBAC example](rbac.yaml) - minimum ServiceAccount + Role + RoleBinding, with all opt-in grants documented.
116180
* [Migrating from `orb.k8s_legacy`](migrating-from-k8s-legacy.md) - template field
117181
mapping, label deltas, coexistence guidance.
118182
* [Security hardening](security-hardening.md) - pod-spec audit, high-risk
119-
field warnings, and how to enable reject mode.
183+
field reject mode (on by default), and how to disable it for legitimate workloads.
120184
* [Authoring a provider plugin](plugin-authoring.md) - extending the
121185
provider via the `orb.providers` entry-point group, with a worked
122186
MPIJob example.

‎docs/root/providers/k8s/migrating-from-k8s-legacy.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ operators can plan their migration with full visibility.
1717
| Code location | `src/orb/k8s_legacy/` | `src/orb/providers/k8s/` |
1818
| Architecture | Standalone HostFactory plugin with its own watchers + storage | First-class ORB provider behind the standard `ProviderStrategy` contract |
1919
| State store | Filesystem workdir (`/var/tmp/hostfactory`) + event log | ORB primary storage (SQLite / DynamoDB / SQL - operator's choice) |
20-
| Templates | Legacy template format with HF-camelCase fields | ORB template aggregate (`provider_api`, `container_image`, etc.) |
20+
| Templates | Legacy template format with HF-camelCase fields | ORB template aggregate (`provider_api`, `image_id`, etc.) |
2121
| Identifying labels | `symphony/open-resource-broker-reqid` | `orb.io/managed`, `orb.io/request-id`, `orb.io/machine-id` |
2222
| Workloads supported | Bare pods | Pod, Deployment, StatefulSet, Job (see [Handlers](handlers.md)) |
2323
| HostFactory API | Native HF JSON (`requestMachines.sh` etc.) | Same HF JSON, via ORB's HostFactory adapter |
@@ -98,7 +98,7 @@ complete, flip it to `false`.
9898
| Legacy template field | Modern equivalent | Notes |
9999
|-----------------------------------|------------------------------------------------|-----------------------------------------------------------------------------|
100100
| `templateId` / `template_id` | `template_id` | Same field, snake_case at rest. |
101-
| `imageId` | `container_image` | Modern field always carries the full OCI ref (`registry/name:tag`). |
101+
| `imageId` | `image_id` | Canonical field on the base `Template` aggregate; carries the full OCI ref (`registry/name:tag`). |
102102
| `attributes.namespace` | `namespace` (on template) or `namespace` (on provider config) | Per-template wins. |
103103
| `attributes.serviceAccountName` | `service_account` | Maps to `spec.serviceAccountName`. |
104104
| `attributes.runtimeClassName` | `runtime_class` | Maps to `spec.runtimeClassName`. |
@@ -107,7 +107,7 @@ complete, flip it to `false`.
107107
| `attributes.resources.requests` | `resource_requests` | Same `dict[str,str]` shape (e.g. `{"cpu":"1","memory":"2Gi"}`). |
108108
| `attributes.resources.limits` | `resource_limits` | Same shape as requests. |
109109
| `attributes.environment` | `env` | `dict[str,str]`. |
110-
| `attributes.imagePullSecrets` | `image_pull_secret` | `str` — single image-pull secret name. |
110+
| `attributes.imagePullSecrets` | `image_pull_secret` | `str` — single image-pull secret name added to `spec.imagePullSecrets`. |
111111
| `attributes.labels` | `labels` | ORB-emitted labels always win when key conflicts arise. |
112112
| `attributes.annotations` | `annotations` | Free-form passthrough. |
113113
| n/a | `provider_api` | New required field - `Pod`, `Deployment`, `StatefulSet`, or `Job`. |

‎docs/root/providers/k8s/plugin-authoring.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,7 @@ class KubernetesMPIJobHandler(K8sHandlerBase):
255255
"containers": [
256256
{
257257
"name": "mpi-launcher",
258-
"image": template.container_image,
258+
"image": template.image_id,
259259
}
260260
],
261261
},
@@ -269,7 +269,7 @@ class KubernetesMPIJobHandler(K8sHandlerBase):
269269
"containers": [
270270
{
271271
"name": "mpi-worker",
272-
"image": template.container_image,
272+
"image": template.image_id,
273273
}
274274
],
275275
},

0 commit comments

Comments
 (0)