Skip to content

Commit a67ceff

Browse files
yrahalyrahal
committed
Add service orchestrator and orchestrator seeding to the chart
- serviceOrchestrator (off by default): a ConfigMap with per-deployment builtin service overrides, mounted into teams-api and wired via FIFTYONE_BUILTIN_SERVICES_PATH, plus a ConfigMap with the jinja2 pod template the kubernetes-service broker renders on-demand service pods from - seedOrchestrators (off by default, independent): a post-install/post-upgrade job that upserts orchestrator registrations of any environment in Mongo from values, script in files/seed_orchestrators.py - Both ship empty lists by default, deployments define their own entries in values
1 parent 5ca0128 commit a67ceff

10 files changed

Lines changed: 656 additions & 0 deletions

‎helm/fiftyone-teams-app/README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1036,10 +1036,28 @@ If pods show unhealthy states (e.g., `0/1`, `CrashLoopBackOff`, `Pending`):
10361036
| secret.fiftyone.mongodbConnectionString | string | `""` | MongoDB Connection String. [Reference][mongodb-connection-string]. |
10371037
| secret.labels | object | `{}` | Additional labels for the generated secret. [Reference][labels-and-selectors]. |
10381038
| secret.name | string | `"fiftyone-teams-secrets"` | Name of the secret (existing or to be created) in the namespace `namespace.name`. |
1039+
| seedOrchestrators.backoffLimit | int | `2` | Number of retries before the seeding job is marked "Failed". [Reference][job-backoff-failure-policy]. |
1040+
| seedOrchestrators.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true}` | Container security configuration for the seeding job container. [Reference][container-security-context]. |
1041+
| seedOrchestrators.enabled | bool | `false` | Controls whether to create a post-install/post-upgrade `Job` that upserts the orchestrator registrations below in Mongo (by `instance_id`), so they are versioned here instead of hand-created. Covers any orchestrator environment (kubernetes delegated-operator jobs, kubernetes-service pods, ...). See files/seed_orchestrators.py. |
1042+
| seedOrchestrators.image | object | `{}` | Container image for the seeding job. Empty fields default to `delegatedOperatorJobTemplates.template.image`. |
1043+
| seedOrchestrators.orchestrators | list | `[]` | Orchestrator registrations to upsert. Each entry requires `instance_id`, `description`, `environment`, and `config`; `secrets` and `available_operators` are optional. |
1044+
| seedOrchestrators.podSecurityContext | object | `{"runAsNonRoot":false}` | Pod-level security attributes for the seeding job pod. [Reference][security-context]. |
1045+
| seedOrchestrators.resources | object | `{"limits":{"cpu":"250m","memory":"512Mi"},"requests":{"cpu":"250m","memory":"512Mi"}}` | Resources for the seeding job container. [Reference][resources]. |
1046+
| seedOrchestrators.ttlSecondsAfterFinished | int | `600` | Seconds the finished job is retained. [Reference][job-termination-and-cleanup]. |
10391047
| serviceAccount.annotations | object | `{}` | Service Account annotations. [Reference][annotations]. |
10401048
| serviceAccount.create | bool | `true` | Controls whether to create the service account named `serviceAccount.name`. |
10411049
| serviceAccount.labels | object | `{}` | Additional labels for the generated service account. [Reference][labels-and-selectors]. |
10421050
| serviceAccount.name | string | `"fiftyone-teams"` | Name of the service account (existing or to be created) in the namespace `namespace.name` used for deployments. [Reference][service-account]. |
1051+
| serviceOrchestrator.builtinServices.configMap.annotations | object | `{}` | ConfigMap annotations. [Reference][annotations]. |
1052+
| serviceOrchestrator.builtinServices.configMap.create | bool | `true` | Controls whether to create the `ConfigMap` named `serviceOrchestrator.builtinServices.configMap.name`. |
1053+
| serviceOrchestrator.builtinServices.configMap.labels | object | `{}` | Additional labels for the generated `ConfigMap`. [Reference][labels-and-selectors]. |
1054+
| serviceOrchestrator.builtinServices.configMap.name | string | `""` | Name of the `ConfigMap` (existing or to be created) in the namespace `namespace.name` holding the builtin services overrides. Defaults to `release-name-fiftyone-teams-app-builtin-services`. |
1055+
| serviceOrchestrator.builtinServices.services | list | `[]` | Builtin service overrides, deep-merged by `id` onto the defaults packaged in fiftyone (`builtin_version` must be bumped for a changed entry to re-apply to an environment that already stores the builtin). |
1056+
| serviceOrchestrator.enabled | bool | `false` | Controls whether to create the service-orchestrator resources below (builtin services overrides and the service pod template) and wire them into `teams-api`. |
1057+
| serviceOrchestrator.podTemplate.configMap.annotations | object | `{}` | ConfigMap annotations. [Reference][annotations]. |
1058+
| serviceOrchestrator.podTemplate.configMap.create | bool | `true` | Controls whether to create the `ConfigMap` named `serviceOrchestrator.podTemplate.configMap.name`. |
1059+
| serviceOrchestrator.podTemplate.configMap.labels | object | `{}` | Additional labels for the generated `ConfigMap`. [Reference][labels-and-selectors]. |
1060+
| serviceOrchestrator.podTemplate.configMap.name | string | `""` | Name of the `ConfigMap` (existing or to be created) in the namespace `namespace.name` holding the service pod template. Defaults to `release-name-fiftyone-teams-app-service-pod-template`. |
10431061
| teamsAppSettings.affinity | object | `{}` | Affinity and anti-affinity for `teams-app`. [Reference][affinity]. |
10441062
| teamsAppSettings.autoscaling.enabled | bool | `false` | Controls horizontal pod autoscaling for `teams-app`. [Reference][autoscaling]. |
10451063
| teamsAppSettings.autoscaling.maxReplicas | int | `5` | Maximum Replicas for horizontal autoscaling for `teams-app`. |

‎helm/fiftyone-teams-app/files/seed_orchestrators.py‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
"""Seeds a deployment's orchestrator registrations in Mongo, so they are
2+
versioned in the deployment's values instead of hand-created.
3+
4+
Runs as a helm post-install/post-upgrade Job (see
5+
seed-orchestrators-job.yaml). Upserts by instance_id: config,
6+
description, environment, and secrets are re-applied on every run;
7+
created_at and available_operators are only written when the document is
8+
first created (the Refresh action in the app re-discovers operators for job
9+
targets). Talks directly to Mongo with the deployment's existing teams
10+
secrets, so no API key is required.
11+
12+
The orchestrator list arrives as JSON in the ORCHESTRATORS env var, rendered
13+
by helm from `seedOrchestrators.orchestrators`. An entry may set its
14+
`config.image` to "" to have it filled from DEFAULT_WORKER_IMAGE (the
15+
deployment's delegated-operator worker image), so the registration tracks
16+
image bumps instead of pinning a tag in values.
17+
"""
18+
19+
import datetime
20+
import json
21+
import os
22+
23+
import pymongo
24+
25+
orchestrators = json.loads(os.environ["ORCHESTRATORS"])
26+
27+
client = pymongo.MongoClient(os.environ["FIFTYONE_DATABASE_URI"])
28+
coll = client[os.environ["FIFTYONE_DATABASE_NAME"]]["orchestrators"]
29+
now = datetime.datetime.now(datetime.timezone.utc)
30+
31+
for orc in orchestrators:
32+
if orc.get("config", {}).get("image") == "":
33+
orc["config"]["image"] = os.environ["DEFAULT_WORKER_IMAGE"]
34+
set_fields = {
35+
"description": orc["description"],
36+
"environment": orc["environment"],
37+
"config": orc["config"],
38+
"secrets": orc.get("secrets", {}),
39+
"updated_at": now,
40+
}
41+
if "available_operators" in orc:
42+
set_fields["available_operators"] = orc["available_operators"]
43+
insert_fields = {
44+
"instance_id": orc["instance_id"],
45+
"created_at": now,
46+
}
47+
result = coll.update_one(
48+
{"instance_id": orc["instance_id"]},
49+
{"$set": set_fields, "$setOnInsert": insert_fields},
50+
upsert=True,
51+
)
52+
action = "created" if result.upserted_id else "updated"
53+
print(f"{action} orchestrator {orc['instance_id']}")
54+
55+
print("orchestrator seeding complete")

‎helm/fiftyone-teams-app/templates/_helpers.tpl‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -387,6 +387,10 @@ Create a merged list of environment variables for fiftyone-teams-api
387387
secretKeyRef:
388388
name: {{ $secretName }}
389389
key: fiftyoneDatabaseName
390+
{{- if and .Values.serviceOrchestrator.enabled (or .Values.serviceOrchestrator.builtinServices.configMap.create .Values.serviceOrchestrator.builtinServices.configMap.name) }}
391+
- name: FIFTYONE_BUILTIN_SERVICES_PATH
392+
value: /opt/builtin-services/builtin_services.yaml
393+
{{- end }}
390394
{{- include "telemetry.redis-url-env" . }}
391395
{{- range $key, $val := .Values.apiSettings.env }}
392396
- name: {{ $key }}

‎helm/fiftyone-teams-app/templates/_service_orchestrator.tpl‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{{/*
2+
Create the name of the builtin-services ConfigMap to use
3+
*/}}
4+
{{- define "service-orchestrator.builtin-services-config-map-name" }}
5+
{{- if .Values.serviceOrchestrator.builtinServices.configMap.name }}
6+
{{- .Values.serviceOrchestrator.builtinServices.configMap.name | trunc 63 | trimSuffix "-" }}
7+
{{- else }}
8+
{{- printf "%s-builtin-services" (include "fiftyone-teams-app.fullname" .) | trunc 63 | trimSuffix "-" }}
9+
{{- end }}
10+
{{- end }}
11+
12+
{{/*
13+
Create the name of the service pod template ConfigMap to use
14+
*/}}
15+
{{- define "service-orchestrator.pod-template-config-map-name" }}
16+
{{- if .Values.serviceOrchestrator.podTemplate.configMap.name }}
17+
{{- .Values.serviceOrchestrator.podTemplate.configMap.name | trunc 63 | trimSuffix "-" }}
18+
{{- else }}
19+
{{- printf "%s-service-pod-template" (include "fiftyone-teams-app.fullname" .) | trunc 63 | trimSuffix "-" }}
20+
{{- end }}
21+
{{- end }}
22+
23+
{{- define "service-orchestrator.builtin-services-config-map-labels" }}
24+
{{- include "fiftyone-teams-app.commonLabels" . }}
25+
app.kubernetes.io/name: {{ include "service-orchestrator.builtin-services-config-map-name" . }}
26+
app.kubernetes.io/instance: {{ .Release.Name }}
27+
app.voxel51.com/component: service-orchestrator
28+
{{- with .Values.serviceOrchestrator.builtinServices.configMap.labels }}
29+
{{ toYaml . }}
30+
{{- end }}
31+
{{- end }}
32+
33+
{{- define "service-orchestrator.pod-template-config-map-labels" }}
34+
{{- include "fiftyone-teams-app.commonLabels" . }}
35+
app.kubernetes.io/name: {{ include "service-orchestrator.pod-template-config-map-name" . }}
36+
app.kubernetes.io/instance: {{ .Release.Name }}
37+
app.voxel51.com/component: service-orchestrator
38+
{{- with .Values.serviceOrchestrator.podTemplate.configMap.labels }}
39+
{{ toYaml . }}
40+
{{- end }}
41+
{{- end }}

‎helm/fiftyone-teams-app/templates/api-deployment.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,16 @@ spec:
8989
- name: do-targets
9090
mountPath: /tmp/do-targets
9191
{{- end }}
92+
{{- if .Values.serviceOrchestrator.enabled }}
93+
{{- if or (.Values.serviceOrchestrator.builtinServices.configMap.create) (.Values.serviceOrchestrator.builtinServices.configMap.name) }}
94+
- name: builtin-services
95+
mountPath: /opt/builtin-services
96+
{{- end }}
97+
{{- if or (.Values.serviceOrchestrator.podTemplate.configMap.create) (.Values.serviceOrchestrator.podTemplate.configMap.name) }}
98+
- name: service-pod-template
99+
mountPath: /opt/service-pod-template
100+
{{- end }}
101+
{{- end }}
92102
{{- with .Values.apiSettings.volumeMounts }}
93103
{{- toYaml . | nindent 12 }}
94104
{{- end }}
@@ -130,6 +140,18 @@ spec:
130140
configMap:
131141
name: {{ include "delegated-operator-templates.config-map-name" . }}
132142
{{- end }}
143+
{{- if .Values.serviceOrchestrator.enabled }}
144+
{{- if or (.Values.serviceOrchestrator.builtinServices.configMap.create) (.Values.serviceOrchestrator.builtinServices.configMap.name) }}
145+
- name: builtin-services
146+
configMap:
147+
name: {{ include "service-orchestrator.builtin-services-config-map-name" . }}
148+
{{- end }}
149+
{{- if or (.Values.serviceOrchestrator.podTemplate.configMap.create) (.Values.serviceOrchestrator.podTemplate.configMap.name) }}
150+
- name: service-pod-template
151+
configMap:
152+
name: {{ include "service-orchestrator.pod-template-config-map-name" . }}
153+
{{- end }}
154+
{{- end }}
133155
{{- with .Values.apiSettings.volumes }}
134156
{{- toYaml . | nindent 8 }}
135157
{{- end }}

‎helm/fiftyone-teams-app/templates/seed-orchestrators-job.yaml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
{{- if .Values.seedOrchestrators.enabled -}}
2+
# Seeds the deployment's orchestrator registrations in Mongo, so they are
3+
# versioned in values instead of hand-created. Covers any orchestrator
4+
# environment (kubernetes delegated-operator jobs, kubernetes-service pods,
5+
# ...). The script lives in files/seed_orchestrators.py; the orchestrator
6+
# list is rendered from `seedOrchestrators.orchestrators` and passed as JSON.
7+
#
8+
# Uses helm hooks so it works under plain helm and under ArgoCD (which maps
9+
# post-install/post-upgrade to PostSync). Talks directly to Mongo with the
10+
# deployment's existing teams secrets, so no API key is required.
11+
{{- $seed := .Values.seedOrchestrators }}
12+
{{- $image := merge (deepCopy ($seed.image | default dict)) .Values.delegatedOperatorJobTemplates.template.image }}
13+
apiVersion: batch/v1
14+
kind: Job
15+
metadata:
16+
name: {{ printf "%s-seed-orchestrators" (include "fiftyone-teams-app.fullname" .) | trunc 63 | trimSuffix "-" }}
17+
namespace: {{ .Values.namespace.name }}
18+
annotations:
19+
helm.sh/hook: post-install,post-upgrade
20+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
21+
labels:
22+
{{- include "fiftyone-teams-app.commonLabels" . | nindent 4 }}
23+
app.voxel51.com/component: orchestrator-seeding
24+
spec:
25+
ttlSecondsAfterFinished: {{ $seed.ttlSecondsAfterFinished }}
26+
backoffLimit: {{ $seed.backoffLimit }}
27+
template:
28+
spec:
29+
restartPolicy: Never
30+
serviceAccountName: {{ include "fiftyone-teams-app.serviceAccountName" . }}
31+
{{- with .Values.imagePullSecrets }}
32+
imagePullSecrets:
33+
{{- toYaml . | nindent 8 }}
34+
{{- end }}
35+
{{- with $seed.podSecurityContext }}
36+
securityContext:
37+
{{- toYaml . | nindent 8 }}
38+
{{- end }}
39+
containers:
40+
- name: seed-orchestrators
41+
image: "{{ $image.repository }}:{{ $image.tag | default .Chart.AppVersion }}"
42+
imagePullPolicy: {{ $image.pullPolicy | default "Always" }}
43+
env:
44+
- name: FIFTYONE_DATABASE_URI
45+
valueFrom:
46+
secretKeyRef:
47+
name: {{ .Values.secret.name }}
48+
key: mongodbConnectionString
49+
- name: FIFTYONE_DATABASE_NAME
50+
valueFrom:
51+
secretKeyRef:
52+
name: {{ .Values.secret.name }}
53+
key: fiftyoneDatabaseName
54+
- name: ORCHESTRATORS
55+
value: |-
56+
{{- toJson ($seed.orchestrators | default list) | nindent 16 }}
57+
# Fills any orchestrator whose `config.image` is the empty-string
58+
# sentinel, so registrations can track the CI-managed image
59+
# instead of hardcoding a tag in values.
60+
- name: DEFAULT_WORKER_IMAGE
61+
value: "{{ $image.repository }}:{{ $image.tag | default .Chart.AppVersion }}"
62+
{{- with $seed.resources }}
63+
resources:
64+
{{- toYaml . | nindent 12 }}
65+
{{- end }}
66+
{{- with $seed.containerSecurityContext }}
67+
securityContext:
68+
{{- toYaml . | nindent 12 }}
69+
{{- end }}
70+
volumeMounts:
71+
- mountPath: /tmp
72+
name: tmpdir
73+
command: [python, -c]
74+
args:
75+
- |
76+
{{ .Files.Get "files/seed_orchestrators.py" | indent 14 }}
77+
volumes:
78+
- emptyDir:
79+
sizeLimit: 100Mi
80+
name: tmpdir
81+
{{- end }}

‎helm/fiftyone-teams-app/templates/service-orchestrator-builtin-services-configmap.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{{- if and .Values.serviceOrchestrator.enabled .Values.serviceOrchestrator.builtinServices.configMap.create -}}
2+
# Per-deployment overrides for the builtin services packaged in fiftyone
3+
# (fiftyone/operators/builtin_services.yaml). teams-api mounts this file and
4+
# reads it via FIFTYONE_BUILTIN_SERVICES_PATH; entries deep-merge by `id` onto
5+
# the packaged defaults at reconcile time. `builtin_version` must be bumped
6+
# for a changed entry to re-apply to an environment that already has the
7+
# builtin in its config store.
8+
apiVersion: v1
9+
kind: ConfigMap
10+
metadata:
11+
name: {{ include "service-orchestrator.builtin-services-config-map-name" . }}
12+
namespace: {{ .Values.namespace.name }}
13+
{{- with .Values.serviceOrchestrator.builtinServices.configMap.annotations }}
14+
annotations:
15+
{{- toYaml . | nindent 4 }}
16+
{{- end }}
17+
labels:
18+
{{- include "service-orchestrator.builtin-services-config-map-labels" . | nindent 4 }}
19+
data:
20+
builtin_services.yaml: |
21+
{{- toYaml (.Values.serviceOrchestrator.builtinServices.services | default list) | nindent 4 }}
22+
{{- end }}

0 commit comments

Comments
 (0)