Skip to content

Commit f435db5

Browse files
mo-getterkevin-dimichel
mo-getter
and
kevin-dimichel
authored
Apply suggestions from code review
Co-authored-by: Kevin DiMichel <56850465+kevin-dimichel@users.noreply.github.com> Signed-off-by: Alan Smith <209585+mo-getter@users.noreply.github.com>
1 parent a27178f commit f435db5

3 files changed

Lines changed: 137 additions & 145 deletions

File tree

‎docker/docs/configuring-gpu-workloads.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -52,18 +52,18 @@ We will configure the
5252
with GPUs.
5353

5454
For most deployments, use the shipped GPU overlay
55-
`compose.delegated-operators.gpu.yaml`, which already includes the GPU
55+
`compose.delegated-operators.gpu.yaml`, which includes the GPU
5656
worker and its paired telemetry sidecar. Activate it alongside the CPU
5757
worker profile (e.g. `COMPOSE_PROFILES=do-1,gpu`); see
5858
[Configuring Telemetry](./configuring-telemetry.md#scaling-teams-do-with-telemetry)
5959
for the full profile reference.
6060

61-
If you need a custom inline example instead, the snippet below shows
61+
The snippet below contains custom inline example of
6262
how to add a GPU-based delegated operator (`teams-do-with-gpu`)
63-
alongside the standard `teams-do` slot. Pair it with telemetry by
64-
adding a matching `teams-do-with-gpu-telemetry` sidecar — see
65-
`docker/<auth-dir>/compose.delegated-operators.gpu.yaml` for a
66-
worked example.
63+
alongside the standard `teams-do`. Pair it with telemetry by
64+
adding a matching `teams-do-with-gpu-telemetry` sidecar. See
65+
`docker/<auth-dir>/compose.delegated-operators.gpu.yaml` for an
66+
example.
6767

6868
```yaml
6969
services:

‎docker/docs/configuring-telemetry.md‎

Lines changed: 120 additions & 121 deletions
Original file line numberDiff line numberDiff line change
@@ -15,52 +15,71 @@
1515
# Configuring FiftyOne Enterprise Telemetry
1616

1717
Telemetry adds a lightweight per-service metrics collector (sidecar
18-
pattern) plus a Redis backend.
19-
The Settings → Metrics page in teams-app displays live CPU / memory /
18+
pattern) and a Redis backend.
19+
In the app, the Settings → Metrics page in displays live CPU / memory /
2020
thread / file-descriptor samples and tailed stdout logs for each
2121
observed service.
2222

2323
**Telemetry is enabled by default.**
24-
The base compose files bundle the `telemetry-redis` service and
25-
per-workload sidecars;
26-
no opt-in flags or overlay files are needed.
24+
The base compose files contain the `telemetry-redis` service and
25+
per-workload sidecars.
2726

2827
## Default deployment
2928

29+
Running
30+
3031
```shell
3132
docker compose -f compose.yaml up -d
3233
```
3334

3435
renders `fiftyone-app`, `teams-api`, `teams-app`, `teams-cas`,
3536
`telemetry-redis`, `fiftyone-app-telemetry`, and `teams-api-telemetry`.
3637

37-
Optional overlays carry their own bundled sidecar.
38-
`compose.yaml`, `compose.plugins.yaml`, and
38+
Optional overlays provide their own bundled sidecar.
39+
The `compose.yaml`, `compose.plugins.yaml`, and
3940
`compose.dedicated-plugins.yaml` are mutually exclusive base files —
4041
pick one, then layer the `compose.delegated-operators.yaml` overlay on
41-
top.
42-
43-
The delegated-operators overlay defines three worker slots —
44-
`teams-do` (always on) and `teams-do-2` / `teams-do-3` gated behind
45-
cumulative Compose profiles (`do-2`, `do-3`). The default (no
46-
profile) runs one observed worker; set `COMPOSE_PROFILES=do-N` to add
47-
slots up to N. `do-N` includes every lower slot, so `do-3` runs three
48-
workers. For example, to run the dedicated-plugins base with two
49-
delegated-operator workers:
50-
51-
```shell
52-
COMPOSE_PROFILES=do-2 docker compose \
53-
-f compose.dedicated-plugins.yaml \
54-
-f compose.delegated-operators.yaml \
55-
up -d
56-
```
57-
58-
(or set `COMPOSE_PROFILES=do-2` in your `.env`).
59-
This renders the dedicated-plugins base set (`fiftyone-app`,
60-
`teams-api`, `teams-app`, `teams-cas`, `teams-plugins`,
61-
`telemetry-redis`, `fiftyone-app-telemetry`, `teams-api-telemetry`,
62-
`teams-plugins-telemetry`) plus `teams-do` / `teams-do-2` and their
63-
paired sidecars from the overlay.
42+
`compose.dedicated-plugins.yaml` are mutually exclusive base files.
43+
Pick one and layer the `compose.delegated-operators.yaml` overlay on
44+
45+
The delegated-operators overlay defines three worker slots
46+
47+
1. `teams-do` (always on)
48+
1. `teams-do-2`
49+
1. Enabled using the `do-2` Compose profile
50+
1. `teams-do-3`
51+
1. Enabled using the `do-3` Compose profile
52+
53+
The default (no profile) runs one delegated operator worker.
54+
Set `COMPOSE_PROFILES=do-<N>` to add slots up to `<N>`.
55+
`do-<N>` includes previous slots.
56+
For example `do-3` runs three workers.
57+
58+
To run the dedicated-plugins base with two
59+
delegated-operator workers either
60+
61+
1. Set `COMPOSE_PROFILES=do-2` in your `.env`
62+
1. Set the environment variable while calling compose up
63+
64+
65+
This renders the services
66+
67+
- FiftyOne Enterprise
68+
- `fiftyone-app`,
69+
- `teams-api`
70+
- `teams-app`
71+
- `teams-cas`
72+
- `teams-plugins`,
73+
- Telemetry
74+
- `telemetry-redis`
75+
- `fiftyone-app-telemetry`
76+
- `teams-api-telemetry`,
77+
- `teams-plugins-telemetry`
78+
- `teams-do-telemetry`
79+
- `teams-do-2-telemetry`
80+
- Delegated Operator
81+
- `teams-do`
82+
- `teams-do-2`
6483

6584
For GPU-enabled delegated operators, layer
6685
`compose.delegated-operators.gpu.yaml` (which includes its own bundled
@@ -76,97 +95,79 @@ docker compose --profile gpu \
7695

7796
## What's bundled by default
7897

79-
- `telemetry-redis` — Redis 7 container that holds metric streams and
80-
log entries.
81-
Data is capped by the `allkeys-lru` maxmemory policy so disk usage
82-
stays bounded.
83-
- `fiftyone-app-telemetry`, `teams-api-telemetry` — sidecar containers,
84-
one per observed service.
85-
Each joins the target's PID namespace via `pid: "service:<target>"`
86-
so it can read `/proc/<pid>/fd/1` and use psutil to sample CPU,
87-
memory, FDs, and thread counts.
88-
Sidecars for delegated operations run with the `SYS_PTRACE` capability so
89-
py-spy can attach to the target.
90-
- `teams-plugins-telemetry` (only with `compose.dedicated-plugins.yaml`)
91-
— sidecar for the dedicated `teams-plugins` service.
92-
- `teams-do-telemetry` (only with `compose.delegated-operators.yaml`)
93-
— sidecar in `EXECUTOR_SIDECAR=true` mode that watches the executor
94-
for per-operation child processes and records per-op metrics back
95-
to the `delegated_ops` MongoDB document. Additional workers
96-
(`teams-do-2`, `teams-do-3`) opt-in via Compose profiles and each
97-
get their own paired sidecar — see [Scaling teams-do with
98-
telemetry](#scaling-teams-do-with-telemetry).
99-
- `teams-do-gpu` + `teams-do-gpu-telemetry` (only with
100-
`compose.delegated-operators.gpu.yaml` and `--profile gpu`) — a
101-
GPU-enabled delegated-operator worker registered as a distinct
102-
orchestrator (`-n teams-do-gpu`) plus its paired sidecar.
103-
The sidecar reads GPU metrics via NVML and requires its own GPU
104-
reservation.
105-
- `FIFTYONE_TELEMETRY_REDIS_URL` is injected on `fiftyone-app`,
106-
`teams-api`, `teams-app`, `teams-plugins`, and (when the DO overlay
107-
is used) `teams-do` so the in-app telemetry blueprint and SSE
108-
endpoints can read from Redis.
109-
110-
## Opt out
98+
- Services
99+
- `telemetry-redis`
100+
- Redis 7 container that holds metric streams and log entries.
101+
- Data is capped by the `allkeys-lru` maxmemory policy so disk usage
102+
stays bounded.
103+
- `teams-do-gpu`
104+
- only with `compose.delegated-operators.gpu.yaml` and `--profile gpu`
105+
- GPU-enabled delegated-operator worker registered as a distinct
106+
orchestrator (`-n teams-do-gpu`) plus its paired sidecar.
107+
- Service Side Cars
108+
- `fiftyone-app-telemetry` and `teams-api-telemetry`
109+
- Each joins the target's PID namespace via `pid: "service:<target>"`
110+
so it can read `/proc/<pid>/fd/1` and use psutil to sample CPU,
111+
memory, file descriptors, and thread counts
112+
- For delegated operations run with the `SYS_PTRACE` capability so
113+
py-spy can attach to the target
114+
- `teams-plugins-telemetry`
115+
- Only with `compose.dedicated-plugins.yaml`
116+
- For the dedicated `teams-plugins` service
117+
- `teams-do-telemetry`
118+
- Only with `compose.delegated-operators.yaml`
119+
- In `EXECUTOR_SIDECAR=true` mode that watches the executor
120+
for per-operation child processes and records per-operation metrics
121+
back to the `delegated_ops` MongoDB document
122+
- Additional workers (`teams-do-2`, `teams-do-3`)
123+
- Opt-in via Compose profiles and each
124+
get their own paired sidecar
125+
- See
126+
[Scaling teams-do with telemetry](#scaling-teams-do-with-telemetry)
127+
- `teams-do-gpu-telemetry`
128+
- Only with `compose.delegated-operators.gpu.yaml` and `--profile gpu`
129+
- The sidecar reads GPU metrics via NVML and requires its own GPU
130+
reservation.
131+
- Environment Variables
132+
- `FIFTYONE_TELEMETRY_REDIS_URL` environment variable is set on these services
133+
so the telemetry blueprint and server-sent events endpoints can read from Redis
134+
- `fiftyone-app`
135+
- `teams-api`
136+
- `teams-app`
137+
- `teams-plugins`
138+
- (when the DO overlay is used) `teams-do`
139+
140+
## Opting out
111141

112142
> [!IMPORTANT]
113143
> Disabling the telemetry sidecar leaves the FiftyOne UI's
114-
> delegated-operator log viewer empty — it depends on the sidecar to
115-
> capture per-operation logs.
116-
117-
To run without telemetry, add a `compose.override.yaml` that scales the
118-
telemetry services to zero replicas:
119-
120-
```yaml
121-
services:
122-
telemetry-redis:
123-
deploy:
124-
replicas: 0
125-
fiftyone-app-telemetry:
126-
deploy:
127-
replicas: 0
128-
teams-api-telemetry:
129-
deploy:
130-
replicas: 0
131-
# Only needed when running the corresponding overlay:
132-
teams-plugins-telemetry:
133-
deploy:
134-
replicas: 0
135-
teams-do-telemetry:
136-
deploy:
137-
replicas: 0
138-
# Additional delegated-operator slots only run with COMPOSE_PROFILES=do-2/do-3:
139-
teams-do-2-telemetry:
140-
deploy:
141-
replicas: 0
142-
teams-do-3-telemetry:
143-
deploy:
144-
replicas: 0
145-
```
144+
> delegated-operator log viewer empty.
145+
> The log viewer depends on the sidecar to capture per-operation logs.
146146
147-
`docker compose -f compose.yaml -f compose.override.yaml up -d` starts
148-
the base services without the telemetry collector.
149-
The main containers still have `FIFTYONE_TELEMETRY_REDIS_URL` set, but
150-
the in-app agent gracefully no-ops when Redis is unreachable.
147+
To run without telemetry
151148

149+
1. Add a `compose.override.yaml` that scales the
150+
telemetry services to zero replicas:
151+
152+
152153
### Scaling teams-do with telemetry
153154

154-
docker-compose's `pid: "service:<name>"` only joins a single replica's
155-
PID namespace, so a single `teams-do` service scaled to N replicas
156-
would leave N-1 of them invisible to the sidecar.
157-
`compose.delegated-operators.yaml` instead declares three worker
158-
slots, each as its own Compose service with its own paired sidecar and
155+
Docker Compose's `pid: "service:<name>"` only joins a single replica's PID namespace.
156+
Thus single `teams-do` service scaled to `<N>` replicas
157+
would leave `<N-1>` of them invisible to the sidecar.
158+
Instead, `compose.delegated-operators.yaml` contains three worker
159+
slots as its own Compose service, paired service sidecar, and
159160
its own executor-socket volume:
160161

161-
| Slot | Service | Sidecar | Activation |
162-
| ---- | ------------- | ----------------------- | ---------------------- |
163-
| 1 | `teams-do` | `teams-do-telemetry` | always on (no profile) |
164-
| 2 | `teams-do-2` | `teams-do-2-telemetry` | profile `do-2`, `do-3` |
165-
| 3 | `teams-do-3` | `teams-do-3-telemetry` | profile `do-3` |
162+
| Slot | Service | Sidecar | Activation |
163+
| ---- | ------------- | ----------------------- | ------------------------ |
164+
| 1 | `teams-do` | `teams-do-telemetry` | always on (no profile) |
165+
| 2 | `teams-do-2` | `teams-do-2-telemetry` | profile `do-2` or `do-3` |
166+
| 3 | `teams-do-3` | `teams-do-3-telemetry` | profile `do-3` |
166167

167-
The default (no profile) runs one observed worker. To add more,
168-
activate the matching Compose profile — `do-N` runs N workers because
169-
higher numbers include every lower slot:
168+
The default (no profile) runs one delegated operator worker. To add more,
169+
activate the matching Compose profile — `do-<N>` runs `<N>` workers because
170+
higher numbers include every prior slot:
170171

171172
```shell
172173
# 1 worker (default):
@@ -183,9 +184,9 @@ COMPOSE_PROFILES=do-3 docker compose -f compose.yaml \
183184
```
184185

185186
Set `COMPOSE_PROFILES` in your `.env` to persist the choice, or pass
186-
`--profile do-N` on the command line. Each worker registers under its
187+
`--profile do-<N>` on the command line. Each worker registers under its
187188
own orchestrator name (slot 2 as `teams-do-2`, slot 3 as `teams-do-3`)
188-
so they surface separately in Settings → Metrics.
189+
so they surface distinctly in Settings → Metrics.
189190

190191
> [!IMPORTANT]
191192
> This replaces the previous `FIFTYONE_DELEGATED_OPERATOR_WORKER_REPLICAS`
@@ -195,14 +196,12 @@ so they surface separately in Settings → Metrics.
195196
> on the previous default, set `COMPOSE_PROFILES=do-3` to restore the
196197
> three-worker behavior (each worker now has its own sidecar).
197198
198-
Additionally,
199-
200199
> [!NOTE]
201-
> The cap of 3 is intentional. The value must not exceed your
200+
> The cap of 3 is intentional as the value must not exceed your
202201
> license's max concurrent delegated operators.
203-
> For more than 3 workers, the slot-2 and slot-3 blocks in
204-
> `compose.delegated-operators.yaml` are copy-paste templates:
205-
> duplicate them as `teams-do-4` / `teams-do-4-telemetry` (and so on),
202+
> For more than 3 workers, use the slot-3 blocks in
203+
> `compose.delegated-operators.yaml` as templates.
204+
> Duplicate them as `teams-do-4` / `teams-do-4-telemetry`,
206205
> bumping the service name, `pid: "service:teams-do-N"`, `POD_NAME`,
207206
> `-n teams-do-N`, and `telemetry-socket-N` volume on each copy.
208207
@@ -242,8 +241,8 @@ All knobs live in your `.env` — see `env.template` for the full list:
242241
Telemetry containers ship with conservative CPU and memory limits sized so the
243242
sidecars do not starve the workloads they observe.
244243
The values are declared under each service's `deploy.resources` block
245-
in the compose files;
246-
compose v2 honors `cpus` and `memory` limits/reservations outside swarm
244+
in the Cpmpose files.
245+
Compose v2 honors `cpus` and `memory` limits and reservations outside swarm
247246
mode.
248247

249248
| Service | CPU limit | Memory limit | Notes |

0 commit comments

Comments
 (0)