feat(telemetry): integrate the new telemetry sidecar (#566)

Enables the new observability features released in FiftyOne Enterprise 2.19.0, including enhanced process metrics, real-time logs, and more, viewable directly within the UI. 

Changes:

* Helm
  * Adds a sidecar container to the following pods:
    * fiftyone-app
    * teams-api
    * plugins
    * do deployments
    * do jobs (as a native sidecar)
  * Adds a role and rolebinding for telemetry
  * Adds get pods/log permission to api role
  * Adds an optional redis deployment and service, with a further optional chart-managed pvc, or a further optional existing PVC by name
* Compose
  * Drops support for `FIFTYONE_DELEGATED_OPERATOR_WORKER_REPLICAS` config option, replacing it with `COMPOSE_PROFILES="do-3"`
  * Adds sidecar containers to same services as Helm
  * Adds a tmpfs volume for the telemetry socket
* Scripts: Adds support for the new telemetry-sidecar image

---------

Signed-off-by: Alan Smith <209585+mo-getter@users.noreply.github.com>
Signed-off-by: Kacey <kacey@voxel51.com>
Co-authored-by: kacey <kacey@voxel51.com>
Co-authored-by: Kevin DiMichel <56850465+kevin-dimichel@users.noreply.github.com>

Author: 3 people

docker/README.md

@@ -75,6 +75,7 @@ regarding FiftyOne Enterprise.
 - [Advanced Configuration](#advanced-configuration)
   - [Backup And Recovery](#backup-and-recovery)
   - [Secrets And Sensitive Data](#secrets-and-sensitive-data)
+  - [Telemetry](#telemetry)
   - [Snapshot Archival](#snapshot-archival)
   - [Static Banner Configuration](#static-banner-configuration)
   - [Storage Credentials and `FIFTYONE_ENCRYPTION_KEY`](#storage-credentials-and-fiftyone_encryption_key)
@@ -656,6 +657,17 @@ and
 [adding secrets](https://docs.voxel51.com/enterprise/secrets.html#adding-secrets)
 for questions regarding storage and encryption.
 
+### Telemetry
+
+FiftyOne Enterprise bundles a telemetry sidecar and Redis backend by
+default in every compose file.
+The Settings → Metrics page in teams-app exposes live per-service
+metrics (CPU, memory, FDs, thread counts) and tailed logs.
+
+Please refer to the
+[telemetry configuration documentation](./docs/configuring-telemetry.md)
+for full details.
+
 ### Snapshot Archival
 
 Since version v1.5, FiftyOne Enterprise supports

docker/common-services.yaml

@@ -17,6 +17,7 @@ services:
       FIFTYONE_MEDIA_CACHE_APP_IMAGES: false
       FIFTYONE_MEDIA_CACHE_SIZE_BYTES: -1
       FIFTYONE_SIGNED_URL_EXPIRATION: ${FIFTYONE_SIGNED_URL_EXPIRATION:-24}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
       # If you are routing through a proxy server you will want to set
       #  HTTP_PROXY_URL, HTTPS_PROXY_URL, and NO_PROXY_LIST in your .env
       #  then add the following environment variables to your
@@ -43,6 +44,7 @@ services:
       FIFTYONE_ENV: ${FIFTYONE_ENV}
       FIFTYONE_INTERNAL_SERVICE: true
       FIFTYONE_LOGGING_FORMAT: ${FIFTYONE_LOGGING_FORMAT:-text}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
       GRAPHQL_DEFAULT_LIMIT: ${GRAPHQL_DEFAULT_LIMIT}
       LOGGING_LEVEL: ${API_LOGGING_LEVEL:-INFO}
       MONGO_DEFAULT_DB: ${FIFTYONE_DATABASE_NAME}
@@ -72,6 +74,7 @@ services:
       FIFTYONE_SERVER_ADDRESS: ""
       FIFTYONE_SERVER_PATH_PREFIX: /api/proxy/fiftyone-teams
       FIFTYONE_TEAMS_PROXY_URL: ${FIFTYONE_TEAMS_PROXY_URL}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
       NODE_ENV: production
       RECOIL_DUPLICATE_ATOM_KEY_CHECKING_ENABLED: false
       FIFTYONE_APP_ANONYMOUS_ANALYTICS_ENABLED: ${FIFTYONE_APP_ANONYMOUS_ANALYTICS_ENABLED:-true}
@@ -144,6 +147,7 @@ services:
       FIFTYONE_MEDIA_CACHE_APP_IMAGES: false
       FIFTYONE_MEDIA_CACHE_SIZE_BYTES: -1
       FIFTYONE_PLUGINS_DIR: /opt/plugins
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
       # If you are routing through a proxy server you will want to set
       #  HTTP_PROXY_URL, HTTPS_PROXY_URL, and NO_PROXY_LIST in your .env
       #  then add the following environment variables to your
@@ -158,8 +162,12 @@ services:
 
   teams-do-common:
     image: voxel51/fiftyone-teams-cv-full:v2.19.0
+    # Telemetry default-on requires teams-do to share its PID namespace
+    # with a single sidecar (compose's `pid: "service:<name>"` only joins
+    # one replica). Force replicas=1 to keep the 1:1 pairing deterministic.
+    # See docker/docs/configuring-telemetry.md for multi-worker patterns.
     deploy:
-      replicas: ${FIFTYONE_DELEGATED_OPERATOR_WORKER_REPLICAS:-3}
+      replicas: 1
     command: >
       /bin/sh -c "fiftyone delegated launch -t remote -m"
     environment:
@@ -171,6 +179,144 @@ services:
       FIFTYONE_INTERNAL_SERVICE: true
       FIFTYONE_MEDIA_CACHE_SIZE_BYTES: -1
       FIFTYONE_PLUGINS_DIR: /opt/plugins
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
+      TELEMETRY_SOCKET: /tmp/telemetry/agent.sock
     restart: always
     volumes:
       - plugins-vol:/opt/plugins:ro
+      - telemetry-socket:/tmp/telemetry
+
+  telemetry-redis-common:
+    image: ${TELEMETRY_REDIS_IMAGE:-redis:7-alpine}
+    command:
+      - redis-server
+      - --save
+      - "60"
+      - "1"
+      - --dir
+      - /data
+      - --maxmemory
+      - ${TELEMETRY_REDIS_MAXMEMORY:-400mb}
+      - --maxmemory-policy
+      - allkeys-lru
+    restart: always
+    deploy:
+      resources:
+        limits:
+          cpus: "0.25"
+          memory: 512M
+        reservations:
+          cpus: "0.10"
+          memory: 256M
+    volumes:
+      - telemetry-redis-data:/data
+
+  # One sidecar per observed service. `pid: "service:<target>"` joins the
+  # target's PID namespace so the sidecar can read /proc/<pid>/fd/1 and
+  # psutil can see the target's process.
+  fiftyone-app-telemetry-common:
+    image: voxel51/telemetry-sidecar:v2.19.0
+    pid: "service:fiftyone-app"
+    # Matches the workload image's USER 1000:1000 so the kernel's
+    # ptrace_may_access() check on /proc/<workload-pid>/fd/1 reads passes via the
+    # same-UID fast path to allow for log tailing
+    user: "1000:1000"
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      POD_NAME: fiftyone-app
+      POD_NAMESPACE: ${TELEMETRY_NAMESPACE:-docker}
+      SERVICE_TYPE: fiftyone-app
+      TARGET_NAME: ${FIFTYONE_APP_TARGET_NAME:-hypercorn}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
+    deploy:
+      resources:
+        limits:
+          cpus: "0.10"
+          memory: 512M
+        reservations:
+          cpus: "0.10"
+          memory: 512M
+    restart: always
+
+  teams-api-telemetry-common:
+    image: voxel51/telemetry-sidecar:v2.19.0
+    pid: "service:teams-api"
+    user: "1000:1000"
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      POD_NAME: teams-api
+      POD_NAMESPACE: ${TELEMETRY_NAMESPACE:-docker}
+      SERVICE_TYPE: teams-api
+      TARGET_NAME: ${TEAMS_API_TARGET_NAME:-fiftyone-teams-api}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
+    deploy:
+      resources:
+        limits:
+          cpus: "0.10"
+          memory: 512M
+        reservations:
+          cpus: "0.10"
+          memory: 512M
+    restart: always
+
+  teams-plugins-telemetry-common:
+    image: voxel51/telemetry-sidecar:v2.19.0
+    pid: "service:teams-plugins"
+    user: "1000:1000"
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      POD_NAME: teams-plugins
+      POD_NAMESPACE: ${TELEMETRY_NAMESPACE:-docker}
+      SERVICE_TYPE: teams-plugins
+      TARGET_NAME: ${TEAMS_PLUGINS_TARGET_NAME:-hypercorn}
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
+    deploy:
+      resources:
+        limits:
+          cpus: "0.10"
+          memory: 512M
+        reservations:
+          cpus: "0.10"
+          memory: 512M
+    restart: always
+
+  teams-do-telemetry-common:
+    image: voxel51/telemetry-sidecar:v2.19.0
+    pid: "service:teams-do"
+    user: "1000:1000"
+    cap_drop:
+      - ALL
+    cap_add:
+      - SYS_PTRACE  # allows for additional profiling metrics (sampled stacks via py-spy)
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      POD_NAME: teams-do
+      POD_NAMESPACE: ${TELEMETRY_NAMESPACE:-docker}
+      SERVICE_TYPE: delegated-operator
+      TARGET_NAME: ${TEAMS_DO_TARGET_NAME:-fiftyone delegated}
+      EXECUTOR_SIDECAR: "true"
+      TELEMETRY_SOCKET: /tmp/telemetry/agent.sock
+      FIFTYONE_TELEMETRY_REDIS_URL: ${FIFTYONE_TELEMETRY_REDIS_URL:-redis://telemetry-redis:6379}
+      FIFTYONE_DATABASE_URI: ${FIFTYONE_DATABASE_URI}
+      FIFTYONE_DATABASE_NAME: ${FIFTYONE_DATABASE_NAME}
+    deploy:
+      resources:
+        limits:
+          cpus: "0.10"
+          memory: 512M
+        reservations:
+          cpus: "0.10"
+          memory: 512M
+    volumes:
+      - telemetry-socket:/tmp/telemetry
+    restart: always

docker/docs/configuring-delegated-operators.md

@@ -32,6 +32,19 @@ in conjunction with one of the three plugin configurations.
       up -d
     ```
 
+> [!IMPORTANT]
+> The `compose.delegated-operators.yaml` overlay defines three worker
+> slots — `teams-do`, `teams-do-2`, `teams-do-3` — each paired with a
+> telemetry sidecar. Slot 1 is always on (the default renders one
+> observed worker); slots 2 and 3 are gated behind cumulative Compose
+> profiles (`do-2`, `do-3`). Set `COMPOSE_PROFILES=do-N` to add slots
+> up to N (cap of 3). This replaces the deprecated
+> `FIFTYONE_DELEGATED_OPERATOR_WORKER_REPLICAS` environment variable.
+> If you used on the <2.19 default of 3 workers, set `COMPOSE_PROFILES=do-3`.
+> See
+> [Scaling teams-do with telemetry](./configuring-telemetry.md#scaling-teams-do-with-telemetry)
+> for the slot/profile reference and the rationale.
+
 Optionally, delegated operation run logs may be uploaded to a
 network-mounted file system or cloud storage path.
 Logs are uploaded in the format

docker/docs/configuring-gpu-workloads.md

@@ -51,11 +51,19 @@ We will configure the
 [delegated operators](./configuring-delegated-operators.md)
 with GPUs.
 
-In your `compose.delegated-operators.yaml`, under `.services.teams-do`,
-add the `deploy.resource.reservation.devices` configuration.
-
-The below will deploy a CPU-based delegated operator (`teams-do`) as well
-as a GPU-based delegated operator (`teams-do-with-gpu`):
+For most deployments, use the shipped GPU overlay
+`compose.delegated-operators.gpu.yaml`, which includes the GPU
+worker and its paired telemetry sidecar. Activate it alongside the CPU
+worker profile (e.g. `COMPOSE_PROFILES=do-1,gpu`); see
+[Configuring Telemetry](./configuring-telemetry.md#scaling-teams-do-with-telemetry)
+for the full profile reference.
+
+The snippet below contains custom inline example of
+how to add a GPU-based delegated operator (`teams-do-with-gpu`)
+alongside the standard `teams-do`. Pair it with telemetry by
+adding a matching `teams-do-with-gpu-telemetry` sidecar. See
+`docker/<auth-dir>/compose.delegated-operators.gpu.yaml` for an
+example.
 
 ```yaml
 services:
@@ -81,7 +89,7 @@ services:
     volumes:
       - plugins-vol:/opt/plugins:ro
     deploy:
-      replicas: ${FIFTYONE_DELEGATED_OPERATOR_WORKER_REPLICAS:-3}
+      replicas: 1
       resources:
         reservations:
           devices: