Skip to content

Commit ce60acd

Browse files
CamronStaleyCamronStaley
committed
add docs for multimodal feature flags and projection DO configuration
1 parent 0c324e7 commit ce60acd

4 files changed

Lines changed: 367 additions & 0 deletions

File tree

‎docker/docs/configuring-multimodal.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
<!-- markdownlint-disable no-inline-html line-length -->
2+
<!-- markdownlint-disable-next-line first-line-heading -->
3+
<div align="center">
4+
<p align="center">
5+
6+
<img alt="Voxel51 Logo" src="https://user-images.githubusercontent.com/25985824/106288517-2422e000-6216-11eb-871d-26ad2e7b1e59.png" height="55px"> &nbsp;
7+
<img alt="Voxel51 FiftyOne" src="https://user-images.githubusercontent.com/25985824/106288518-24bb7680-6216-11eb-8f10-60052c519586.png" height="50px">
8+
9+
</p>
10+
</div>
11+
<!-- markdownlint-enable no-inline-html line-length -->
12+
13+
---
14+
15+
# Configuring Multimodal Datasets
16+
17+
<!-- toc -->
18+
19+
- [Overview](#overview)
20+
- [Enabling the `VFF_MULTIMODAL` Feature Flag](#enabling-the-vff_multimodal-feature-flag)
21+
- [Services That Require The Flag](#services-that-require-the-flag)
22+
- [Delegated Operator Scratch Space Requirements](#delegated-operator-scratch-space-requirements)
23+
- [Why Multimodal Needs Extra Scratch Space](#why-multimodal-needs-extra-scratch-space)
24+
- [Sizing Guidance](#sizing-guidance)
25+
- [Pinning Projection Processing With `FIFTYONE_PROJECTION_DELEGATION_TARGET`](#pinning-projection-processing-with-fiftyone_projection_delegation_target)
26+
- [Behavior When Set](#behavior-when-set)
27+
- [Naming Your `teams-do` Worker](#naming-your-teams-do-worker)
28+
- [Example Configuration](#example-configuration)
29+
30+
<!-- tocstop -->
31+
32+
## Overview
33+
34+
FiftyOne Enterprise's multimodal datasets store large, non-sample-centric
35+
modalities (e.g. sensor streams, point clouds, telemetry) as Parquet-backed
36+
Iceberg tables rather than as ordinary Mongo-backed samples.
37+
A background delegated-operator pipeline (`run_projections` followed by
38+
`compact_projections`) continuously ingests new data and periodically
39+
compacts it into larger, size-bounded files.
40+
41+
Running multimodal datasets requires:
42+
43+
1. The `VFF_MULTIMODAL` feature flag, set on every service that serves or
44+
processes multimodal data.
45+
1. Enough scratch disk space on `teams-do` (delegated operator) containers
46+
for projection compaction to succeed.
47+
1. Optionally, `FIFTYONE_PROJECTION_DELEGATION_TARGET` to pin projection
48+
processing to a specific `teams-do` worker instead of relying on
49+
automatic selection.
50+
51+
## Enabling the `VFF_MULTIMODAL` Feature Flag
52+
53+
`VFF_MULTIMODAL` is a presence-only feature flag: setting it to any value
54+
(e.g. `1`) enables the feature. Multimodal support is off by default.
55+
56+
### Services That Require The Flag
57+
58+
Add `VFF_MULTIMODAL` to the `environment:` block of each of the following
59+
services:
60+
61+
| Compose service | Why it's needed |
62+
| --- | --- |
63+
| `teams-api` | Runs the periodic background task that queues projection delegated operations for datasets with pending data. |
64+
| `fiftyone-app` | Serves the GraphQL/REST routes and grid queries that read multimodal Parquet data; without the flag, requests against a multimodal dataset are rejected. |
65+
| `teams-do` (and any additional worker slots/GPU workers you run) | Runs `run_projections`/`compact_projections`, which write multimodal dataset metadata and raise an error if the flag isn't enabled. |
66+
| `teams-app` (frontend) | Gates rendering of multimodal-specific UI components. |
67+
| `teams-plugins` | Serves operator schema/input resolution for the projection pipeline; recommended for consistency even though projection execution itself only ever runs on `teams-do` workers. |
68+
69+
> [!NOTE]
70+
> Set the flag on **every** service in the table above. A partial
71+
> configuration (e.g. only `fiftyone-app`) leaves the UI able to query
72+
> multimodal data while ingestion silently fails, or vice versa.
73+
74+
## Delegated Operator Scratch Space Requirements
75+
76+
### Why Multimodal Needs Extra Scratch Space
77+
78+
`compact_projections` reads all not-yet-compacted Parquet files for a
79+
projection table, merges/sorts them, and writes back a consolidated file, up
80+
to a configurable target size (1 GiB by default). For cloud warehouse
81+
locations (`gs://`, `s3://`, `az://`), both the download of source files and
82+
the write of the compacted output stage through the container's local
83+
`/tmp` before being uploaded.
84+
85+
### Sizing Guidance
86+
87+
Unlike a Kubernetes deployment, the provided Docker Compose files for
88+
`teams-do` don't set `read_only: true` and don't mount a separate,
89+
size-limited volume at `/tmp` — it's just part of the container's normal
90+
writable filesystem, which draws on whatever free disk space is actually
91+
available on the Docker host. There's no artificial per-directory cap to
92+
raise here; the practical requirement is simply:
93+
94+
- Ensure the host machine running your `teams-do` container(s) has enough
95+
free disk space to comfortably stage compaction's largest expected
96+
output file (1 GiB by default) plus the current backlog of
97+
not-yet-compacted source files for a projection table.
98+
- If you customize your `teams-do` service to run with `read_only: true`,
99+
you must add your own writable mount at `/tmp` (e.g. a bind mount or a
100+
named volume) with enough capacity for the above — a read-only root
101+
filesystem with no `/tmp` mount will make compaction fail outright, not
102+
just run low on space.
103+
104+
## Pinning Projection Processing With `FIFTYONE_PROJECTION_DELEGATION_TARGET`
105+
106+
By default, `teams-api` automatically selects the lowest-compute active
107+
orchestrator (excluding GPU orchestrators) to run each dataset's projection
108+
pipeline. Set `FIFTYONE_PROJECTION_DELEGATION_TARGET` in `teams-api`'s
109+
`environment:` block to pin all projection processing to one specific
110+
`teams-do` worker instead — for example, to ensure it always lands on the
111+
worker sized for compaction per the previous section.
112+
113+
### Behavior When Set
114+
115+
There is **no fallback to automatic selection**. If the configured value
116+
does not match an active orchestrator capable of running the projection
117+
operator, `teams-api` logs an error and skips queuing any projection
118+
delegated operations that cycle — pending datasets simply won't be
119+
processed until the value is corrected or removed.
120+
121+
### Naming Your `teams-do` Worker
122+
123+
The value must match the exact name a `teams-do` worker registers under,
124+
which comes from the `-n <name>` argument in its `command:`. The default
125+
`teams-do` slot in `compose.delegated-operators.yaml` does **not** pass
126+
`-n`, so it registers under an unpredictable default name — if you want to
127+
pin projection processing to a worker, add an explicit `-n <name>` to its
128+
`command:` (as the `teams-do-2`/`teams-do-3`/`teams-do-gpu` slots already
129+
do) rather than relying on the default slot.
130+
131+
### Example Configuration
132+
133+
```yaml
134+
services:
135+
teams-do-multimodal:
136+
image: voxel51/fiftyone-teams-cv-full:v2.22.0
137+
command: >
138+
/bin/sh -c "fiftyone delegated launch -t remote -m -n teams-do-multimodal"
139+
environment:
140+
VFF_MULTIMODAL: 1
141+
# ... plus the other teams-do environment variables
142+
143+
teams-api:
144+
environment:
145+
VFF_MULTIMODAL: 1
146+
FIFTYONE_PROJECTION_DELEGATION_TARGET: teams-do-multimodal
147+
```

‎docker/docs/upgrading.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
- [Upgrading From Previous Versions](#upgrading-from-previous-versions)
2020
- [A Note On Database Migrations](#a-note-on-database-migrations)
2121
- [From FiftyOne Enterprise Version 2.0.0 and Later](#from-fiftyone-enterprise-version-200-and-later)
22+
- [FiftyOne Enterprise v2.22+ Multimodal Datasets](#fiftyone-enterprise-v222-multimodal-datasets)
2223
- [FiftyOne Enterprise v2.19+ Telemetry Sidecars](#fiftyone-enterprise-v219-telemetry-sidecars)
2324
- [Host Requirements](#host-requirements)
2425
- [Opting out of Telemetry](#opting-out-of-telemetry)
@@ -102,6 +103,26 @@ quickstart 0.21.2
102103
fiftyone migrate --info
103104
```
104105

106+
#### FiftyOne Enterprise v2.22+ Multimodal Datasets
107+
108+
FiftyOne Enterprise v2.22.0 introduces multimodal dataset support: large,
109+
non-sample-centric modalities stored as Parquet-backed Iceberg tables and
110+
ingested/compacted via a background delegated-operator pipeline.
111+
112+
Multimodal datasets require:
113+
114+
- The `VFF_MULTIMODAL` feature flag, set on `teams-api`, `fiftyone-app`,
115+
`teams-do` workers, `teams-app`, and `teams-plugins`.
116+
- Sufficient host disk space for `teams-do` containers to stage projection
117+
compaction files in `/tmp`.
118+
- Optionally, `FIFTYONE_PROJECTION_DELEGATION_TARGET` on `teams-api` to pin
119+
projection processing to a specific `teams-do` worker.
120+
121+
See the
122+
[Configuring Multimodal Datasets](./configuring-multimodal.md)
123+
documentation for full details, required services, and example
124+
configuration.
125+
105126
#### FiftyOne Enterprise v2.19+ Telemetry Sidecars
106127

107128
FiftyOne Enterprise v2.19.0 adds observability features viewable by

‎helm/docs/configuring-multimodal.md‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
<!-- markdownlint-disable no-inline-html line-length -->
2+
<!-- markdownlint-disable-next-line first-line-heading -->
3+
<div align="center">
4+
<p align="center">
5+
6+
<img alt="Voxel51 Logo" src="https://user-images.githubusercontent.com/25985824/106288517-2422e000-6216-11eb-871d-26ad2e7b1e59.png" height="55px"> &nbsp;
7+
<img alt="Voxel51 FiftyOne" src="https://user-images.githubusercontent.com/25985824/106288518-24bb7680-6216-11eb-8f10-60052c519586.png" height="50px">
8+
9+
</p>
10+
</div>
11+
<!-- markdownlint-enable no-inline-html line-length -->
12+
13+
---
14+
15+
# Configuring Multimodal Datasets
16+
17+
<!-- toc -->
18+
19+
- [Overview](#overview)
20+
- [Enabling the `VFF_MULTIMODAL` Feature Flag](#enabling-the-vff_multimodal-feature-flag)
21+
- [Workloads That Require The Flag](#workloads-that-require-the-flag)
22+
- [Delegated Operator Storage Requirements](#delegated-operator-storage-requirements)
23+
- [Why Multimodal Needs Extra Scratch Space](#why-multimodal-needs-extra-scratch-space)
24+
- [Minimum Recommended Sizing](#minimum-recommended-sizing)
25+
- [Example Configuration](#example-configuration)
26+
- [Pinning Projection Processing With `FIFTYONE_PROJECTION_DELEGATION_TARGET`](#pinning-projection-processing-with-fiftyone_projection_delegation_target)
27+
- [Behavior When Set](#behavior-when-set)
28+
- [Example Configuration](#example-configuration-1)
29+
30+
<!-- tocstop -->
31+
32+
## Overview
33+
34+
FiftyOne Enterprise's multimodal datasets store large, non-sample-centric
35+
modalities (e.g. sensor streams, point clouds, telemetry) as Parquet-backed
36+
Iceberg tables rather than as ordinary Mongo-backed samples.
37+
A background delegated-operator pipeline (`run_projections` followed by
38+
`compact_projections`) continuously ingests new data and periodically
39+
compacts it into larger, size-bounded files.
40+
41+
Running multimodal datasets requires three pieces of configuration beyond a
42+
standard install:
43+
44+
1. The `VFF_MULTIMODAL` feature flag, set on every workload that serves or
45+
processes multimodal data.
46+
1. Enough ephemeral/scratch storage on delegated-operator workloads for
47+
projection compaction to succeed.
48+
1. Optionally, `FIFTYONE_PROJECTION_DELEGATION_TARGET` to pin projection
49+
processing to a specific orchestrator instead of relying on automatic
50+
selection.
51+
52+
## Enabling the `VFF_MULTIMODAL` Feature Flag
53+
54+
`VFF_MULTIMODAL` is a presence-only feature flag: setting it to any value
55+
(e.g. `1`) enables the feature.
56+
Multimodal support is off by default.
57+
58+
### Workloads That Require The Flag
59+
60+
| Workload | Helm values path | Why it's needed |
61+
| --- | --- | --- |
62+
| `teams-api` | `apiSettings.env` | Runs the periodic background task that queues projection delegated operations for datasets with pending data. |
63+
| `fiftyone-app` | `appSettings.env` | Serves the GraphQL/REST routes and grid queries that read multimodal Parquet data; without the flag, requests against a multimodal dataset are rejected. |
64+
| Delegated operator workloads | `delegatedOperatorDeployments.template.env` and/or `delegatedOperatorJobTemplates.template.env` | Runs `run_projections`/`compact_projections`, which write multimodal dataset metadata and raise an error if the flag isn't enabled. |
65+
| `teams-app` (frontend) | `teamsAppSettings.env` | Gates rendering of multimodal-specific UI components. |
66+
| `teams-plugins` | `pluginsSettings.env` | Serves operator schema/input resolution for the projection pipeline; recommended for consistency even though projection execution itself only ever runs on delegated-operator workloads. |
67+
68+
> [!NOTE]
69+
> Set the flag on **every** workload in the table above. A partial
70+
> configuration (e.g. only `fiftyone-app`) leaves the UI able to query
71+
> multimodal data while ingestion silently fails, or vice versa.
72+
73+
## Delegated Operator Storage Requirements
74+
75+
### Why Multimodal Needs Extra Scratch Space
76+
77+
`compact_projections` reads all not-yet-compacted Parquet files for a
78+
projection table, merges/sorts them, and writes back a consolidated file, up
79+
to a configurable target size (1 GiB by default). For cloud warehouse
80+
locations (`gs://`, `s3://`, `az://`), both the download of source files and
81+
the write of the compacted output stage through the pod's local `/tmp`
82+
before being uploaded.
83+
84+
If your delegated-operator workloads run with `securityContext.readOnlyRootFilesystem: true`
85+
(recommended for Pod Security Admission `restricted` compliance), `/tmp`
86+
is not writable at all unless you explicitly mount a writable volume there.
87+
An `emptyDir` volume mounted at `/tmp` without its own `sizeLimit` draws
88+
from the pod's overall `ephemeral-storage` resource budget; an `emptyDir`
89+
with an explicit `sizeLimit` is capped separately, on top of that shared
90+
budget. A `sizeLimit` that's too small for the amount of unconsolidated
91+
data will cause compaction to be evicted or fail before it can complete,
92+
even if the pod's aggregate `ephemeral-storage` limit is otherwise generous.
93+
94+
### Minimum Recommended Sizing
95+
96+
For any `delegatedOperatorDeployments`/`delegatedOperatorJobTemplates`
97+
instance that will run the projection pipeline:
98+
99+
- `resources.limits`/`requests.ephemeral-storage`: at least `1.5Gi`
100+
- A `tmpdir` `emptyDir` mounted at `/tmp` with `sizeLimit: 1Gi`
101+
102+
These are minimums for typical projection volumes, not a hard guarantee —
103+
size up further if your projections accumulate a large backlog of
104+
unconsolidated data between compaction runs (for example, if compaction has
105+
been disabled or failing for a period of time).
106+
107+
### Example Configuration
108+
109+
```yaml
110+
delegatedOperatorDeployments:
111+
template:
112+
env:
113+
VFF_MULTIMODAL: 1
114+
resources:
115+
limits:
116+
ephemeral-storage: 1.5Gi
117+
requests:
118+
ephemeral-storage: 1.5Gi
119+
securityContext:
120+
readOnlyRootFilesystem: true
121+
volumeMounts:
122+
- name: tmpdir
123+
mountPath: /tmp
124+
volumes:
125+
- name: tmpdir
126+
emptyDir:
127+
sizeLimit: 1Gi
128+
```
129+
130+
Apply the equivalent `env`, `resources`, `volumeMounts`, and `volumes`
131+
settings under `delegatedOperatorJobTemplates.template` if you also use
132+
on-demand (Kubernetes `Job`-based) delegated operators for projection
133+
processing.
134+
135+
> [!NOTE]
136+
> If your delegated-operator workloads do **not** set
137+
> `readOnlyRootFilesystem: true`, `/tmp` is already part of the container's
138+
> writable filesystem and draws directly from the pod's `ephemeral-storage`
139+
> limit with no separate cap — you only need the explicit `tmpdir` volume
140+
> and `sizeLimit` when the root filesystem is read-only.
141+
142+
## Pinning Projection Processing With `FIFTYONE_PROJECTION_DELEGATION_TARGET`
143+
144+
By default, `teams-api` automatically selects the lowest-compute active
145+
orchestrator (excluding GPU orchestrators) to run each dataset's projection
146+
pipeline. Set `FIFTYONE_PROJECTION_DELEGATION_TARGET` on `apiSettings.env`
147+
to pin all projection processing to one specific orchestrator instead —
148+
for example, to ensure it always lands on the orchestrator instance sized
149+
for compaction per the previous section.
150+
151+
The value must be the exact registered name of an always-on delegated
152+
operator instance — i.e. the kebab-cased key under
153+
`delegatedOperatorDeployments.deployments.<key>` (the same name each
154+
instance registers with via `fiftyone delegated launch -n <name>`).
155+
On-demand (`delegatedOperatorJobTemplates`) instances cannot be used as a
156+
delegation target, since they don't run as a persistent, named orchestrator.
157+
158+
### Behavior When Set
159+
160+
There is **no fallback to automatic selection**. If the configured value
161+
does not match an active orchestrator capable of running the projection
162+
operator, `teams-api` logs an error and skips queuing any projection
163+
delegated operations that cycle — pending datasets simply won't be
164+
processed until the value is corrected or removed.
165+
166+
### Example Configuration
167+
168+
```yaml
169+
delegatedOperatorDeployments:
170+
deployments:
171+
teamsDoMultimodal:
172+
# ... sized per "Delegated Operator Storage Requirements" above
173+
174+
apiSettings:
175+
env:
176+
FIFTYONE_PROJECTION_DELEGATION_TARGET: teams-do-multimodal
177+
```

‎helm/docs/upgrading.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
- [Upgrading From Previous Versions](#upgrading-from-previous-versions)
2020
- [A Note On Database Migrations](#a-note-on-database-migrations)
2121
- [From FiftyOne Enterprise Version 2.0.0 or Higher](#from-fiftyone-enterprise-version-200-or-higher)
22+
- [FiftyOne Enterprise v2.22+ Multimodal Datasets](#fiftyone-enterprise-v222-multimodal-datasets)
2223
- [FiftyOne Enterprise v2.19+ Telemetry Sidecars](#fiftyone-enterprise-v219-telemetry-sidecars)
2324
- [Cluster Requirements](#cluster-requirements)
2425
- [Opting out of Telemetry](#opting-out-of-telemetry)
@@ -144,6 +145,27 @@ quickstart 0.21.2
144145
fiftyone migrate --info
145146
```
146147
148+
#### FiftyOne Enterprise v2.22+ Multimodal Datasets
149+
150+
FiftyOne Enterprise v2.22.0 introduces multimodal dataset support: large,
151+
non-sample-centric modalities stored as Parquet-backed Iceberg tables and
152+
ingested/compacted via a background delegated-operator pipeline.
153+
154+
Multimodal datasets require:
155+
156+
- The `VFF_MULTIMODAL` feature flag, set on `teams-api`, `fiftyone-app`,
157+
delegated-operator workloads, `teams-app`, and `teams-plugins`.
158+
- Sufficient `ephemeral-storage` and `/tmp` scratch space on
159+
delegated-operator workloads for projection compaction to complete
160+
successfully.
161+
- Optionally, `FIFTYONE_PROJECTION_DELEGATION_TARGET` on `teams-api` to pin
162+
projection processing to a specific orchestrator.
163+
164+
See the
165+
[Configuring Multimodal Datasets](./configuring-multimodal.md)
166+
documentation for full details, required workloads, and example
167+
configuration.
168+
147169
#### FiftyOne Enterprise v2.19+ Telemetry Sidecars
148170
149171
FiftyOne Enterprise v2.19.0 adds observability features viewable by

0 commit comments

Comments
 (0)