docs: refine workflow-label documentation (#1308)

refine workflow-label documentation

Author: jiaenren

docs/deployment_guide/advanced_config/index.rst

@@ -70,6 +70,12 @@ This section provides additional advanced configurations to customize and optimi
 
           Grant shared cloud bucket access to every workflow in a pool, so team members don't supply credentials per workflow.
 
+      .. grid-item-card:: :octicon:`tag` Workflow Labels
+          :link: ./workflow_labels
+          :link-type: doc
+
+          Require and constrain workflow label keys, and namespace the label keys stamped onto task pods.
+
       .. grid-item-card:: :octicon:`plug` MCP
           :link: ./mcp
           :link-type: doc

docs/deployment_guide/advanced_config/workflow_labels.rst

@@ -0,0 +1,180 @@
+..
+  SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+  SPDX-License-Identifier: Apache-2.0
+
+
+.. _workflow_labels_config:
+
+=======================================================
+Workflow Labels
+=======================================================
+
+Users attach labels to a workflow to record who owns it and what it is for. Label syntax is checked on every submission even when nothing is configured here; a policy adds requirements on top, so you can decide which keys must be present and which values are accepted.
+
+See :ref:`workflow_spec_labels` for how users set labels, and :ref:`workflow_submission` for the CLI flags and list filters. Field definitions are in :ref:`workflow_config`.
+
+
+Why Configure a Policy?
+=======================
+
+✓ **Attribute usage**
+  Require a key such as ``team`` or ``cost-center`` so every workflow can be traced back to an owner.
+
+✓ **Keep values consistent**
+  An allow-list rejects typos and near-duplicates, which keeps reports and dashboards aligned.
+
+✓ **Roll out gradually**
+  Warn before you enforce, so users see what to change while their submissions still succeed.
+
+
+Enforcement Modes
+=================
+
+Each entry in ``policy`` controls one key independently.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 15 85
+
+   * - **Mode**
+     - **Behavior when the key is missing, or its value is outside a non-empty allow-list**
+   * - ``"off"``
+     - Accepted, with no warning. Same as omitting the key from ``policy``.
+   * - ``warn``
+     - Accepted, and the submit response carries a warning.
+   * - ``enforce``
+     - Rejected. No workflow row and no stored specification are created.
+
+.. code-block:: yaml
+
+   labels_config:
+     policy:
+     - key: team
+       allow_list:
+       - robotics
+       - simulation
+       enforcement: warn
+     - key: cost-center
+       allow_list: []
+       enforcement: enforce
+
+An empty ``allow_list`` accepts any well-formed value, so the key becomes required without constraining what it holds.
+
+.. note::
+
+   Quote ``"off"``. Unquoted YAML ``off`` parses as boolean false.
+
+The same policy applies to new submissions, resubmission by ID, restart, and validation-only requests. Warnings are recomputed from the stored labels and the current policy, so a workflow always shows the warnings its policy would produce today, including after it completes.
+
+
+Rolling Out a Requirement
+=========================
+
+.. grid:: 3
+    :gutter: 2
+
+    .. grid-item-card::
+        :class-header: sd-bg-info sd-text-white
+
+        **1. Announce** 📣
+        ^^^
+
+        Add the key with ``enforcement: warn``
+
+        +++
+
+        Submissions succeed; users see what to add
+
+    .. grid-item-card::
+        :class-header: sd-bg-warning sd-text-white
+
+        **2. Watch** 📊
+        ^^^
+
+        Check the :ref:`validation metrics <workflow_labels_metrics>`
+
+        +++
+
+        See who is still missing the key before it starts rejecting
+
+    .. grid-item-card::
+        :class-header: sd-bg-success sd-text-white
+
+        **3. Enforce** ✅
+        ^^^
+
+        Switch to ``enforcement: enforce``
+
+        +++
+
+        Remaining violations are rejected at submission
+
+Changing a policy leaves existing and in-flight workflows untouched. To roll back, set ``enforcement: warn``; to disable the key entirely, set ``enforcement: "off"`` or remove the entry. In ConfigMap mode, an invalid edit is rejected and the previous valid snapshot stays active.
+
+.. warning::
+
+   Removing a value from an allow-list while the key is in ``enforce`` mode rejects every later submission that uses it, including restarts of workflows that were accepted earlier. Soak the change in ``warn`` first.
+
+
+Prefixing Pod Labels
+====================
+
+``pod_label_prefix`` is prepended to every workflow label key when labels are stamped onto task pods, and nowhere else:
+
+.. code-block:: yaml
+
+   labels_config:
+     pod_label_prefix: example.com/
+     policy:
+     - key: team
+       enforcement: warn
+
+A workflow submitted with ``team: robotics`` then carries ``example.com/team=robotics`` on its pods. Everywhere else keeps the bare ``team`` key: the stored specification, the workflow API, list filters, the CLI, and the metric attributes below. Users never type or query the prefix.
+
+The prefix is an opaque string, not an assumed DNS prefix: it is joined to the label key, and the merged key is validated as a Kubernetes label key at submission, in the same check as the policy. An invalid merged key is rejected with an error reporting the original key, the prefix, and the result.
+
+.. tip::
+
+   Set a prefix when task pods share a cluster with unrelated workloads. Pod labels are exported by key name, so a short key such as ``team`` can match an identically named label on another pod.
+
+
+.. _workflow_labels_metrics:
+
+Metrics
+=======
+
+Only configured policy keys become workflow-label dimensions on ``osmo_tasks_count``. Attribute names start with ``workflow_label_``. Letters and numbers are unchanged; ``_``, ``-``, ``.``, and ``/`` are encoded as ``__``, ``_dash_``, ``_dot_``, and ``_slash_`` respectively. For example, ``project`` is exported as ``workflow_label_project``.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - **Exported value**
+     - **When**
+   * - The value itself
+     - The value is in the configured allow-list.
+   * - ``<other>``
+     - The key is present with a value outside that list. An empty allow-list exports every value this way.
+   * - ``<missing>``
+     - The key is absent.
+
+This bounds the series count to the allow-list plus two sentinels per key, so keep the policy list small to control cardinality.
+
+Admission also emits ``osmo_label_validation_total{key, outcome}``, where ``outcome`` is ``ok``, ``missing``, ``invalid``, or ``rejected``. The counter covers rejected submissions that never create a workflow row.
+
+.. seealso::
+
+   Exporting a Pod label through ``kube_pod_labels`` is a separate kube-state-metrics allow-list decision; see :ref:`adding_observability`.

docs/deployment_guide/index.rst

@@ -149,6 +149,7 @@ An OSMO deployment consists of two main components:
   advanced_config/scheduler
   advanced_config/rsync
   advanced_config/workflow_pod_workload_identity
+  advanced_config/workflow_labels
   advanced_config/mcp
 
 .. toctree::

docs/deployment_guide/references/configs_definitions/workflow.rst

@@ -72,7 +72,8 @@ Top-Level Configuration
      - See Plugins section
    * - ``labels_config``
      - `Workflow Labels`_
-     - Workflow-label policies, accepted values, and staged enforcement.
+     - Workflow-label policies, accepted values, staged enforcement, and the
+       optional pod-label prefix.
      - ``policy: []``
    * - ``max_num_tasks``
      - Integer
@@ -247,29 +248,31 @@ Workflow Information
 Workflow Labels
 ===============
 
-Workflow labels are optional and format-checked even when no label is required.
-The default configuration applies no label policies:
+Policy and prefix behavior, enforcement rollout, and the exported metrics are
+described in :ref:`workflow_labels_config`.
 
-.. code-block:: yaml
-
-   labels_config:
-     policy: []
-
-Each entry in ``policy`` controls one key independently. Use ``off`` or omit a
-key from ``policy`` to disable both warnings and enforcement for that key:
+.. list-table::
+   :header-rows: 1
+   :widths: 25 12 43 20
 
-.. code-block:: yaml
+   * - **Field**
+     - **Type**
+     - **Description**
+     - **Default Values**
+   * - ``policy``
+     - List of `Label Policy`_
+     - One entry per label key to check. At most 16 entries; duplicate keys are
+       rejected.
+     - ``[]``
+   * - ``pod_label_prefix``
+     - String
+     - Prepended to each workflow label key on task pods only. Empty disables
+       it. Rejected at configuration time if it contains a space, tab, or line
+       break, or exceeds 253 characters.
+     - ``""``
 
-   labels_config:
-     policy:
-     - key: team
-       allow_list:
-       - robotics
-       - simulation
-       enforcement: warn
-     - key: cost-center
-       allow_list: []
-       enforcement: enforce
+Label Policy
+============
 
 .. list-table::
    :header-rows: 1
@@ -281,55 +284,19 @@ key from ``policy`` to disable both warnings and enforcement for that key:
      - **Default Values**
    * - ``key``
      - String
-     - Kubernetes label key to check. Duplicate policy keys are rejected,
-       and at most 16 keys can be configured.
+     - Kubernetes label key to check.
      - Required
    * - ``allow_list``
      - List of Strings
      - Exact accepted values. An empty list accepts any well-formed value.
      - ``[]``
    * - ``enforcement``
      - String (``"off"``, ``warn``, ``enforce``)
-     - ``off`` accepts without policy warnings. ``warn`` accepts but warns
-       when the key is missing or its value is outside a non-empty allow-list.
+     - ``off`` accepts without policy warnings. ``warn`` accepts but warns when
+       the key is missing or its value is outside a non-empty allow-list.
        ``enforce`` rejects those violations.
      - ``"off"``
 
-The same policy applies to new submissions, resubmission by ID, restart, and
-validation-only requests. An ``enforcement: enforce`` rejection creates
-neither a workflow row nor a stored specification. Submit responses carry
-warnings from that admission check. Warnings are not stored with the
-workflow: detail responses recompute warn-mode violations from the stored
-labels and the current configuration, so displayed warnings track policy
-changes even for completed workflows.
-
-To roll back enforcement immediately, use ``enforcement: warn``. To disable both
-warnings and enforcement, use ``enforcement: "off"`` (quoted: unquoted YAML
-``off`` parses as boolean false) or remove the policy entry.
-Existing and in-flight workflows are not modified, although their detail-page
-warnings always reflect the current warn policy. In ConfigMap mode, an invalid
-edit is rejected and the previous valid snapshot remains active.
-
-Only configured policy keys become workflow-label dimensions on
-``osmo_tasks_count``. Attribute names start with ``workflow_label_``. Letters
-and numbers are unchanged; ``_``, ``-``, ``.``, and ``/`` are encoded as
-``__``, ``_dash_``, ``_dot_``, and ``_slash_`` respectively. For example,
-``project`` is exported as ``workflow_label_project``. Values in the configured
-allow-list are exported verbatim; a present value outside that list is clamped
-to ``<other>``, and a missing key is reported as ``<missing>``. Angle
-brackets are not valid in label values, so the sentinels never collide with
-real values. An empty allow-list exports every present value as ``<other>``.
-This keeps the number of series bounded to the allow-list plus two sentinels
-per key.
-
-Admission also emits
-``osmo_label_validation_total{key, outcome}``, where ``outcome`` is ``ok``,
-``missing``, ``invalid``, or ``rejected``. The counter covers rejected
-submissions that do not create a workflow row. Keep the policy list small to
-control metric cardinality. Exporting a Pod label through ``kube_pod_labels`` is a
-separate kube-state-metrics allow-list decision; see
-:ref:`adding_observability`.
-
 Backend Images
 ==============
 

docs/spelling_wordlist.txt

@@ -248,6 +248,7 @@ preflight
 Preflight
 prem
 prepend
+prepended
 Preprocess
 programmatically
 prometheus

docs/user_guide/workflows/specification/index.rst

@@ -69,13 +69,12 @@ Workflow
 Workflow Labels
 ---------------
 
-Labels identify a workflow independently of mutable tags. Keys and values use
-Kubernetes label syntax, values must be non-empty, and a workflow can define
-at most 16 labels. Labels are stored with the submitted specification and
-copied only to task pods, not to Services, Secrets, scheduler groups, or
-other objects. Any syntactically valid key is accepted; where a workflow
-label collides with a system-owned pod label (the ``osmo.`` selectors or
-scheduler queue labels), the system value wins on the pod.
+Labels are immutable metadata for a workflow, set at submission. Keys and
+values use Kubernetes label syntax, values must be non-empty, and a workflow
+can define at most 16 labels. Labels are stored with the submitted
+specification and copied to task pods only. Any syntactically valid key is
+accepted; if a key collides with a system-owned pod label (an ``osmo.``
+selector or a scheduler queue label), the system value wins on the pod.
 
 .. code-block:: yaml
 
@@ -90,10 +89,14 @@ scheduler queue labels), the system value wins on the pod.
        command: [bash]
        args: [-lc, "echo training"]
 
-Your administrator may configure particular keys in ``off``, ``warn``, or
-``enforce`` mode. A submission can succeed and still print a warning while an
-administrator is rolling out a requirement. Use :ref:`workflow_submission` to
-validate and override labels without editing a shared specification.
+Your administrator may require particular keys. In ``warn`` mode a submission
+succeeds but prints a warning, so a requirement can be announced before it is
+enforced; in ``enforce`` mode it is rejected. Use :ref:`workflow_submission` to
+validate labels, or to override them without editing a shared specification.
+
+Your administrator may also configure a prefix added to label keys on task
+pods, so a pod can show ``example.com/team`` where the specification said
+``team``.
 
 .. _workflow_spec_task:
 

docs/user_guide/workflows/submission.rst

@@ -223,18 +223,16 @@ or a missing key:
    $ osmo workflow list --label 'team=robotics_(a|b)'
    $ osmo workflow list --no-label team
 
-Label selectors are case-sensitive. In a glob selector, ``*`` matches zero or
-more characters; every other character, including ``_``, is literal. A
-parenthesized ``|`` group can appear within a selector and matches any one of
-its alternatives. Alternatives can contain ``*`` wildcards. Groups are flat
-(not nested), each group must contain at least two non-empty alternatives,
-and the alternatives in a selector may multiply out to at most 32
-combinations. Quote pattern selectors so the shell does not interpret them.
-
-Repeated ``--label`` filters are combined with AND, so every supplied
-selector must match. Alternatives within one selector are combined with OR. Pattern syntax
-applies only to workflow list filters; submission and validation labels must
-still contain exact Kubernetes label values.
+Label selectors are case-sensitive. ``*`` matches zero or more characters;
+every other character, including ``_``, is literal. A parenthesized ``|`` group
+matches any one of its alternatives, which can themselves contain ``*``. Groups
+cannot nest, each needs at least two alternatives, and one selector can expand
+to at most 32 combinations. Quote selectors so the shell does not interpret
+them.
+
+Repeated ``--label`` filters combine with AND; alternatives within one selector
+combine with OR. Pattern syntax applies only to list filters: submission and
+validation still require exact Kubernetes label values.
 
 Dry Run Validation
 ~~~~~~~~~~~~~~~~~~