Add workflow label CLI support (#1225)

* Add workflow label CLI support

Add repeatable --label KEY=VALUE flags to workflow submit, workflow
validate, and app submit (flags override the YAML per run); --label
selector and --no-label filters on workflow list; a Labels column in
list output and a Labels line in workflow query; and WARNING lines for
warn-mode policy messages on submit and validate responses. Document
the spec field, submit flags, list filters, and the labels_config
admin reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Apply CLI review simplifications

Simplify the Labels column fallback, drop two low-value argparse tests,
remove a dead spelling-wordlist entry, and move the osmo_tasks_count
label-dimension reference docs to the metrics PR that implements them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Drop the workflow-label reserved-key deny-list

Any syntactically valid Kubernetes label key is accepted, including
system domains such as app.kubernetes.io/name, which the deny-list
wrongly blocked. Protection against user labels actuating scheduling or
platform machinery comes from merge order at stamping time: system
labels are applied after the user map on every path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Polish CLI help texts and label documentation

Align argparse continuation style with sibling flags, give the list
filter help a subject and generic key examples, add missing test class
docstrings, restructure the admin policy table to the file's standard
format, quote 'off' in the rollback guidance, title-case headings, and
replace design-register phrasing in the user guide with the actual
warning output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: jiaenren

docs/deployment_guide/references/configs_definitions/workflow.rst

@@ -70,6 +70,10 @@ Top-Level Configuration
      - `Plugins`_
      - Configuration for workflow plugins.
      - See Plugins section
+   * - ``labels_config``
+     - `Workflow Labels`_
+     - Workflow-label policies, accepted values, and staged enforcement.
+     - ``policy: []``
    * - ``max_num_tasks``
      - Integer
      - Maximum number of tasks allowed in a workflow.
@@ -240,6 +244,80 @@ Workflow Information
      - Maximum allowed length for workflow names.
      - ``64``
 
+Workflow Labels
+===============
+
+Workflow labels are optional and format-checked even when no label is required.
+The default configuration applies no label policies:
+
+.. code-block:: yaml
+
+   labels_config:
+     policy: []
+
+Each entry in ``policy`` controls one key independently. Use ``off`` or omit a
+key from ``policy`` to disable both warnings and enforcement for that key:
+
+.. code-block:: yaml
+
+   labels_config:
+     policy:
+     - key: team
+       allow_list:
+       - robotics
+       - simulation
+       enforcement: warn
+     - key: cost-center
+       allow_list: []
+       enforcement: enforce
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 12 43 20
+
+   * - **Field**
+     - **Type**
+     - **Description**
+     - **Default Values**
+   * - ``key``
+     - String
+     - Kubernetes label key to check. Duplicate policy keys are rejected,
+       and at most 16 keys can be configured.
+     - Required
+   * - ``allow_list``
+     - List of Strings
+     - Exact accepted values. An empty list accepts any well-formed value.
+     - ``[]``
+   * - ``enforcement``
+     - String (``"off"``, ``warn``, ``enforce``)
+     - ``off`` accepts without policy warnings. ``warn`` accepts but warns
+       when the key is missing or its value is outside a non-empty allow-list.
+       ``enforce`` rejects those violations.
+     - ``"off"``
+
+The same policy applies to new submissions, resubmission by ID, restart, and
+validation-only requests. An ``enforcement: enforce`` rejection creates
+neither a workflow row nor a stored specification. Submit responses carry
+warnings from that admission check. Warnings are not stored with the
+workflow: detail responses recompute warn-mode violations from the stored
+labels and the current configuration, so displayed warnings track policy
+changes even for completed workflows.
+
+To roll back enforcement immediately, use ``enforcement: warn``. To disable both
+warnings and enforcement, use ``enforcement: "off"`` (quoted: unquoted YAML
+``off`` parses as boolean false) or remove the policy entry.
+Existing and in-flight workflows are not modified, although their detail-page
+warnings always reflect the current warn policy. In ConfigMap mode, an invalid
+edit is rejected and the previous valid snapshot remains active.
+
+Admission emits
+``osmo_label_validation_total{key, outcome}``, where ``outcome`` is ``ok``,
+``missing``, ``invalid``, or ``rejected``. The counter covers rejected
+submissions that do not create a workflow row. Keep the policy list small to
+control metric cardinality. Exporting a Pod label through ``kube_pod_labels`` is a
+separate kube-state-metrics allow-list decision; see
+:ref:`adding_observability`.
+
 Backend Images
 ==============
 

docs/spelling_wordlist.txt

@@ -266,6 +266,7 @@ resrcs
 restitutions
 retryable
 reusability
+rollout
 ros
 routable
 rpc

docs/user_guide/workflows/specification/index.rst

@@ -42,6 +42,11 @@ Workflow
      - ``string``
      - No
      - The pool to submit the workflow to.
+   * - :kbd:`labels`
+     - ``dict``
+     - No
+     - Immutable workflow metadata copied to every task pod. See
+       :ref:`workflow_spec_labels`.
    * - :kbd:`timeout`
      - ``dict``
      - No
@@ -59,6 +64,37 @@ Workflow
      - **Yes** (or :kbd:`tasks`)
      - List of :ref:`group definitions <workflow_spec_group>`.
 
+.. _workflow_spec_labels:
+
+Workflow Labels
+---------------
+
+Labels identify a workflow independently of mutable tags. Keys and values use
+Kubernetes label syntax, values must be non-empty, and a workflow can define
+at most 16 labels. Labels are stored with the submitted specification and
+copied only to task pods, not to Services, Secrets, scheduler groups, or
+other objects. Any syntactically valid key is accepted; where a workflow
+label collides with a system-owned pod label (the ``osmo.`` selectors or
+scheduler queue labels), the system value wins on the pod.
+
+.. code-block:: yaml
+
+   workflow:
+     name: training
+     labels:
+       team: robotics
+       experiment: run42
+     tasks:
+     - name: train
+       image: ubuntu:24.04
+       command: [bash]
+       args: [-lc, "echo training"]
+
+Your administrator may configure particular keys in ``off``, ``warn``, or
+``enforce`` mode. A submission can succeed and still print a warning while an
+administrator is rolling out a requirement. Use :ref:`workflow_submission` to
+validate and override labels without editing a shared specification.
+
 .. _workflow_spec_task:
 
 Task

docs/user_guide/workflows/submission.rst

@@ -179,6 +179,63 @@ Override workflow values at submission time:
        --set-string model_name="bert-base" \
        --set-env WANDB_PROJECT=my-experiment
 
+Set Workflow Labels
+~~~~~~~~~~~~~~~~~~~
+
+Add or override immutable workflow labels with repeatable ``--label key=value``
+flags. A command-line value wins over the same key in the workflow YAML.
+
+.. code-block:: bash
+
+   $ osmo workflow submit training.yaml \
+       --label team=robotics \
+       --label experiment=run42
+
+The same flag works for app submission and resubmission by workflow ID
+(``osmo workflow submit <workflow-id>``). Restart does not accept label
+overrides because it reuses the stored specification; resubmit the workflow
+when a stored label must change.
+
+Validate labels and any administrator policy without creating a workflow:
+
+.. code-block:: bash
+
+   $ osmo workflow validate training.yaml --label team=robotics
+
+If a label policy is in warn mode, the submission succeeds but prints a
+``WARNING:`` line explaining what to change before the policy is enforced,
+for example::
+
+   WARNING: Workflow is missing label 'team'; add it now to avoid rejected
+   submissions once it is required.
+
+In enforce mode the submission is rejected and the error identifies the
+missing key or disallowed value.
+
+Find workflows with an exact label, a glob selector, an alternative selector,
+or a missing key:
+
+.. code-block:: bash
+
+   $ osmo workflow list --label team=robotics --label experiment=run42
+   $ osmo workflow list --label 'project=robotics_*'
+   $ osmo workflow list --label 'project=(sim_*|hil_*)'
+   $ osmo workflow list --label 'team=robotics_(a|b)'
+   $ osmo workflow list --no-label team
+
+Label selectors are case-sensitive. In a glob selector, ``*`` matches zero or
+more characters; every other character, including ``_``, is literal. A
+parenthesized ``|`` group can appear within a selector and matches any one of
+its alternatives. Alternatives can contain ``*`` wildcards. Groups are flat
+(not nested), each group must contain at least two non-empty alternatives,
+and the alternatives in a selector may multiply out to at most 32
+combinations. Quote pattern selectors so the shell does not interpret them.
+
+Repeated ``--label`` filters are combined with AND, so every supplied
+selector must match. Alternatives within one selector are combined with OR. Pattern syntax
+applies only to workflow list filters; submission and validation labels must
+still contain exact Kubernetes label values.
+
 Dry Run Validation
 ~~~~~~~~~~~~~~~~~~
 

src/cli/app.py

@@ -219,6 +219,13 @@ def setup_parser(parser: argparse._SubParsersAction):
                                     'LOW. LOW workflows may be preempted to allow a '
                                     'higher priority workflow to run.',
                                choices=[p.value for p in wf_priority.WorkflowPriority])
+    submit_parser.add_argument('--label',
+                               action='append',
+                               dest='labels',
+                               default=[],
+                               metavar='KEY=VALUE',
+                               help='Set a workflow label. Repeat to set multiple labels. '
+                                    'Values override labels declared by the app.')
     submit_parser.set_defaults(func=_submit_app)
 
 
@@ -511,6 +518,8 @@ def _submit_app(service_client: client.ServiceClient, args: argparse.Namespace):
 
     if args.priority:
         params['priority'] = args.priority
+    if args.labels:
+        params['label'] = args.labels
 
     template_data = workflow.parse_file_for_template(app_spec_result,
                                                      args.set,

src/cli/tests/BUILD

@@ -28,6 +28,16 @@ py_test(
     ]
 )
 
+py_test(
+    name = "test_workflow",
+    srcs = [
+        "test_workflow.py"
+    ],
+    deps = [
+        "//src/cli:cli_lib",
+    ]
+)
+
 py_test(
     name = "test_config_update",
     srcs = [
@@ -87,13 +97,3 @@ py_test(
         "//src/cli:cli_lib",
     ]
 )
-
-py_test(
-    name = "test_workflow",
-    srcs = [
-        "test_workflow.py"
-    ],
-    deps = [
-        "//src/cli:cli_lib",
-    ]
-)

src/cli/tests/test_app.py

@@ -158,6 +158,15 @@ def test_submit_command_set_options(self):
         self.assertEqual(args.set_string, ['s=foo'])
         self.assertEqual(args.set_env, ['K=V'])
 
+    def test_submit_command_labels(self):
+        parser = self._build_parser()
+        args = parser.parse_args([
+            'app', 'submit', 'my-app',
+            '--label', 'team=alpha', '--label', 'run=42',
+        ])
+
+        self.assertEqual(args.labels, ['team=alpha', 'run=42'])
+
 
 class TestCreateApp(unittest.TestCase):
     """Test cases for _create_app."""
@@ -649,6 +658,7 @@ def _make_args(self, **overrides) -> argparse.Namespace:
             'local_path': '/some/path',
             'rsync': None,
             'format_type': 'text',
+            'labels': [],
         }
         defaults.update(overrides)
         return argparse.Namespace(**defaults)
@@ -674,6 +684,22 @@ def test_submit_app_with_pool_and_priority(self):
         self.assertEqual(params['app_version'], 3)
         self.assertEqual(params['priority'], 'HIGH')
 
+    def test_submit_app_forwards_label_overrides(self):
+        service_client = mock.Mock(spec=client.ServiceClient)
+        service_client.request.side_effect = [
+            {'uuid': 'uuid-1', 'versions': [{'version': 3}]},
+            'spec-text',
+        ]
+        args = self._make_args(labels=['team=alpha', 'run=42'])
+
+        with mock.patch('src.cli.app.workflow.parse_file_for_template',
+                        return_value='template-data'), \
+             mock.patch('src.cli.app.workflow.submit_workflow_helper') as submit_mock:
+            app._submit_app(service_client, args)
+
+        params = submit_mock.call_args.args[4]
+        self.assertEqual(params['label'], ['team=alpha', 'run=42'])
+
     def test_submit_app_no_pool_fetches_default(self):
         service_client = mock.Mock(spec=client.ServiceClient)
         service_client.request.side_effect = [