feat: Add pip package management samples (aws-deadline#251)

crowecawcaw · web-flow · commit d7684df94393 · 2026-07-08T22:23:24.000Z
Add samples for managing Python dependencies with pip, mirroring the
existing Conda samples:

- queue_environments/pip_queue_env.yaml: a queue environment that builds
  a Python virtual environment with venv and installs the PipPackages
  parameter into it, with optional PipIndexUrl/PipExtraIndexUrls for
  private indexes. No-ops when PipPackages is empty.
- job_bundles/pip_package_job: a job bundle that relies on the pip queue
  environment to provide its dependencies.
- job_bundles/pip_self_contained_job: a job bundle that manages its own
  pip virtual environment inline via a jobEnvironment, needing no queue
  environment.

The pip self-upgrade and package install both pass the index options so a
private PipIndexUrl works on workers that cannot reach public PyPI.

Both bundles were verified end-to-end on a Deadline Cloud service-managed
Linux fleet, and locally with the Open Job Description CLI.

Signed-off-by: Stephen Crowe &lt;6042774+crowecawcaw@users.noreply.github.com&gt;
diff --git a/job_bundles/pip_package_job/README.md b/job_bundles/pip_package_job/README.md
@@ -0,0 +1,53 @@
+# Pip Package Job
+
+This job bundle demonstrates providing a job's Python dependencies with
+[pip](https://pip.pypa.io/) through a **queue environment**. The job itself only
+declares the `PipPackages`, `PipIndexUrl`, and `PipExtraIndexUrls` parameters —
+the [`pip_queue_env.yaml`](../../queue_environments/pip_queue_env.yaml) queue
+environment reads those parameters, builds a Python virtual environment, installs
+the packages, and puts the environment on the `PATH` before the step runs.
+
+This is the pip analogue of the Conda samples that pass `CondaPackages` to a
+Conda queue environment (see [`monte_carlo_simulation`](../monte_carlo_simulation)).
+Use this style when you want to define the pip environment once and share it
+across many jobs on a queue.
+
+## Prerequisites
+
+- A queue with the [`pip_queue_env.yaml`](../../queue_environments/pip_queue_env.yaml)
+  queue environment added to it. See the
+  [queue_environments README](../../queue_environments/README.md) for how to
+  create a queue environment.
+- A Linux worker with `python3` available on the `PATH`. Deadline Cloud
+  service-managed fleets satisfy this.
+
+## Submit the job
+
+```bash
+deadline bundle submit job_bundles/pip_package_job
+```
+
+By default it installs the [`cowsay`](https://pypi.org/project/cowsay/) package
+and prints a message. Override the parameters to install your own packages, for
+example:
+
+```bash
+deadline bundle submit job_bundles/pip_package_job \
+  -p PipPackages="requests rich" \
+  -p Message="Hello from pip"
+```
+
+To install from a private index such as AWS CodeArtifact, set `PipIndexUrl` (and
+optionally `PipExtraIndexUrls`) to the index endpoint.
+
+## Test it locally
+
+You can run the job with the queue environment locally using the
+[Open Job Description CLI](https://github.com/OpenJobDescription/openjd-cli):
+
+```bash
+openjd run job_bundles/pip_package_job/template.yaml \
+  --step SayHello \
+  --environment queue_environments/pip_queue_env.yaml \
+  --job-param PipPackages=cowsay
+```
diff --git a/job_bundles/pip_package_job/template.yaml b/job_bundles/pip_package_job/template.yaml
@@ -0,0 +1,66 @@
+specificationVersion: 'jobtemplate-2023-09'
+name: Pip Package Job
+
+description: |
+  Demonstrates running a job whose Python dependencies are provided by pip through
+  a queue environment. The job declares the PipPackages and PipIndexUrl parameters
+  that the pip_queue_env.yaml queue environment consumes to build a Python virtual
+  environment before the step runs.
+
+  This mirrors how the Conda samples pass CondaPackages to a Conda queue environment.
+  Add the queue_environments/pip_queue_env.yaml queue environment to your queue for
+  this job to work.
+
+parameterDefinitions:
+- name: Message
+  type: STRING
+  default: "Hello from a pip-managed virtual environment!"
+  description: "The message the cowsay package prints."
+  userInterface:
+    control: LINE_EDIT
+    label: Message
+- name: PipPackages
+  type: STRING
+  default: "cowsay"
+  description: "A space-separated list of pip requirement specifiers the queue environment installs."
+  userInterface:
+    control: LINE_EDIT
+    label: Pip Packages
+    groupLabel: Software Environment
+- name: PipIndexUrl
+  type: STRING
+  default: ""
+  description: "The base Python package index URL. Leave empty to use the default PyPI index."
+  userInterface:
+    control: LINE_EDIT
+    label: Pip Index URL
+    groupLabel: Software Environment
+- name: PipExtraIndexUrls
+  type: STRING
+  default: ""
+  description: "A space-separated list of additional package index URLs."
+  userInterface:
+    control: LINE_EDIT
+    label: Pip Extra Index URLs
+    groupLabel: Software Environment
+
+steps:
+- name: SayHello
+  hostRequirements:
+    attributes:
+    - name: attr.worker.os.family
+      anyOf: ["linux"]
+  script:
+    actions:
+      onRun:
+        command: python
+        args: ['{{Task.File.Run}}']
+    embeddedFiles:
+    - name: Run
+      type: TEXT
+      data: |
+        # cowsay is provided by the pip queue environment's virtual environment,
+        # which is on the PATH for this step.
+        import cowsay
+
+        cowsay.cow("{{Param.Message}}")
diff --git a/job_bundles/pip_self_contained_job/README.md b/job_bundles/pip_self_contained_job/README.md
@@ -0,0 +1,47 @@
+# Pip Self-Contained Job
+
+This job bundle demonstrates managing [pip](https://pip.pypa.io/) packages
+entirely **within the job bundle**, with no queue environment required. The
+template defines its own Open Job Description
+[job environment](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas#4-environment)
+that creates a Python virtual environment, installs the requested packages into
+it, and adds it to the `PATH` for all steps of the job.
+
+Use this style when you want a self-contained bundle that runs on any Linux
+worker with `python3` available, without configuring anything on the queue. If
+you would rather define the pip environment once and share it across many jobs,
+see the [`pip_package_job`](../pip_package_job) sample and the
+[`pip_queue_env.yaml`](../../queue_environments/pip_queue_env.yaml) queue
+environment instead.
+
+This is the pip analogue of an inline job environment; compare it with the
+[`job_env_with_new_command`](../job_env_with_new_command) sample.
+
+## Prerequisites
+
+- A Linux worker with `python3` available on the `PATH`. Deadline Cloud
+  service-managed fleets satisfy this.
+
+## Submit the job
+
+```bash
+deadline bundle submit job_bundles/pip_self_contained_job
+```
+
+By default it installs the [`cowsay`](https://pypi.org/project/cowsay/) package
+and prints a message. Override the parameters to install your own packages:
+
+```bash
+deadline bundle submit job_bundles/pip_self_contained_job \
+  -p PipPackages="requests rich" \
+  -p Message="Hello from pip"
+```
+
+## Test it locally
+
+You can run the job locally using the
+[Open Job Description CLI](https://github.com/OpenJobDescription/openjd-cli):
+
+```bash
+openjd run job_bundles/pip_self_contained_job/template.yaml --step SayHello
+```
diff --git a/job_bundles/pip_self_contained_job/template.yaml b/job_bundles/pip_self_contained_job/template.yaml
@@ -0,0 +1,170 @@
+specificationVersion: 'jobtemplate-2023-09'
+name: Pip Self-Contained Job
+
+description: |
+  Demonstrates managing pip packages entirely within a job bundle, without
+  relying on a queue environment. The job defines its own Open Job Description
+  job environment that creates a Python virtual environment, installs the
+  requested pip packages into it, and adds it to the PATH for all steps.
+
+  Use this style when you want a self-contained job bundle that runs on any
+  Linux worker with python3 available, and you do not want to configure a
+  queue environment. If you would rather share one pip environment definition
+  across many jobs, see the pip_package_job sample and the
+  queue_environments/pip_queue_env.yaml queue environment instead.
+
+parameterDefinitions:
+- name: Message
+  type: STRING
+  default: "Hello from a self-contained pip virtual environment!"
+  description: "The message the cowsay package prints."
+  userInterface:
+    control: LINE_EDIT
+    label: Message
+- name: PipPackages
+  type: STRING
+  default: "cowsay"
+  description: "A space-separated list of pip requirement specifiers to install into the job's virtual environment."
+  userInterface:
+    control: LINE_EDIT
+    label: Pip Packages
+    groupLabel: Software Environment
+- name: PipIndexUrl
+  type: STRING
+  default: ""
+  description: "The base Python package index URL. Leave empty to use the default PyPI index."
+  userInterface:
+    control: LINE_EDIT
+    label: Pip Index URL
+    groupLabel: Software Environment
+
+jobEnvironments:
+- name: PipVirtualEnv
+  description: Creates a Python virtual environment and installs the PipPackages into it.
+  script:
+    actions:
+      onEnter:
+        command: bash
+        args:
+        - "{{Env.File.Enter}}"
+    embeddedFiles:
+    - name: Enter
+      type: TEXT
+      data: |
+        #!/bin/env bash
+        set -euo pipefail
+
+        if [ -z '{{Param.PipPackages}}' ]; then
+            echo "Skipping pip virtual environment as PipPackages parameter was empty."
+            exit 0
+        fi
+
+        PIP_PACKAGES='{{Param.PipPackages}}'
+
+        INDEX_OPTS=""
+        if [ -n '{{Param.PipIndexUrl}}' ]; then
+            INDEX_OPTS="--index-url {{Param.PipIndexUrl}}"
+        fi
+
+        # Find a Python interpreter. Prefer python3, but fall back to python.
+        if command -v python3 > /dev/null 2>&1; then
+            PYTHON=python3
+        else
+            PYTHON=python
+        fi
+
+        echo "Creating a Python virtual environment in the session directory..."
+
+        # Create the virtual environment in the session working directory so it
+        # is cleaned up automatically when the session ends.
+        ENV_DIR="$(mktemp -d '{{Session.WorkingDirectory}}/.venv-XXXXX')"
+        "$PYTHON" -m venv "$ENV_DIR"
+
+        # A venv puts its executables in bin/ on Linux and macOS, and in
+        # Scripts/ on Windows.
+        if [ -d "$ENV_DIR/bin" ]; then
+            VENV_BIN="$ENV_DIR/bin"
+        else
+            VENV_BIN="$ENV_DIR/Scripts"
+        fi
+
+        # Capture the environment variables before activating the virtual environment
+        "$PYTHON" '{{Env.File.OpenJDVarsStart}}' .vars
+
+        # Activate the virtual environment.
+        set +u
+        source "$VENV_BIN/activate"
+        set -u
+
+        # Pass the index options to both commands so a private PipIndexUrl also
+        # works on workers that cannot reach public PyPI.
+        pip install $INDEX_OPTS --quiet --upgrade pip
+        pip install $INDEX_OPTS $PIP_PACKAGES
+
+        # Export the environment variables set by activation to all steps
+        "$VENV_BIN/python" '{{Env.File.OpenJDVarsCapture}}' .vars
+
+        echo "Virtual environment created at $ENV_DIR"
+        pip list
+    - name: OpenJDVarsStart
+      filename: openjd-vars-start.py
+      type: TEXT
+      data: |
+        import json
+        import os
+        import sys
+
+        # Exclude the env var "_" as it has special meaning to shells
+        before = dict(os.environ)
+        if "_" in before:
+            del before["_"]
+
+        with open(sys.argv[1], "w", encoding="utf8") as f:
+            json.dump(before, f)
+    - name: OpenJDVarsCapture
+      filename: openjd-vars-capture.py
+      type: TEXT
+      data: |
+        import json
+        import os
+        import sys
+
+        # Get the snapshot from `openjd-vars-start.py`, and the current environment state.
+        with open(sys.argv[1], "r", encoding="utf8") as f:
+            before = json.load(f)
+        after = dict(os.environ)
+        # Exclude the env var "_" as it has special meaning to shells
+        if "_" in after:
+            del after["_"]
+
+        # Identify the modified and deleted environment variables
+        vars_to_put = {k: v for k, v in after.items() if v != before.get(k)}
+        vars_to_delete = {k for k in before if k not in after}
+
+        # Print the env var changes following the Open Job Description specification
+        for k, v in vars_to_put.items():
+            kv = json.dumps(f"{k}={v}", ensure_ascii=True)
+            print(f"openjd_env: {kv}")
+        for k in vars_to_delete:
+            print(f"openjd_unset_env: {k}")
+
+steps:
+- name: SayHello
+  hostRequirements:
+    attributes:
+    - name: attr.worker.os.family
+      anyOf: ["linux"]
+  script:
+    actions:
+      onRun:
+        command: python
+        args: ['{{Task.File.Run}}']
+    embeddedFiles:
+    - name: Run
+      type: TEXT
+      data: |
+        # cowsay is provided by the PipVirtualEnv job environment, which put its
+        # virtual environment on the PATH for this step.
+        import cowsay
+
+        cowsay.cow("{{Param.Message}}")
diff --git a/queue_environments/README.md b/queue_environments/README.md
@@ -8,7 +8,8 @@ follow the environment template specification from
 
 The Conda and Rez queue environments let you provide software applications to jobs in your
 Deadline Cloud queue, so each job only needs a parameter value for `CondaPackages` or `RezPackages`
-to tell it the list of packages to use.
+to tell it the list of packages to use. The pip queue environment does the same for Python
+packages, so a job only needs to provide a value for `PipPackages`.
 
 ## Create a queue environment for your queue
 
@@ -210,6 +211,26 @@ jobs. The default environment name uses the hash of the Conda channels and packa
 set the name in the job. It also includes a parameter for how long to use an environment without running a package
 update, so that most of the time it will take seconds to activate an environment that's being reused.
 
+### Pip queue environment
+
+The file [pip_queue_env.yaml](pip_queue_env.yaml) lets you provide Python packages to jobs using
+[pip](https://pip.pypa.io/) and the standard library [venv](https://docs.python.org/3/library/venv.html)
+module, rather than a package manager like Conda or Rez. When a job provides a `PipPackages` parameter
+value, the queue environment creates a Python virtual environment in the session working directory,
+installs the requested packages into it with pip, and activates it so subsequent steps run with those
+packages available. If `PipPackages` is empty, the queue environment does nothing, so it is safe to add
+to a queue that also runs jobs which do not use it.
+
+The `PipIndexUrl` and `PipExtraIndexUrls` parameters let jobs install from a private package index, such
+as an [AWS CodeArtifact](https://docs.aws.amazon.com/codeartifact/) repository, instead of the default
+[PyPI](https://pypi.org/) index.
+
+Unlike Conda and Rez, pip and venv are included with Python itself, so worker hosts only need a `python3`
+(or `python`) interpreter on the `PATH`. Deadline Cloud service-managed fleets provide one. The
+[pip_package_job](../job_bundles/pip_package_job) job bundle shows how to submit a job that uses this
+queue environment, and [pip_self_contained_job](../job_bundles/pip_self_contained_job) shows the same
+pip environment defined inline in a job bundle when you do not want to configure a queue environment.
+
 ### Disconnect UBL queue environment
 
 The file [disconnect_ubl_queue_env.yaml](disconnect_ubl_queue_env.yaml) unsets Deadline Cloud Usage Based
diff --git a/queue_environments/pip_queue_env.yaml b/queue_environments/pip_queue_env.yaml
