Add OpenFold3 SageMaker DLC (build + tests, incl. precomputed-MSA under network isolation) (#6285)

* Add OpenFold3 SageMaker DLC (build + basic tests)

OpenFold3 protein-structure prediction image for SageMaker async inference,
private release to JumpStart.

- docker/openfold3/Dockerfile: 2-stage build on devel-cu130-amzn2023, conda-lock
  env, OpenFold3 pinned to tag 0.4.1, baked checkpoint of3-p2-155k.pt
- scripts/docker/openfold3/: Flask+gunicorn serve, SageMaker handler (GPU pool,
  per-GPU warmup), entrypoint
- .github/config/image/openfold3/sagemaker.yml: release:false, public_registry:
  false (private-only); tag 0.4-cu130-amzn2023
- .github/workflows/openfold3.{pipeline,pr-amzn2023,autorelease-sagemaker-amzn2023}.yml:
  minimal pipeline (build + sanity + security + release)
- release-schedule.yml: register autorelease cron

Model tests and SageMaker endpoint tests intentionally omitted (added later
once test artifacts are uploaded to S3).

* Apply dockerfmt formatting to openfold3 Dockerfile

* Enable pull_request trigger for openfold3 PR workflow

* Fix and allowlist openfold3 ECR scan CVEs

ECR enhanced scan flagged 43 CRITICAL/HIGH CVEs. Remediation:

Fixed (14, in Dockerfile): upgrade pure-Python utility libs not in the
model dependency graph — GitPython>=3.1.50, urllib3>=2.7.0, pillow>=12.2.0,
dulwich>=1.2.5, msgpack>=1.2.1.

Allowlisted (29, test/security/data/ecr_scan_allowlist/openfold3/):
- openssl (21) + curl (3): conda-vendored native libs pinned by OpenFold3
  0.4.1 upstream lock; upgrading risks breaking the locked env.
- torch 2.5.1 (3): official upstream pin (cuda12.4); CVE-2025-32434 torch.load
  RCE not reachable (baked checkpoint, no untrusted load).
- quinn-proto (1): inside the uv build-tool binary, not runtime.
- jaraco.context (1): vendored in setuptools.
All allowlist entries carry review_by 2026-09-30.

* Fully remediate openfold3 Python CVEs in both envs + purge conda cache

The first remediation pass left 14 CVEs because:
- conda's extracted package cache (/opt/conda/pkgs/<pkg>-<oldver>) kept old
  METADATA the scanner reads, even after the active env was upgraded
- awscli (installed for the checkpoint download) pinned urllib3<2.7, downgrading
  the patched urllib3
- msgpack was only upgraded in the openfold3 env, not the base env

Fix: upgrade the libs in BOTH base and openfold3 envs, run remediation last
(after awscli is removed), and purge the conda pkg cache + conda clean.

* Fetch checkpoint via curl instead of awscli

awscli pulled in botocore, which pins urllib3<2.7 and conflicted with the CVE
remediation (capping urllib3 at 2.6.3). The checkpoint is a public, anonymously
readable S3 object, so fetch it over HTTPS with curl like the other DLCs do —
removing the awscli/botocore dependency entirely (no install/uninstall dance).

* Add OpenFold3 SageMaker async-inference tests

Deploys one 4-GPU ml.g6.12xlarge async endpoint (reused across cases) and
exercises it via S3 in/out. Three cases:
- small protein, 1 request (smoke)
- large protein, 1 request (baseline timing)
- large protein, 4 concurrent: asserts the GPU pool keeps wall-time under 2x
  the single-request time (vs ~4x if serialized)

Validation is smoke-level (valid non-empty CIF). Wires a sagemaker-tests job
into the pipeline (gated on customer_type==sagemaker) + PR change detection.

* Inline SageMaker test fixtures, drop conftest.py

Match the repo convention: vllm/sglang/ray/vllm-omni all keep their SageMaker
endpoint fixtures in the test file rather than a per-suite conftest.py. Shared
fixtures (image_uri, region, aws_session) still come from the root
test/conftest.py.

* Align openfold3 SageMaker test with pytorch pattern (env vars, no conftest)

Follow test/pytorch/integration/sagemaker convention (per kwanggg/reenable-pytorch-pr):
- config via env vars TEST_IMAGE_URI / SM_ROLE_ARN / AWS_REGION instead of the
  --image-uri pytest arg + aws_session fixture
- workflow builds SM_ROLE_ARN from CI_AWS_ACCOUNT_ID + SageMakerRole, single
  sagemaker-test job with check-image-exists graceful skip
- requirements pin sagemaker>=3.0.0 (SDK v3) like pytorch

The endpoint deploy/teardown stays a module-scoped fixture in the test file
(persistent endpoint needs cleanup, unlike pytorch's fire-and-forget training
jobs) — but no separate conftest.py.

* Harden OpenFold3 SageMaker async test (fix CI hang + orphan leak)

Root cause of the 2-hour CI hang: the SageMaker execution role
(AmazonSageMakerFullAccess) can read but NOT write s3://dlc-cicd-models — that
managed policy only grants S3 write to buckets named *sagemaker*. The endpoint
ran the prediction but could not write the async OUTPUT object, so the test's
3600s poll waited a full hour with no result and no failure object, then the
2h merge-conditions cap canceled the job (skipping fixture teardown -> orphaned
ml.g6.12xlarge endpoints).

Fixes:
- async input + output now use the account default sagemaker-<region>-<account>
  bucket (writable by the role); queries are staged there from dlc-cicd-models
  (read-only is permitted). QUERY_SRC_BUCKET/PREFIX are env-overridable.
- read FailureLocation + POLL_TIMEOUT=900s + 5s interval: a dropped/failed
  request now fails fast and legibly instead of a 1h hang.
- first-request retry + 30s settle after InService: SageMaker async can silently
  drop the first request submitted immediately after InService (the discrepancy
  vs the POC, where a human waits after warmup).
- cleanup-first _sweep_stale() in the fixture so a canceled run never strands or
  blocks the next one.

Validated end-to-end on the real DLC image (ml.g6.12xlarge, account
897880167187): all 3 tests pass. Timings: small 80s, large 206s, 4-concurrent
268s vs 206s single (ratio 1.30 < 2.0 -> GPU pool parallelizes). Endpoint
auto-deleted, no orphan.

* Simplify SageMaker async test: single bucket, preflight, no retry

- Query inputs + async output both use the account default sagemaker bucket
  (queries submitted directly, no cross-bucket staging from dlc-cicd-models).
- Add _preflight_s3() to fail fast (before the ~10-min deploy) if queries are
  missing or the output bucket is not writable.
- Drop the first-request retry; keep the 30s post-InService settle.
- Validated end-to-end on the DLC image (account 897880167187): all 3 pass.

* Expand SageMaker test module header to document env vars, GPU model, and S3 bucket rationale

* Restructure sagemaker-test workflow into preflight + endpoint-test (vllm/sglang pattern), drop skip notice

* TEMP(gamma-validation): release openfold3 to gamma (028) from PR pipeline

One-off gamma pre-prod validation push. force_release bypasses the active
us-west-2 change-control window (2026-06-23..27). Release-gate temporarily
relaxed to allow the .pr caller / non-main ref. REVERT all three files after
the gamma run completes.

- sagemaker.yml: release/force_release=true, environment=gamma
- openfold3.pipeline.yml: drop main-ref + autorelease/dispatch gate conditions
- openfold3.pr-amzn2023.yml: release=true

* Revert "TEMP(gamma-validation): release openfold3 to gamma (028) from PR pipeline"

This reverts commit df8a7d2.

* Enable network isolation in SageMaker async test

Validates the production no-outbound-network posture: weights are baked into
the image and inputs arrive via S3, so the default single-sequence inference
path needs no internet. MSA is off by default.

* Support customer precomputed MSA + add isolated MSA test

Handler: a chain may carry an optional precomputed_msa list of {name, content}
a3m/sto alignments. Names are validated against OF3 0.4.1's recognized keys
(unrecognized names are silently ignored by OF3 → single-sequence), then staged
to a per-request temp dir with main_msa_file_paths set. No network needed, so it
works under network isolation. Subprocess engine only; optional and backward
compatible.

Test: test_large_precomputed_msa reads the uploaded a3m fixture, derives the
query sequence from its first record, and submits both inline to the isolated
endpoint — validating the bring-your-own-MSA path end-to-end.

* Submit prebuilt large_msa.json instead of assembling MSA request at runtime

Mirrors small.json/large.json: the MSA request (622-res protein + inline
colabfold_main.a3m) is a committed fixture uploaded to the sagemaker bucket,
submitted directly. Drops the runtime read/parse/stage helpers.

* Reject use_msa_server under network isolation with an actionable error

The endpoint runs network-isolated, so the MSA server has no outbound internet
and a use_msa_server=true request hangs until the predict timeout (~55min,
holds a GPU) — confirmed by a manual repro. Reject it up front (before leasing
a GPU) with a clear error that points to the precomputed_msa format + supported
names. Env-overridable via OPENFOLD3_ALLOW_MSA_SERVER=1 for a non-isolated
deploy. Adds test_msa_server_rejected_under_isolation asserting fast rejection.

* Use committed small_msa_server.json fixture for the MSA-server rejection test

Replaces the inline _submit_inline helper: the negative test now submits a
committed query (small protein, use_msa_server=true) via the standard _invoke
path, matching small/large/large_msa. Uploaded by upload_openfold3_test_queries.sh.

* Enable openfold3 release now that CDK ECR repo + ReleaseLogic tag are deployed

Both companion CRs are deployed: the openfold3 ECR repo exists in the release
account and the openfold3 framework/tag entry is live in the release-logic S3
dist. Flip release:true so the scheduled autorelease (and dispatch) publishes
to the private registry. PR runs still never release (release-gate requires
main-ref + an autorelease/dispatch caller).

* Address review: guard release job on !failure/!cancelled; move all Dockerfile ARGs to config

(1) release job had a custom if: that overrides the default 'skip on failed
needs', so a failed/cancelled build or release-gate could still let release run.
Add !failure() && !cancelled() (matches PR #6307 fix).

(2) Add base_image + miniforge_version to the build: block so every Dockerfile
ARG is declared in the config (config = source of truth, Dockerfile ARG = fallback
default). All 6 ARGs (BASE_IMAGE, MINIFORGE_VERSION, OPENFOLD3_REF, CHECKPOINT,
DLC_MAJOR/MINOR_VERSION) are now config-driven.

* nit: order release if: as !failure && !cancelled && should-release for consistency

Author: Eren-Jeager123

.github/config/image/openfold3/sagemaker.yml

@@ -0,0 +1,33 @@
+image:
+  name: "openfold3-sagemaker"
+  description: "OpenFold3 protein structure prediction for SageMaker async inference"
+
+metadata:
+  framework: "openfold3"
+  framework_version: "0.4.1"
+  os_version: "amzn2023"
+  customer_type: "sagemaker"
+  arch_type: "x86"
+  device_type: "gpu"
+  job_type: "inference"
+  prod_image: "openfold3:0.4-cu130-amzn2023"
+
+build:
+  dockerfile: "docker/openfold3/Dockerfile"
+  target: "openfold3-sagemaker"
+  base_image: "public.ecr.aws/deep-learning-containers/base:devel-cu130-amzn2023"
+  python_version: "3.12"
+  cuda_version: "13.0.2"
+  miniforge_version: "25.11.0-1"
+  openfold3_ref: "0.4.1"
+  checkpoint: "of3-p2-155k.pt"
+  dlc_major_version: "1"
+  dlc_minor_version: "0"
+
+release:
+  release: true
+  force_release: false
+  public_registry: false
+  private_registry: true
+  enable_soci: true
+  environment: "production"

.github/release-schedule.yml

@@ -73,3 +73,7 @@ schedules:
   - cron: "00 22 * * 2,4"
     workflow: vllm-omni.autorelease-sagemaker.yml
     gpu: medium
+
+  - cron: "30 22 * * 2"
+    workflow: openfold3.autorelease-sagemaker-amzn2023.yml
+    gpu: heavy

.github/workflows/openfold3.autorelease-sagemaker-amzn2023.yml

@@ -0,0 +1,38 @@
+name: "Auto Release - OpenFold3 SageMaker AL2023"
+
+on:
+  schedule:
+    - cron: '30 22 * * 2'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+jobs:
+  discover:
+    runs-on: ubuntu-latest
+    outputs:
+      configs: ${{ steps.discover.outputs.configs }}
+    steps:
+      - uses: actions/checkout@v6
+      - id: discover
+        uses: ./.github/actions/discover-configs
+        with:
+          pattern: ".github/config/image/openfold3/sagemaker.yml"
+
+  pipeline:
+    needs: [discover]
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJson(needs.discover.outputs.configs) }}
+    uses: ./.github/workflows/openfold3.pipeline.yml
+    with:
+      config-file: ${{ matrix.config_file }}
+      tag-suffix: ${{ github.run_id }}
+      release: true
+    secrets: inherit

.github/workflows/openfold3.pipeline.yml

@@ -0,0 +1,147 @@
+name: "Pipeline - OpenFold3"
+
+on:
+  workflow_call:
+    inputs:
+      config-file:
+        description: "Path to image config YAML"
+        required: true
+        type: string
+      tag-suffix:
+        description: "CI tag suffix (run_id or pr-NUMBER)"
+        required: true
+        type: string
+      build:
+        description: "Whether to build the image (false = skip build, test against prod)"
+        required: false
+        type: boolean
+        default: true
+      run-sanity-test:
+        description: "Run sanity tests"
+        required: false
+        type: boolean
+        default: true
+      run-security-test:
+        description: "Run security tests"
+        required: false
+        type: boolean
+        default: false
+      run-sagemaker-test:
+        description: "Run SageMaker endpoint tests"
+        required: false
+        type: boolean
+        default: true
+      release:
+        description: "Run release job after tests pass"
+        required: false
+        type: boolean
+        default: false
+    outputs:
+      image-uri:
+        description: "Built image URI (empty if build was skipped)"
+        value: ${{ jobs.build.outputs.image-uri }}
+
+jobs:
+  ci-config:
+    runs-on: ubuntu-latest
+    outputs:
+      customer-type: ${{ steps.parse.outputs.customer-type }}
+    steps:
+      - uses: actions/checkout@v6
+      - id: parse
+        run: |
+          echo "customer-type=$(yq '.metadata.customer_type' '${{ inputs.config-file }}')" >> $GITHUB_OUTPUT
+
+  build:
+    if: ${{ inputs.build }}
+    concurrency:
+      group: ${{ github.workflow }}-build-${{ inputs.config-file }}-${{ github.ref }}
+      cancel-in-progress: true
+    runs-on:
+      - codebuild-runner-${{ github.run_id }}-${{ github.run_attempt }}
+        fleet:x86-build-runner
+        buildspec-override:true
+    timeout-minutes: 720
+    outputs:
+      image-uri: ${{ steps.build.outputs.image-uri }}
+    steps:
+      - uses: actions/checkout@v6
+      - id: build
+        uses: ./.github/actions/build-image
+        with:
+          config-file: ${{ inputs.config-file }}
+          aws-account-id: ${{ vars.CI_AWS_ACCOUNT_ID }}
+          aws-region: ${{ vars.AWS_REGION }}
+          tag-suffix: ${{ inputs.tag-suffix }}
+
+  sanity-test:
+    if: ${{ always() && !failure() && !cancelled() && inputs.run-sanity-test }}
+    needs: [build]
+    concurrency:
+      group: ${{ github.workflow }}-sanity-test-${{ inputs.config-file }}-${{ github.ref }}
+      cancel-in-progress: true
+    uses: ./.github/workflows/_reusable.sanity-tests.yml
+    with:
+      config-file: ${{ inputs.config-file }}
+      image-uri: ${{ needs.build.outputs.image-uri || '' }}
+
+  security-test:
+    if: ${{ always() && !failure() && !cancelled() && inputs.run-security-test }}
+    needs: [build]
+    concurrency:
+      group: ${{ github.workflow }}-security-test-${{ inputs.config-file }}-${{ github.ref }}
+      cancel-in-progress: true
+    uses: ./.github/workflows/_reusable.security-tests.yml
+    with:
+      config-file: ${{ inputs.config-file }}
+      image-uri: ${{ needs.build.outputs.image-uri || '' }}
+
+  sagemaker-tests:
+    if: ${{ always() && !failure() && !cancelled() && inputs.run-sagemaker-test && needs.ci-config.outputs.customer-type == 'sagemaker' }}
+    needs: [build, ci-config]
+    concurrency:
+      group: ${{ github.workflow }}-sagemaker-tests-${{ inputs.config-file }}-${{ github.ref }}
+      cancel-in-progress: false
+    uses: ./.github/workflows/openfold3.tests-sagemaker.yml
+    with:
+      config-file: ${{ inputs.config-file }}
+      image-uri: ${{ needs.build.outputs.image-uri || '' }}
+
+  release-gate:
+    if: >-
+      github.ref == 'refs/heads/main' &&
+      (contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
+      !contains(github.workflow_ref, '.pr') &&
+      inputs.release &&
+      !failure() && !cancelled() &&
+      needs.build.result == 'success'
+    needs: [build, sanity-test, security-test, sagemaker-tests]
+    runs-on: ubuntu-latest
+    outputs:
+      should-release: ${{ steps.check.outputs.should-release }}
+      release-spec: ${{ steps.spec.outputs.release-spec }}
+      environment: ${{ steps.check.outputs.environment }}
+    steps:
+      - uses: actions/checkout@v6
+      - id: check
+        run: |
+          SHOULD_RELEASE=$(yq '.release.release' "${{ inputs.config-file }}")
+          ENVIRONMENT=$(yq '.release.environment' "${{ inputs.config-file }}")
+          echo "should-release=${SHOULD_RELEASE}" >> $GITHUB_OUTPUT
+          echo "environment=${ENVIRONMENT}" >> $GITHUB_OUTPUT
+      - id: spec
+        if: steps.check.outputs.should-release == 'true'
+        uses: ./.github/actions/generate-release-spec
+        with:
+          config-file: ${{ inputs.config-file }}
+
+  release:
+    if: ${{ !failure() && !cancelled() && needs.release-gate.outputs.should-release == 'true' }}
+    needs: [build, release-gate]
+    uses: ./.github/workflows/_reusable.release-image.yml
+    with:
+      source-image-uri: ${{ needs.build.outputs.image-uri }}
+      release-spec: ${{ needs.release-gate.outputs.release-spec }}
+      environment: ${{ needs.release-gate.outputs.environment }}
+      aws-region: ${{ vars.AWS_REGION }}
+    secrets: inherit

.github/workflows/openfold3.pr-amzn2023.yml

@@ -0,0 +1,84 @@
+name: "PR - OpenFold3 AL2023"
+
+on:
+  workflow_dispatch:
+  pull_request:
+    branches: [main]
+    types: [opened, reopened, synchronize]
+    paths:
+      - ".github/config/image/openfold3/*.yml"
+      - ".github/workflows/openfold3.pipeline.yml"
+      - ".github/workflows/openfold3.pr-amzn2023.yml"
+      - ".github/workflows/openfold3.tests-sagemaker.yml"
+      - "docker/openfold3/Dockerfile"
+      - "scripts/docker/common/**"
+      - "scripts/docker/openfold3/**"
+      - "test/sanity/**"
+      - "test/openfold3/sagemaker/**"
+      - "!docs/**"
+
+permissions:
+  contents: read
+  pull-requests: read
+
+jobs:
+  gatekeeper:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: ${{ github.event.pull_request.base.sha }}
+          fetch-depth: 1
+      - uses: ./.github/actions/pr-permission-gate
+
+  check-changes:
+    needs: [gatekeeper]
+    runs-on: ubuntu-latest
+    outputs:
+      build-change: ${{ steps.changes.outputs.build-change }}
+      sanity-test-change: ${{ steps.changes.outputs.sanity-test-change }}
+      sagemaker-test-change: ${{ steps.changes.outputs.sagemaker-test-change }}
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dorny/paths-filter@v4
+        id: changes
+        with:
+          filters: |
+            build-change:
+              - ".github/config/image/openfold3/*.yml"
+              - "docker/openfold3/Dockerfile"
+              - "scripts/docker/common/**"
+              - "scripts/docker/openfold3/**"
+            sanity-test-change:
+              - "test/sanity/**"
+            sagemaker-test-change:
+              - "test/openfold3/sagemaker/**"
+
+  discover:
+    needs: [gatekeeper]
+    runs-on: ubuntu-latest
+    outputs:
+      configs: ${{ steps.discover.outputs.configs }}
+    steps:
+      - uses: actions/checkout@v6
+      - id: discover
+        uses: ./.github/actions/discover-configs
+        with:
+          pattern: ".github/config/image/openfold3/*.yml"
+
+  pipeline:
+    needs: [check-changes, discover]
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJson(needs.discover.outputs.configs) }}
+    uses: ./.github/workflows/openfold3.pipeline.yml
+    with:
+      config-file: ${{ matrix.config_file }}
+      tag-suffix: pr-${{ github.event.pull_request.number }}
+      build: ${{ needs.check-changes.outputs.build-change == 'true' }}
+      run-sanity-test: ${{ needs.check-changes.outputs.build-change == 'true' || needs.check-changes.outputs.sanity-test-change == 'true' }}
+      run-security-test: ${{ needs.check-changes.outputs.build-change == 'true' }}
+      run-sagemaker-test: ${{ needs.check-changes.outputs.build-change == 'true' || needs.check-changes.outputs.sagemaker-test-change == 'true' }}
+      release: false
+    secrets: inherit

.github/workflows/openfold3.tests-sagemaker.yml

@@ -0,0 +1,78 @@
+name: Reusable OpenFold3 SageMaker Tests
+
+on:
+  workflow_call:
+    inputs:
+      config-file:
+        description: "Path to image config YAML"
+        required: true
+        type: string
+      image-uri:
+        description: "Image URI to test. If empty, derives prod URI from config."
+        required: false
+        type: string
+        default: ""
+
+permissions:
+  contents: read
+
+jobs:
+  preflight:
+    runs-on:
+      - codebuild-runner-${{ github.run_id }}-${{ github.run_attempt }}
+        fleet:default-runner
+        buildspec-override:true
+    outputs:
+      image-uri: ${{ steps.image.outputs.image-uri }}
+      aws-account-id: ${{ steps.image.outputs.aws-account-id }}
+      image-exists: ${{ steps.check.outputs.exists }}
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Resolve image URI
+        id: image
+        uses: ./.github/actions/resolve-image-uri
+        with:
+          image-uri: ${{ inputs.image-uri }}
+          ci-aws-account-id: ${{ vars.CI_AWS_ACCOUNT_ID }}
+          prod-aws-account-id: ${{ vars.PROD_AWS_ACCOUNT_ID }}
+          aws-region: ${{ vars.AWS_REGION }}
+          config-file: ${{ inputs.config-file }}
+
+      - name: Check image exists
+        id: check
+        uses: ./.github/actions/check-image-exists
+        with:
+          image-uri: ${{ steps.image.outputs.image-uri }}
+
+  endpoint-test:
+    if: ${{ needs.preflight.outputs.image-exists == 'true' }}
+    needs: [preflight]
+    runs-on:
+      - codebuild-runner-${{ github.run_id }}-${{ github.run_attempt }}
+        fleet:default-runner
+        buildspec-override:true
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: ECR login
+        uses: ./.github/actions/ecr-authenticate
+        with:
+          aws-account-id: ${{ needs.preflight.outputs.aws-account-id }}
+          aws-region: ${{ vars.AWS_REGION }}
+          image-uri: ${{ needs.preflight.outputs.image-uri }}
+
+      - name: Run SageMaker async endpoint tests
+        env:
+          TEST_IMAGE_URI: ${{ needs.preflight.outputs.image-uri }}
+          SM_ROLE_ARN: arn:aws:iam::${{ vars.CI_AWS_ACCOUNT_ID }}:role/SageMakerRole
+          AWS_REGION: ${{ vars.AWS_REGION }}
+        run: |
+          uv venv --python 3.12
+          source .venv/bin/activate
+          uv pip install -r test/requirements.txt
+          uv pip install -r test/openfold3/sagemaker/requirements.txt
+          cd test/
+          PYTHONPATH=$(pwd):$PYTHONPATH python3 -m pytest -v --tb=short -rA --log-cli-level=INFO \
+            openfold3/sagemaker/test_async_endpoint.py