RHOAIENG-64892: CVE-2026-48710 rhoai/odh-pipeline-runtime-pytorch-cuda-py312-rhel9: Starlette: Security restriction bypass via malformed HTTP Host header [rhoai-3.3]

.agents/skills/README.md

@@ -0,0 +1,8 @@
+# Project Agent Skills
+
+This directory holds **Agent Skills** for this repository — subfolders,
+each with a `SKILL.md`, following the [Agent Skills](https://agentskills.io/) convention.
+
+Clients discover skills by scanning standard paths; see **where to scan**
+(project vs user scope):
+https://agentskills.io/client-implementation/adding-skills-support#where-to-scan

.gitignore

@@ -88,6 +88,7 @@ target/
 
 # Jupyter Notebook
 .ipynb_checkpoints
+CLAUDE.local.md
 
 # IPython
 profile_default/

ARCHITECTURE.md

@@ -64,7 +64,7 @@ The `runtimes/` directory mirrors the same flavor structure (minimal, datascienc
 | `base-images/` | CUDA and ROCm GPU-accelerated base image definitions |
 | `dependencies/` | Shared dependency constraints (CVE pinning) and meta packages for common dependency groups |
 | `examples/` | Example JupyterLab notebooks for validating workbench functionality |
-| `docs/adr/` | Architecture Decision Records |
+| `docs/architecture/decisions/` | Architecture Decision Records |
 
 ## Build system
 
@@ -76,9 +76,18 @@ make all-images                              # build everything
 make test                                    # run quick static tests (pytest + lint)
 ```
 
-The build system supports two modes:
-- **ODH mode** (default): `KONFLUX=no`, uses standard Dockerfiles
-- **RHOAI/Konflux mode**: `KONFLUX=yes`, uses `Dockerfile.konflux.*` variants with prefetched dependencies
+The `KONFLUX` Makefile variable selects the **product variant**, not whether the build
+runs on Konflux/Tekton:
+
+- **ODH mode** (default: `KONFLUX=no` or unset): uses `build-args/<variant>.conf`
+  and `manifests/odh/`
+- **RHOAI mode** (`KONFLUX=yes`): uses `build-args/konflux.<variant>.conf`
+  and `manifests/rhoai/`
+
+Since RHAIENG-4516, `Dockerfile.<variant>` paths and `Dockerfile.konflux.<variant>`
+paths resolve to the same content, so the meaningful difference is the selected
+build-args file and manifest set rather than a separate Dockerfile implementation.
+Both variants can be built locally or on Konflux/Tekton.
 
 ## Testing layers
 

CONTRIBUTING.md

@@ -103,11 +103,72 @@ for details.
 
 ### Testing your PR locally
 
-- Test the changes locally by running `make jupyter-${NOTEBOOK_NAME}-ubi9-python-3.12` from the terminal.
+The fuller test catalog is documented in [docs/agents/testing.md](docs/agents/testing.md)
+(tracked by [issue #3174](https://github.com/opendatahub-io/notebooks/issues/3174)).
+
+#### Environment setup
+
+- Use `uv` for local development:
+  ```bash
+  uv venv --python "$(which python3.14)"
+  uv sync --locked
+  ```
+- On macOS, install Homebrew GNU Make 4.x and put `gnubin` on `PATH` so `make`
+  resolves to the same binary used on Linux:
+  ```bash
+  brew install make
+  PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
+  ```
+- After dependency changes, regenerate locks with `make refresh-lock-files`.
+
+#### Common local commands
+
+```bash
+make test
+make test-unit
+make test-integration PYTEST_ARGS="--image=<image>"
+make jupyter-${NOTEBOOK_NAME}-ubi9-python-3.12
+make test-${NOTEBOOK_NAME}
+```
+
+#### Local gotchas
+
+- Run tests from a checkout whose directory name is exactly `notebooks`.
+  Hyphenated worktree paths such as `/tmp/pr-2293-ci` break
+  `test_image_pyprojects_version_alignment`.
+- If static tests fail in a surprising way, retry from a clean clone first. Extra
+  top-level files and directories such as `.cursor-tmp-*` can interfere with repo-wide
+  assertions.
+- For filesystem-heavy unit tests, prefer the `pyfakefs` `fs` fixture over `tmp_path`.
+- When renaming image labels or similar CI metadata, update only the current `-n`
+  entries in `ci/check-params-env.sh` and `ci/expected-image-metadata.yaml`. Historical
+  entries intentionally keep old names.
+
+#### ODH vs RHOAI local builds
+
+`KONFLUX` selects the product variant, not whether the build runs on Konflux/Tekton.
+Both variants are built on Konflux in CI; the variable changes which `build-args/*.conf`
+and manifest tree the Makefile uses locally.
+
+| `KONFLUX` value | Product variant | Build args | Manifests |
+|-----------------|-----------------|------------|-----------|
+| unset or `no` | ODH (midstream default) | `build-args/<variant>.conf` | `manifests/odh/` |
+| `yes` | RHOAI downstream | `build-args/konflux.<variant>.conf` | `manifests/rhoai/` |
+
+Keep the same `KONFLUX` value for both the build step and the matching `make test-*`
+step so the imagestream manifests and expected package versions line up.
+
+For deeper references:
+
+- Konflux and Tekton details: [`docs/konflux.md`](docs/konflux.md)
+- Subscribed/AIPCC local builds: [`docs/subscribed-builds.md`](docs/subscribed-builds.md)
+- Hermetic build internals: [`docs/hermetic-guide.md`](docs/hermetic-guide.md)
+- CI failure ownership and systems: [`docs/ci.md`](docs/ci.md)
+- CVE-driven lockfile work: [`docs/cves/python.md`](docs/cves/python.md) and [`docs/cves/nodejs.md`](docs/cves/nodejs.md)
 
 ### Working with RHDS and ODH Repositories
 
-When contributing to notebook-related changes in the Red Hat Data Science (RHDS) ecosystem, it's important to understand the repository structure and contribution workflow:
+When contributing to notebook-related changes in the Red Hat Data Services (RHDS) ecosystem, it's important to understand the repository structure and contribution workflow:
 
 #### Repository Responsibilities
 
@@ -121,6 +182,10 @@ When contributing to notebook-related changes in the Red Hat Data Science (RHDS)
 - **Konflux Integration**: This repository contains Konflux-specific build configurations and pipeline definitions
 - **Automated Sync**: Receives updates from the ODH notebooks repository automatically
 
+In this terminology, ODH is the **midstream** repo and RHOAI/RHDS is the **downstream**
+product repo. When shared files need one convention, follow the Konflux/RHOAI naming and
+behavior that the downstream automation expects.
+
 #### Contribution Workflow
 
 1. **For General Changes**:
@@ -134,8 +199,9 @@ When contributing to notebook-related changes in the Red Hat Data Science (RHDS)
    - Changes to these files flow to RHOAI upcoming release and need careful coordination
 
 3. **For Tekton Pipeline Changes**:
-   - Modify Tekton pipelines in the central repository as specified in the specific README documentation
-   - Follow the centralized pipeline management guidelines
+   - ODH `.tekton/` changes are made in this repository
+   - RHDS PR PipelineRun changes are authored in `konflux-central`, then synced into the downstream notebooks repo
+   - Follow the repository-specific Konflux documentation before editing pipeline definitions
 
 #### Best Practices
 

README.md

@@ -14,9 +14,9 @@ For a deeper understanding of the architecture underlying this repository, pleas
 ### Prerequisites
 Make sure the following tools are installed in your environment:
  - podman/docker
- - python
- - pipenv
- - make (on macOS install/use: gmake)
+ - python 3.14
+ - [uv](https://docs.astral.sh/uv/)
+ - make (on macOS: `brew install make` and add `/opt/homebrew/opt/make/libexec/gnubin` to `PATH`)
  - curl
  - git, git-lfs (for `*.vsix` files)
 
@@ -56,11 +56,11 @@ Use podman/docker to execute the workbench images as container.
 podman  run -it -p  8888:8888  quay.io/opendatahub/workbench-images:jupyter-minimal-ubi9-python-3.9-2024a-20240317-6f4c36b
 ```
 
-### Pipfile.lock Generation
+### Lock file generation
 
-Users can update Pipfile.lock files using the [piplock-renewal.yaml](https://github.com/opendatahub-io/notebooks/blob/main/.github/workflows/piplock-renewal.yaml) GitHub Action. This workflow enables users to specify a target branch for updating and automerging Pipfile.lock files, select the desired Python version for the update as well as to choose whether to include optional directories in the update process. After the action completes, the updated files can be retrieved with a simple git pull.
-
-Note: To ensure the GitHub Action runs successfully, users must add a `GH_ACCESS_TOKEN` secret in their fork.
+Image lock files (`pylock.toml` / `uv.lock.d/`) are regenerated with `make refresh-lock-files`.
+For details on the dual `uv` version policy (dev vs image locks), see the
+[Deploy & Test](#deploy--test) section below.
 
 ### Deploy & Test
 
@@ -100,7 +100,9 @@ system uv does not already match that exact version.
 
 #### Running tests
 
-By completing configuration in the previous section, you can run tests using the following targets (use `gmake` instead of `make` on macOS):
+For the full test catalog (types, markers, CI parity), see [docs/agents/testing.md](docs/agents/testing.md).
+
+By completing configuration in the previous section, you can run tests using the following targets:
 
 ```bash
 make test              # Quick static tests (pytest + Dockerfile alignment check)
@@ -246,7 +248,7 @@ Whether you're fixing bugs, adding new notebooks, or improving documentation, yo
 
 ### For AI Agents
 
-If you're an AI agent working with this repository, please refer to our [Agents Guide](Agents.md) for comprehensive instructions on project structure, development workflows, and best practices.
+If you're an AI agent working with this repository, please refer to our [Agents Guide](AGENTS.md) for comprehensive instructions on project structure, development workflows, and best practices.
 
 ## Acknowledgments
 

dependencies/cve-constraints.txt

@@ -22,5 +22,5 @@ pillow>=12.2.0
 jupyterlab>=4.5.7
 # RHAIENG-3644: CVE-2026-0846 NLTK: Arbitrary file read via improper path validation in `filestring()` function
 nltk>=3.9.3
-# RHAIENG-5355: CVE-2026-48710 Starlette: Security restriction bypass via malformed HTTP Host header
+# RHOAIENG-64892: CVE-2026-48710 starlette security bypass via malformed HTTP Host header
 starlette>=1.0.1

docs/agents/testing.md

@@ -0,0 +1,134 @@
+# Testing Guide
+
+This document is the test catalog for the notebooks repository. It describes what
+tests exist, where they live, how to run them, and how they map to CI.
+
+For operational gotchas (worktree naming, pyfakefs, KONFLUX matching, CI `-n` metadata),
+see [CONTRIBUTING.md](../../CONTRIBUTING.md).
+
+## Setup
+
+```bash
+uv venv --python "$(which python3.14)"
+uv sync --locked
+```
+
+On macOS, install Homebrew GNU Make so `make` resolves to GNU Make 4.x:
+
+```bash
+brew install make
+PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
+```
+
+## Test targets
+
+| Target | What it runs | Requires |
+|--------|-------------|----------|
+| `make test` | Static tests (pytest) + Dockerfile alignment check | Nothing |
+| `make test-unit` | Python unit tests + doctests + Go tests | Nothing (Go auto-downloads) |
+| `make test-integration PYTEST_ARGS="--image=<img>"` | Container integration tests | Podman/Docker |
+| `make test-<notebook>` | Notebook smoke test via papermill | kubectl + deployed workbench |
+
+### Deploying for notebook smoke tests
+
+`make test-<notebook>` requires a deployed workbench on OpenShift. For the full
+deploy/test/undeploy cycle, see [README.md § Notebooks](../../README.md#notebooks)
+and [README.md § Runtimes](../../README.md#runtimes).
+
+## Test types and locations
+
+| Type | Location | Naming | Organization |
+|------|----------|--------|--------------|
+| Static / manifest | `tests/test_*.py` | Module-level functions | Group with `subtests` |
+| Unit | `tests/unit/` | `test_*.py` mirroring source layout | Module-level functions |
+| Container integration | `tests/containers/` | `*_test.py` (class-based) | `Test*` classes with fixtures |
+| Browser E2E | `tests/browser/tests/` | `*.spec.ts` | Playwright + pnpm |
+| Go | `scripts/buildinputs/` | Standard Go test files | `make test-unit` runs these |
+| K8s notebook smoke | `scripts/` | Shell scripts | `make test-<notebook>` |
+
+### Collection behavior
+
+- `tests/containers/` is **excluded from default collection** via `collect_ignore` in
+  `tests/conftest.py`. Run container tests explicitly with `pytest tests/containers --image=<img>`.
+- Default `make test` collects from `tests/`, `ntb/`, and `ci/` (doctests).
+- `--strict-markers` is on — unregistered markers fail the run.
+
+## Markers
+
+All markers must be registered in `pytest.ini`:
+
+| Marker | Meaning | Excluded from default run |
+|--------|---------|--------------------------|
+| `openshift` | Needs live OpenShift cluster | Yes |
+| `cuda` | Needs NVIDIA GPU | Yes |
+| `rocm` | Needs AMD GPU | Yes |
+| `manifest_validation` | Slow registry/skopeo checks | Yes |
+| `buildonlytest` | Runs inside docker build only | Yes (different filter) |
+| `codeserver` | Code-Server workbench specific | No |
+
+## CI parity
+
+Local `make test` + `make test-unit` covers the `pytest-tests` job in
+`.github/workflows/code-quality.yaml`. Other CI checks that are **not yet**
+exposed as `make` targets:
+
+- yamllint (inline in workflow)
+- hadolint (inline in workflow)
+- gotestsum (inline in workflow)
+- prek (inline in workflow)
+
+Closing this gap (moving inline CI logic into Makefile targets) is tracked in
+[#3174](https://github.com/opendatahub-io/notebooks/issues/3174).
+
+## Frameworks and tools
+
+| Tool | Purpose |
+|------|---------|
+| pytest | Test runner for all Python tests |
+| pytest-subtests | Granular sub-assertions within a single test |
+| pytest-cov | Coverage (XML + terminal) |
+| allure-pytest | Issue tracking + step decoration |
+| testcontainers | Container lifecycle for integration tests |
+| pyfakefs | Filesystem mocking for unit tests |
+| Playwright | Browser tests (TypeScript) |
+| papermill | Notebook execution verification |
+
+## Troubleshooting
+
+- **Container tests hang:** Ensure the container runtime (podman/docker) is running.
+  On Linux: `systemctl --user start podman.service`.
+- **Dependency conflicts after lock regen:** Run `make refresh-lock-files` and check
+  for "unsatisfiable" errors. See [docs/cves/python.md](../cves/python.md) for
+  constraint resolution.
+- **`make test` fails on stray files:** Extra top-level directories (`.cursor-tmp-*`)
+  break repo-wide assertions. Clean clone or `git clean -fdx` the offending paths.
+- **Security scanning:** Weekly Quay vulnerability reports are generated by
+  `ci/security-scan/quay_security_analysis.py` (triggered via `.github/workflows/sec-scan.yml`).
+
+## External test suites
+
+The images built by this repo are also tested by other projects:
+
+| Suite | Framework | What it tests |
+|-------|-----------|---------------|
+| [odh-dashboard](https://github.com/opendatahub-io/odh-dashboard/tree/main/packages/cypress/cypress/tests/e2e/dataScienceProjects/workbenches) | Cypress | Workbench CRUD, image selection, RBAC via ODH dashboard |
+| [ods-ci](https://github.com/red-hat-data-services/ods-ci/tree/master/ods_ci/tests/Tests/0500__ide) | Robot Framework | GPU/CUDA validation, Elyra pipelines, plugin consistency |
+| [opendatahub-tests](https://github.com/opendatahub-io/opendatahub-tests/tree/main/tests/workbenches) | Pytest | ImageStream health, Notebook CR spawning, package availability |
+
+## Running images locally
+
+Build with a predictable tag and push disabled, then run:
+
+```bash
+# JupyterLab (port 8888)
+make jupyter-minimal-ubi9-python-3.12 IMAGE_TAG=latest PUSH_IMAGES=no
+podman run -it -p 8888:8888 quay.io/opendatahub/workbench-images:jupyter-minimal-ubi9-python-3.12-latest
+
+# Code-Server (port 8787)
+make codeserver-ubi9-python-3.12 IMAGE_TAG=latest PUSH_IMAGES=no
+podman run -it -p 8787:8787 quay.io/opendatahub/workbench-images:codeserver-ubi9-python-3.12-latest
+```
+
+For published image references, see the [README.md Image Inventory](../../README.md#image-inventory-list).
+Override the entrypoint to inspect an image without starting the workbench
+(see [AGENTS.md](../../AGENTS.md) operational notes).