Skip to content

DESIGN.md

Latest commit

 

History

History
237 lines (198 loc) · 11 KB

DESIGN.md

File metadata and controls

237 lines (198 loc) · 11 KB
Raw

docker-variant: v1 Design

This document freezes the concrete contracts for v1. The rationale and roadmap live in PLAN.md. Each non-obvious choice is recorded as an ADR at the bottom.

1. Terminology

  • Base version: a repository tag chosen by the publisher as the release key, e.g. 2.1.0 in myimg:2.1.0. Any tag string is allowed as long as appending -<label> keeps it a valid OCI tag.
  • Variant label: a short slug identifying one variant of a base version. Same rule as PEP 817: ^[0-9a-z._]{1,16}$. The label null is reserved for the null variant (zero properties).
  • Variant tag: <base-version>-<label>, e.g. 2.1.0-cu128.
  • Variant property: a namespace :: feature :: value triple. Namespaces and features match ^[a-z0-9_]+$; values additionally allow dots (^[a-z0-9_.]+$) since version-like values such as 12.8 are the primary use case. A feature may carry multiple values (multi-value semantics: the variant is usable if any of its values is supported by the system).
  • Variant index: the per-(repository, base version) JSON document mapping labels to property sets and manifest digests — the analogue of PEP 817's {name}-{version}-variants.json.

2. Image labeling schema

Variant metadata is carried in image config labels (Dockerfile LABEL / docker build --label), so plain docker build is the only authoring tool required:

Label key Value Required
dev.pep817.variant-label the variant label, e.g. cu128 yes (on variant images)
dev.pep817.variant.<namespace>.<feature> comma-separated values, e.g. 12.8 or 12.8,12.9 one per property

Example Dockerfile fragment:

LABEL dev.pep817.variant-label="cu128" \
      dev.pep817.variant.nvidia.cuda_version_lower_bound="12.8"

The null variant sets only dev.pep817.variant-label="null".

The base version is derived from the tag at publish time: pushing myimg:2.1.0-cu128 with variant-label=cu128 yields base version 2.1.0. The tooling validates that the tag suffix equals the image's variant label and fails otherwise (guards against tag/label drift).

3. Variant index

3.1 Storage (ADR-3)

The index is an OCI artifact in the same repository, addressed by the tag convention:

<repository>:<base-version>-variants        e.g. myimg:2.1.0-variants
  • Manifest artifactType: application/vnd.pep817.container-variants.v1+json
  • Single layer blob, media type same as artifactType, containing the JSON.
  • variants is reserved as a label (it matches the label regex), i.e. no variant may be named variants; tooling rejects it.

OCI 1.1 referrers-based attachment is deferred (see ADR-3).

3.2 Schema (schema-version "1")

{
  "schema-version": "1",
  "repository": "registry.example.com/myimg",
  "version": "2.1.0",
  "default-priorities": {
    "namespace": ["nvidia", "x86_64", "aarch64"],
    "feature": {
      "nvidia": ["cuda_version_lower_bound"]
    },
    "property": {
      "nvidia": {
        "cuda_version_lower_bound": ["12.8", "12.0"]
      }
    }
  },
  "variants": {
    "cu128": {
      "properties": {
        "nvidia": { "cuda_version_lower_bound": ["12.8"] }
      },
      "tag": "2.1.0-cu128",
      "digest": "sha256:…"
    },
    "null": {
      "properties": {},
      "tag": "2.1.0-null",
      "digest": "sha256:…"
    }
  }
}

Notes:

  • properties nests namespace → feature → ordered value list, mirroring PEP 817's variants.json.
  • digest is the digest of the manifest (or manifest list) the variant tag pointed to at index time. Selection always pulls by digest, never by the variant tag (immune to tag mutation; signable later).
  • default-priorities is optional at every level; resolution falls back to deterministic defaults (§4).
  • Unlike PEP 817 there is no providers section in v1: detection providers are compiled into the client (ADR-5).

4. Selection algorithm

Given the index and the system properties (namespace → feature → ordered list of supported values, most-preferred first, as reported by detection providers):

  1. Compatibility filter: a variant is compatible iff for every property (ns, feature, values) it declares, values ∩ system[ns][feature] ≠ ∅. The null variant (no properties) is always compatible.
  2. Sort key per property: (namespace_rank, feature_rank, value_rank):
    • namespace_rank: position in default-priorities.namespace; unlisted namespaces rank after listed ones, ordered lexicographically.
    • feature_rank: position in default-priorities.feature[ns]; same fallback rule.
    • value_rank: minimum over the values that intersect the system set, of: position in default-priorities.property[ns][feature] if listed, else position in the system's own preference-ordered value list, else lexicographic fallback. Index priorities always outrank both fallbacks.
  3. Variant ordering: each variant's property sort keys are sorted ascending and compared lexicographically; where one list is a strict prefix of another, the longer list (more matched properties) wins. Final tie-break: variant label, lexicographic. The null variant therefore naturally sorts after every property-bearing compatible variant.
  4. Fallback chain (client): best compatible variant → null variant if present → the plain base tag <repository>:<base-version>.

5. Client (Docker CLI plugin)

Binary docker-variant, installed to ~/.docker/cli-plugins/, exposing docker variant <subcommand>:

Subcommand Behavior
detect Run providers, print system properties (text or --format json)
pull <repo>:<version> Fetch index → detect → rank → docker pull <repo>@<digest> → tag result as both <repo>:<version> and <repo>:<version>-<label> locally
push <repo>:<version>-<label> docker push the tag, read labels via docker image inspect, upsert the variant index artifact
list <repo>:<version> Print the index as a table (label, properties, digest, compatible-with-this-host marker)
inspect <repo>:<version> Print raw index JSON
index update <repo>:<version> Rebuild the index by scanning registry tags + labels (same builder the proxy uses)

Common flags: --plain-http (also auto-enabled for localhost/127.0.0.1), --properties-file <json> (override detection — used by tests/demo), --dry-run on pull (print the decision, don't pull).

Engine interaction is by shelling out to the docker binary; registry interaction uses oras-go v2 with Docker's credential store (ADR-4).

6. Registry side

  • Phase A (no registry changes): docker variant push / index update maintain the index artifact client-side. Works with any V2 registry.

  • Phase B (variant-proxy): a stateless reverse proxy in front of a registry. All /v2/ traffic passes through, plus:

    GET /v2/<name>/_variants/<version>   →  computed index JSON (200)
                                        404 if no variant tags exist

    The index is computed on demand by listing tags matching <version>-<label-pattern>, reading each config's labels. With the proxy in place, plain docker push of labeled images is sufficient — no client-side index maintenance. The client tries _variants first, then falls back to the -variants tag.

7. Detection providers (v1, compiled-in)

Namespace Features Source
x86_64 level = v4/v3/v2/v1 (preference-ordered, all supported levels emitted); plus <cpu-flag> = on per supported flag in the WheelNext x86_64 provider feature list (avx2, sha_ni, …) CPUID via golang.org/x/sys/cpu; /proc/cpuinfo flags (Linux)
aarch64 version = 9.0a/8.5a/8.4a/…/8a (preference-ordered, all supported versions emitted; WheelNext aarch64 provider value scheme); plus <cpu-flag> = on per supported flag in the WheelNext aarch64 provider feature list (sve2, i8mm, …) HWCAP via golang.org/x/sys/cpu; /proc/cpuinfo Features (Linux)
nvidia cuda_version_lower_bound = all known CUDA versions ≤ detected, descending nvidia-smi header parse
(mock) anything --properties-file / DOCKER_VARIANT_PROPERTIES_FILE

cuda_version_lower_bound semantics: the provider expands the "lower bound" predicate into the explicit list of satisfiable values from a known CUDA release table, so the generic intersection rule of §4 applies unchanged.

8. Testing strategy

  • pkg/variant: table-driven unit tests for label/property validation, JSON round-trips, compatibility, and ordering (fixtures covering the scenarios in PEP 817's examples: preferred-value selection, namespace priority, multi-value intersection, null fallback).
  • pkg/providers: parsing tests with canned nvidia-smi/CPUID inputs; the mock provider used everywhere else.
  • e2e (e2e/): script + CI job against a disposable local registry:2 container — build labeled fixture images, variant push, variant pull under several mocked hardware profiles, assert the selected digests; proxy variant of the same flow.
  • demo/: human-runnable walkthrough (not asserted in CI) showing the same flow with readable output.

ADRs

  • ADR-1 — Client is a CLI plugin, not a moby fork. Docker CLI plugins cannot shadow built-in commands, so the UX is docker variant pull. A fork would give native docker pull but costs a permanent rebase treadmill and a patched-engine adoption barrier. All logic lives in importable packages so a future fork/containerd port reuses it.
  • ADR-2 — Properties live in image config labels. LABEL is authorable from any Dockerfile with no new tooling, survives docker build/push unchanged, and is inspectable locally. Manifest annotations would be cheaper for server-side scanning (no config blob fetch) but cannot be set by plain docker build; may be added as a mirror later.
  • ADR-3 — Index artifact via tag convention, not OCI 1.1 referrers. -variants tags work on every V2 registry incl. Docker Hub and registry:2; referrers require OCI 1.1 servers and would fragment the v1 story. Race window on concurrent push is accepted for v1 (documented); index update reconciles. Referrers attachment is a planned follow-up.
  • ADR-4 — Engine ops shell out to docker; registry ops use oras-go. Shelling out inherits contexts, credential helpers, and proxies exactly as the user's CLI has them, and avoids the heavy moby client dependency for pull/push/tag/inspect. oras-go v2 is needed anyway for artifact push and tag listing.
  • ADR-5 — No pluggable detection providers in v1. PEP 817's main flagged risk is running third-party detection code at install time. Compiled-in providers + a mock file cover the demo and real NVIDIA/CPU cases; an exec-based provider protocol is future work behind an explicit opt-in.
  • ADR-6 — pull tags the result as both the base tag and the variant tag. The base tag matches what the user asked for (scripts keep working); the variant tag preserves visibility of what was actually selected.