Docker composer + GH action coded as wrapper for CU repo forks

[motyc]

https://github.com/ARUP-CAS/atrium-page-classification
https://github.com/ARUP-CAS/atrium-alto-postprocess
https://github.com/ARUP-CAS/atrium-nlp-enrich
https://github.com/ARUP-CAS/atrium-translator

[K4TEL]

Strategy by Opus:

summary

The repos are already pre-wired for this: the shared atrium_paradata.py provenance logger reads ATRIUM_RUNNER_IMAGE, ATRIUM_RUNNER_REPO, ATRIUM_RUNNER_REF env vars and records them in every output JSON, with the documented example ghcr.io/ufal/atrium-translator:v0.5.2. The maintainers clearly intended GHCR-published, self-identifying containers — this plan delivers exactly that.

Outcome: anyone can docker compose run <tool> over a mounted data dir with zero manual setup; tagged releases auto-publish a provenance-stamped image to GHCR; PRs are gated by the existing fast test suite.

Decisions (confirmed with user)

• Scope: the 4 component repos only. atrium-project gets just a short strategy/index doc — no umbrella orchestration compose.
• GH Action (no preference given → recommended): build & publish to GHCR on tag/release, feeding the paradata ATRIUM_RUNNER_* hook; pytest -m "not slow" gate on PRs.
• Run mode: batch CLI (one-shot, mounted in→out, then exit) by default; an optional api Compose profile only for the two repos that ship a FastAPI service/ (alto-postprocess, page-classification). translator & nlp-enrich are batch-only (confirmed: no service/ dir).
• Rollout: pilot atrium-translator (lightest — no GPU, no API), verify end-to-end, then replicate the proven template to the other three.
• Branch: all work on claude/determined-hypatia-fi57y4 (already exists in every repo; remote origin → ufal/<repo>).

Reuse — existing things to wire into, not reinvent

• Provenance contract: atrium_paradata.py (every repo) reads ATRIUM_RUNNER_IMAGE/REPO/REF (translator lines 54-58, 144-148, 253-255). Dockerfile build-ARGs → ENV must set these; nothing in app code changes.
• Version source: para_config.txt [tool] version (e.g. page-classification v1.1.1-beta) — drives image tags.
• Test gate: existing pytest.ini with the slow marker in all repos → CI runs pytest -m "not slow" (network/GPU-free).
• Setup logic to translate into Dockerfile steps (not re-run at runtime): atrium-alto-postprocess/setup_api_server.sh (LayoutReader v3/ sparse-checkout + lid.176.bin download), atrium-page-classification/setup_api_service.sh.
• Existing .gitignores (present in 3 of 4; absent in alto-postprocess) inform the .dockerignore.

---

The templated pattern (identical vs parameterized)

Every repo gets the same 4 files: Dockerfile, docker-compose.yml, .dockerignore, .github/workflows/docker.yml. The two service repos also get docker-compose.gpu.yml and an api profile.

Identical across all four: the ARG ATRIUM_RUNNER_* → ENV block; PYTHONUNBUFFERED/
PYTHONDONTWRITEBYTECODE/pip env; HF_HOME=/cache/huggingface; non-root user atrium (uid 10001) owning /app /cache /data; layer order (copy requirements → pip install → copy source); volume convention (./data:/data bind + hf-cache named volume); the whole CI workflow except IMAGE_NAME and the test-deps apt line; the .dockerignore.

Parameterized per repo:

Knob	translator (pilot)	alto-postprocess	nlp-enrich	page-classification
Image	ufal/atrium-translator	ufal/atrium-alto-postprocess	ufal/atrium-nlp-enrich	ufal/atrium-page-classification
Base	python:3.11-slim (all repos — satisfies 3.8+/3.10+)	same	same	same
apt build deps	g++ libxml2-dev libxslt1-dev (fasttext, lxml)	g++ git wget	none beyond base	g++ libgl1 libglib2.0-0 poppler-utils
Extra build steps	—	clone LayoutReader v3/, fetch lid.176.bin	—	—
torch in default image	no	yes	no (LLM variant: yes)	yes
ENTRYPOINT / default CMD	python main.py / batch args	python run_pipeline.py / []	bash -lc / 4-script chain	python3 run.py / -d /data/input --hf -rev v5.3
GPU	no	optional override	optional (LLM only)	optional override
api profile	no	yes (service/text_api.py)	no	yes (uvicorn service.api:app)
Runtime secret	none	none	HF_TOKEN (gated LLM)	HF_TOKEN (gated/push only)

---

Pilot: atrium-translator (reference template — full contents)

Dockerfile

# syntax=docker/dockerfile:1.7
FROM python:3.11-slim AS base

ARG ATRIUM_RUNNER_IMAGE=""
ARG ATRIUM_RUNNER_REPO="https://github.com/ufal/atrium-translator"
ARG ATRIUM_RUNNER_REF=""
ENV ATRIUM_RUNNER_IMAGE=${ATRIUM_RUNNER_IMAGE} \
    ATRIUM_RUNNER_REPO=${ATRIUM_RUNNER_REPO} \
    ATRIUM_RUNNER_REF=${ATRIUM_RUNNER_REF} \
    PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1 \
    HF_HOME=/cache/huggingface

RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential g++ libxml2-dev libxslt1-dev ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt requirements-test.txt ./
RUN pip install -r requirements.txt -r requirements-test.txt
COPY . .

RUN useradd --create-home --uid 10001 atrium \
    && mkdir -p /cache/huggingface /data \
    && chown -R atrium:atrium /app /cache /data
USER atrium

ENTRYPOINT ["python", "main.py"]
CMD ["/data/input", "--alto", "--formats", "alto.xml", "--target_lang", "en", "-o", "/data/output"]

docker-compose.yml

services:
  translator:
    image: ghcr.io/ufal/atrium-translator:${ATRIUM_VERSION:-dev}
    build:
      context: .
      args:
        ATRIUM_RUNNER_IMAGE: ghcr.io/ufal/atrium-translator:${ATRIUM_VERSION:-dev}
        ATRIUM_RUNNER_REPO: https://github.com/ufal/atrium-translator
        ATRIUM_RUNNER_REF: ${ATRIUM_RUNNER_REF:-dev}
    environment:
      ATRIUM_RUNNER_IMAGE: ghcr.io/ufal/atrium-translator:${ATRIUM_VERSION:-dev}
      ATRIUM_RUNNER_REPO: https://github.com/ufal/atrium-translator
      ATRIUM_RUNNER_REF: ${ATRIUM_RUNNER_REF:-dev}
      HF_HOME: /cache/huggingface
    volumes:
      - ./data:/data
      - hf-cache:/cache/huggingface
      - ./config.txt:/app/config.txt:ro
volumes:
  hf-cache:

Usage: docker compose run --rm translator (defaults), or override args after the service name, e.g. ... translator /data/input --alto -tgt en -o /data/output; aux script via docker compose run --rm --entrypoint python translator load_vocab.py --help.

.github/workflows/docker.yml

name: docker
on:
  pull_request:
  push:
    branches: ["claude/determined-hypatia-fi57y4", main]
    tags: ["v*"]
  release:
    types: [published]
env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ufal/atrium-translator
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends build-essential g++ libxml2-dev libxslt1-dev
      - run: pip install -r requirements.txt -r requirements-test.txt
      - run: pytest -m "not slow"
  build-and-push:
    needs: test
    if: startsWith(github.ref, 'refs/tags/v') || github.event_name == 'release'
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with: { registry: "${{ env.REGISTRY }}", username: "${{ github.actor }}", password: "${{ secrets.GITHUB_TOKEN }}" }
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=semver,pattern={{version}}
            type=sha
            type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          build-args: |
            ATRIUM_RUNNER_REPO=https://github.com/${{ github.repository }}
            ATRIUM_RUNNER_REF=${{ github.sha }}
            ATRIUM_RUNNER_IMAGE=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}

Build-args → Dockerfile ARG → ENV → paradata, so a published :v0.5.2 image self-reports docker_image=ghcr.io/ufal/atrium-translator:v0.5.2 and runner_ref=<sha>.

.dockerignore (identical in all four repos)

.git
.gitignore
.idea
.vscode
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.coverage
.venv/
venv/
env/
data_samples/
small_data_samples/
result/
model/
models/
checkpoint/
TEMP/
*.log
*.png
*.html
.cache/
*.bin
*.pt
*.onnx
.env

(Essential for page-classification: data_samples/use-case.png alone is ~17 MB.)

---

Per-repo deltas

atrium-nlp-enrich (replicate 2nd — light, no torch)

• apt: none beyond ca-certificates (pure-Python base deps; bash already present).
• pip install -r requirements.txt -r requirements-test.txt (base only — not requirements_llm.txt).
• Command runs the 4-stage LINDAT pipeline in order:
  ENTRYPOINT ["bash","-lc"] / CMD ["./api_1_manifest.sh && ./api_2_udp.sh && ./api_3_nt.sh && ./api_4_stats.sh"] (chmod +x first). set -euo pipefail in each script aborts the chain on failure.
• Scripts source config_api.txt with relative OUTPUT_DIR=./data_samples; bind-mount a tweaked config_api.txt (or set OUTPUT_DIR=/data) so output lands on the mount.
• Optional LLM/GPU variant = separate Dockerfile target llm that adds pip install -r requirements_llm.txt (torch 2.6.0, transformers, bitsandbytes), published as :<ver>-llm, command python llm_run.py …, behind compose profile llm + GPU override; HF_TOKEN env for gated models.

atrium-page-classification (replicate 3rd — largest image)

• apt: g++ libgl1 libglib2.0-0 poppler-utils (opencv/ultralytics + pdf2image in service).
• pip install -r requirements.txt -r service/requirements.txt -r requirements-test.txt.
• ENTRYPOINT ["python3","run.py"] / CMD ["-d","/data/input","--hf","-rev","v5.3"].
• Model volume page-model:/app/model; HF ViT (ufal/vit-historical-page) downloads at runtime.
• HF token: the [HF] token in config.txt is dormant (only commented push_to_hub uses it; the active download path passes no token). Do not bake it; supply HF_TOKEN/HUGGING_FACE_HUB_TOKEN via env only when needed. Follow-up PR: scrub the plaintext token from config.txt and read from env.
• api profile: entrypoint: ["uvicorn","service.api:app","--host","0.0.0.0","--port","8000"], ports: ["8000:8000"], optional HF_TOKEN.

atrium-alto-postprocess (replicate last — trickiest)

• apt: build-essential g++ git wget.
• pip install -r requirements.txt -r service/requirements.txt -r requirements-test.txt.
• ENV adds MODEL_DIR=/app/models and LANGID_CONFIG=/app/config_langID.txt.
• Translate setup_api_server.sh into build steps:
  1. Sparse-checkout LayoutReader v3/ from github.com/FreeOCR-AI/layoutreader into /app/v3 (on sys.path).
  2. wget lid.176.bin (~130 MB) into $MODEL_DIR, and symlink to /app/lid.176.bin — langID_classify.py:96 loads the bare filename lid.176.bin from CWD while the API reads MODEL_DIR.
• ENTRYPOINT ["python","run_pipeline.py"] / CMD [] (default method layoutreader).
• Model volume alto-models:/app/models; LayoutLMv3 + Qwen2.5-0.5B auto-cache to hf-cache.
• api profile uses extends: of the batch service + entrypoint: ["python","service/text_api.py"], ports: ["8000:8000"], ALLOWED_ORIGINS.
• GLM-4v-9b (--method glm, ~18 GB, GPU) stays opt-in at runtime — never pre-downloaded.

---

Shared conventions

• HF cache: HF_HOME=/cache/huggingface + named volume hf-cache — models never baked, downloaded once, reused.
• Model dirs: alto-models:/app/models (holds lid.176.bin), page-model:/app/model.
• I/O: single bind ./data:/data; convention input/ + output/ subdirs; paradata JSON lands under the output dir's paradata/.
• Secrets: HF_TOKEN supplied at runtime only (compose environment: from a git-ignored .env, or compose secrets) — never a build ARG, never committed.
• .env (git-ignored): ATRIUM_VERSION, ATRIUM_RUNNER_REF, HF_TOKEN.
• GPU (alto, page, nlp-llm): default images are CPU. Ship docker-compose.gpu.yml adding deploy.resources.reservations.devices (driver nvidia, capabilities:[gpu]) + NVIDIA_VISIBLE_DEVICES; used via -f docker-compose.yml -f docker-compose.gpu.yml. Requires NVIDIA Container Toolkit on host. We keep the slim base + torch's bundled CUDA rather than an nvidia/cuda base (smaller, no CUDA-version coupling).
• atrium-project: add docs/containerization.md (this strategy, condensed) + a links table; no code.

Risks / notes

• Image size: torch repos (alto ~5–7 GB, page ~6–8 GB CPU wheels). Pin the CPU torch wheel index in CI/Dockerfile to avoid accidentally pulling multi-GB CUDA wheels into the default image.
• GHCR is private by default — after first publish, repo admins must make the package public or have consumers authenticate.
• Network: full batch runs and API startup need outbound access to lindat.mff.cuni.cz and huggingface.co. The network-free checks are pytest -m "not slow" and alto's run_pipeline.py --dry-run.
• alto-postprocess has no .gitignore — add one alongside the .dockerignore.

Verification (per repo, on the feature branch)

1. Build with provenance args:
   docker compose build --build-arg ATRIUM_RUNNER_REF=$(git rev-parse HEAD) --build-arg ATRIUM_RUNNER_IMAGE=ghcr.io/ufal/atrium-translator:smoketest
2. Fast tests in-container (mirrors CI gate): docker compose run --rm --entrypoint pytest translator -m "not slow".
3. Batch smoke test on bundled samples (needs LINDAT/HF network):
   • translator: docker compose run --rm translator data_samples/my_documents --alto --formats alto.xml -tgt en -o /data/output
   • alto: docker compose run --rm alto --dry-run (network-free) then a real data_samples/ALTO run
   • nlp: docker compose run --rm nlp (4-script chain over data_samples/DOC_LINE_CATEG)
   • page: docker compose run --rm classify -d /data/input --hf -rev v5.3
4. Confirm provenance: after a run, read the emitted paradata/*.json and assert
   repository, runner_ref, docker_image equal the injected values — proves ARG→ENV→paradata end-to-end.
5. API smoke (alto, page): docker compose --profile api up, then curl localhost:8000/ and a sample upload.
6. CI validation: open a PR → only test runs and passes; push a v0.0.0-rc tag → build-and-push produces :0.0.0-rc, :sha-…, :latest in GHCR with the right ATRIUM_RUNNER_* baked.

Rollout order

1. Land translator artifacts → verify (steps 1–4, 6) → tag an RC → confirm GHCR image + paradata.
2. Replicate to nlp-enrich (fast), then page-classification, then alto-postprocess (LayoutReader sparse-checkout).
3. Add api profile + docker-compose.gpu.yml to the two service repos.
4. Add docs/containerization.md to atrium-project.
5. Follow-up PR: scrub plaintext HF token from page-classification config.txt.
6. Commit per repo on claude/determined-hypatia-fi57y4; git push -u origin claude/determined-hypatia-fi57y4 (no PRs unless requested).

Critical files

• */atrium_paradata.py — provenance contract (don't change; wire env into it).
• */para_config.txt — [tool] version → image tag.
• */pytest.ini — slow marker → CI gate.
• atrium-alto-postprocess/setup_api_server.sh, atrium-alto-postprocess/langID_classify.py (bare lid.176.bin load, line 96).
• atrium-nlp-enrich/api_1_manifest.sh … api_4_stats.sh — container command chain.
• atrium-page-classification/config.txt (line 43 token), run.py — batch entry.
• New per repo: Dockerfile, docker-compose.yml, .dockerignore, .github/workflows/docker.yml (+ docker-compose.gpu.yml & api profile for alto & page).

[K4TEL]

Commit ufal/atrium-page-classification@31f8af7 adds page classifier docker container draft (not tested in practice yet)

Commit ufal/atrium-translator@18942f4 adds translator docker container (not tested in practice yet)

Commit ufal/atrium-alto-postprocess@6477f30 adds alto postprocessing docker container (not tested in practice yet)

Commit ufal/atrium-nlp-enrich@9327d9e adds nlp enrichment docker container (not tested in practice yet)

[K4TEL]

Per-repo Docker integrations — current state (similarities & differences)

All four component repos now carry a draft Docker integration built from one shared template, so
they already behave like interchangeable building blocks for the composer/wrapper this issue
tracks. Summary of what's in place and where they diverge.

Shared foundation (identical by design)

• Base & build: python:3.11-slim, BuildKit (# syntax=docker/dockerfile:1.7), a single
  base stage (nlp-enrich adds a second — see below).
• Provenance contract: ARG → ENV ATRIUM_RUNNER_IMAGE / ATRIUM_RUNNER_REPO / ATRIUM_RUNNER_REF,
  consumed by the shared atrium_paradata.py and stamped into every output paradata JSON as
  docker_image / repository / runner_ref. CI injects them as build-args
  (github.repository, github.sha, IMAGE:ref_name), so a published image self-reports its origin.
• Runtime env: PYTHONUNBUFFERED, PYTHONDONTWRITEBYTECODE, PIP_NO_CACHE_DIR,
  PIP_DISABLE_PIP_VERSION_CHECK, HF_HOME=/cache/huggingface.
• Layer order: copy requirements → pip install → COPY . . (cache-friendly).
• Non-root user atrium (uid 10001) owning /app, /cache, /data.
• Compose: image: ghcr.io/ufal/atrium-<tool>:${ATRIUM_VERSION:-dev}, build passes the
  provenance args, environment: mirrors them + HF_HOME, volumes = ./data:/data + a named
  hf-cache volume + a read-only bind of the tool's config file. HF models are never baked —
  pulled at runtime into hf-cache.
• CI (.github/workflows/docker.yml): a test job runs pytest -m "not slow" on every
  PR/push; a build-and-push job runs only on v* tags / releases and pushes
  ghcr.io/ufal/atrium-<tool> tagged {semver, sha, latest} via docker/metadata-action +
  docker/build-push-action (GHA cache), with provenance passed as build-args.

Per-repo differences

Tool	Version	apt build deps	torch	Extra build steps	Entrypoint → default CMD	Compose service(s) / profiles	GPU	Persisted model volume
translator	v0.5.0	none (manylinux wheels: fasttext-wheel, lxml)	none	—	python main.py → /data/input --alto --formats alto.xml --target_lang en -o /data/output	translator	CPU-only	—
nlp-enrich	v0.10.1	ca-certificates, bash	only in optional llm target (CPU extra-index)	chmod +x api_*.sh; multi-stage base + llm	base: python run_pipeline.py ([]) · llm: python llm_run.py llm_config.txt	nlp, nlp-llm (profile llm)	optional (llm + GPU overlay)	—
alto-postprocess	v0.15.5	build-essential g++ git wget ca-certificates	CPU torch pinned (TORCH_INDEX_URL=…/cpu, overridable to CUDA)	LayoutReader v3/ sparse-checkout; fastText lid.176.bin download + symlink; MODEL_DIR, LANGID_CONFIG env	python run_pipeline.py ([])	alto, api (profile api, service/text_api.py :8000)	optional (GPU overlay; CPU default)	alto-models (lid.176.bin)
page-classification	v1.2.1-beta	build-essential g++ libgl1 libglib2.0-0 poppler-utils ca-certificates	GPU torch+torchvision (TORCH_INDEX_URL=…/cu121 default)	generates gpu_check.py; entrypoint enforces CUDA	sh -c "gpu_check && run.py" → -d /data/input --hf -rev v4.3	classify, api (profile api, uvicorn service.api:app :8000)	required (aborts on CPU)	page-model (ViT weights)

Cross-cutting notes

• GPU overlays: alto, nlp, and page each ship a docker-compose.gpu.* adding NVIDIA device
  reservations; translator has none (CPU-only tool). page-classification is the only image that
  requires a GPU; the others are CPU-first with optional GPU.
• API services: alto and page expose an opt-in FastAPI on :8000 behind an api profile;
  translator and nlp are batch-only (nlp instead has an llm profile for semantic enrichment).
• Runtime secrets: HF_TOKEN is wired for nlp-llm (gated models) and page (commented;
  gated / push) — supplied at runtime only, never as a build arg.
• Heaviest images are page-classification (CUDA torch) and alto-postprocess (CPU torch +
  LayoutReader); translator and base nlp are slim.

Minor inconsistencies worth aligning during integration

• nlp-enrich uses docker-compose.yaml / docker-compose.gpu.yaml; the others use .yml.
• The CI push.branches trigger lists differ per repo (leftover dev branch names, e.g.
  claude/intelligent-newton-…, claude/determined-hypatia-…, claude/exciting-sagan-…,
  claude/youthful-ptolemy-…).
• page-classification pins a model revision in its default CMD (-rev v4.3); the others don't.
• alto-postprocess is currently the only repo with a .env.example and a
  README.docker-section.md.

Takeaway

Because all four share the same registry (ghcr.io/ufal/atrium-*), provenance contract, /data
mount, and hf-cache convention, they already line up as drop-in stages. The remaining work for
this issue is the thin umbrella — a docker-compose over the published images plus a
run-pipeline.sh wrapper (and a workflow_dispatch GH Action) — rather than any change to the
per-repo images.

Next steps questions:

To proceed with the Docker wrapper and GH actions automation, the following information is needed:

• An expert in DevOps from ARUP side should check and test the existing drafts

• Overall Docker wrapper for the 4 repository tools - should it be added at all?

• Automation steps to be developed on GH actions:

  • - building in docker new tags/releases
  • - test set coverage for new commits and report generation
  • - pip dependencies update

[motyc]

This is to be discussed mainly with @rharasim

I believe we won´t need any overall Docker wrapper, i.e. we need to be able to run the individual containers with the tools reachable via API. But again, this is more to be discussed with Ronald, who will be the one handling the orchestration.