[WIP] Adding an example of a Jenkins CI enviornment using OIDC and optional Harbor pull-through mirror#330
Closed
ericsmalling wants to merge 47 commits into
Closed
[WIP] Adding an example of a Jenkins CI enviornment using OIDC and optional Harbor pull-through mirror#330ericsmalling wants to merge 47 commits into
ericsmalling wants to merge 47 commits into
Conversation
Self-contained Jenkins demo running entirely on cgr.dev/smalls.xyz images. The controller (jenkins:2-lts-jdk21-dev) is built with a fixed plugin set and the Chainguard docker-cli, talks to the host Docker daemon via mounted socket, and seeds pipeline jobs on startup via JCasC + job-dsl. Includes the first of the apps planned in PLAN.md: corretto-java17-maven, a Spring Boot console app whose pipeline builds on maven:3-jdk17-dev, smoke-tests on amazon-corretto-jre:17-dev, and archives the runnable JAR. Subsequent apps will land in follow-up commits. setup.sh creates a chainctl pull-token for cgr.dev/smalls.xyz pulls and writes a Docker config to .secrets/ (gitignored), avoiding any need for chainctl/cred helpers inside the Jenkins container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second sample app from PLAN.md: a JSP web page built with Maven on Adoptium JDK 8 (cgr.dev/smalls.xyz/maven:3-jdk8-dev) and smoke-tested on adoptium-jre:adoptium-openjdk-8-dev. The artifact is a self-contained runnable WAR — `java -jar app.war` boots embedded Jetty 9.4 and serves the JSP. A small com.example.Main launcher lives at the WAR root (placed there via maven-war-plugin webResources), uses only JDK classes to extract WEB-INF/lib jars to a temp dir, and reflectively constructs Jetty's Server/WebAppContext. AnnotationConfiguration is wired in so JettyJasperInitializer runs and installs the InstanceManager that Apache Jasper needs for JSP compile. The pipeline's Test stage launches the WAR, polls until the port is listening, fetches index.jsp via wget, and asserts the rendered HTML contains the expected greeting. Also adds apps/*/target/ to the local .gitignore so verifying pipelines locally doesn't dirty the working tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third sample app from PLAN.md: a hello-world standalone runnable JAR built with Gradle on Chainguard's plain JDK 21 image and smoke-tested on jre:openjdk-21-dev. Build image is jdk:openjdk-21-dev which ships only the JDK, so the app includes the Gradle wrapper (bootstrapped from the gradle:8-jdk21-dev image). The pipeline runs `./gradlew --no-daemon clean jar` and overrides GRADLE_USER_HOME to point inside $WORKSPACE — Jenkins runs build containers as uid 1000 with no writable HOME, which otherwise breaks the wrapper's distribution cache. The Test stage runs the JAR and asserts it prints the expected greeting plus a JDK 21 version string. Note that this image reports `java.vendor: wolfi` rather than `Chainguard` (unlike the Corretto and Adoptium variants), so the smoke test asserts on java.version instead. Also folds the user's "Future enhancements" notes into PLAN.md (config- urable org, optional Harbor pull-through, push artifacts to Harbor instead of ttl.sh) and adds Gradle build dirs to the local .gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth sample app from PLAN.md and the first that produces an OCI
image artifact rather than a file. Pipeline builds a Flask app with
`uv pip install` on cgr.dev/smalls.xyz/python:3.14-dev, multi-stage
docker-builds a minimal image whose runtime layer is the shell-less
python:3.14, and pushes to ttl.sh/smalls-pytest:3-14.
The Test stage runs the just-built image with
`docker run --entrypoint=python ... -c "<flask test client>"` for an
in-process smoke test — no port mapping or cross-container networking
required, which sidesteps the shell-less runtime image cleanly.
Two non-obvious gotchas, both around the build user:
- The Dockerfile's build stage adds `USER 0` because the chainguard
python:3.14-dev defaults to uid 65532, which can't write to
/usr/lib/python3.14/site-packages.
- The Jenkinsfile's Build deps stage passes `args '--user 0 --entrypoint='`
for the same reason — Jenkins' docker-workflow plugin always runs
agent containers as -u 1000:1000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fifth sample app from PLAN.md. Same shape as the python314-uv-flask pipeline (build deps in dev image, multi-stage docker build, in-process test client smoke test, push to ttl.sh) with two substitutions: pip for uv, and Django for Flask. The Django site is a single app.py using settings.configure() — keeps the demo focused on packaging+containerization rather than scaffolding. `python app.py` defaults to runserver 0.0.0.0:8080 --noreload. Pushed image: ttl.sh/smalls-pytest:3-12. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sixth sample app — substituted Node 22 LTS for the PLAN.md original
"Node 21" since Node 21 is EOL and not in the smalls.xyz catalog.
Same shape as the python OCI-image samples: Build deps stage on the
dev image (sanity-checks the package.json), multi-stage docker build
into the shell-less runtime image, in-process smoke test, push to
ttl.sh/smalls-nodetest:22.
The smoke test runs the runtime image with `--entrypoint=/usr/bin/node`
and an inline `-e` script that boots the express app on an ephemeral
loopback port, issues a self-`http.get('/')`, and asserts the response.
No host port mapping or cross-container networking required.
Sets HOME=$WORKSPACE in the Build deps stage — Jenkins runs the agent
container as uid 1000 with no writable home, which otherwise breaks
npm's cache (~/.npm). Same kind of fix used for Gradle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seventh and final sample app from PLAN.md. Same shape as the Node 22
sibling but with three substitutions: pnpm for npm, node:25-slim
(smaller runtime variant) for node:22, and nanoid as the npm library.
The slim variant has the same shape as the regular runtime image —
shell-less, /usr/bin/node entrypoint, /app workdir — so the smoke test
uses the same in-process self-request trick.
Notes on pnpm:
- The dev image (node:25-dev) ships pnpm 10.33 pre-installed, no
install step needed.
- `pnpm install --prod --frozen-lockfile` fails fast if pnpm-lock.yaml
drifts from package.json — useful in CI.
- pnpm's symlink-based node_modules layout survives the Dockerfile
`COPY --from=build /app/node_modules` because all symlinks point
inside that directory tree.
Sets HOME=$WORKSPACE in the Build deps stage to give pnpm a writable
store/cache, same fix used for npm in the Node 22 sample.
Adds apps/*/node_modules/ and apps/*/.pnpm-store/ to the local
.gitignore so local lockfile bootstraps don't leak install artifacts
into commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a quick-reference table of every job (build/runtime images and artifact format), updates the architecture diagram to show ttl.sh and generic per-stage images, and writes a "Common gotchas" section that captures the recurring traps hit while building out the samples (Chainguard ENTRYPOINTs, -dev variants in test stages, uid-1000 HOME issues for npm/pnpm/gradle, uid-65532 site-packages writes for python). Also updates the "Adding another sample app" walkthrough to include the JCasC reload-via-API step that's needed after restart. Drops the stale "Currently only the first app is implemented" note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PLAN.md "Future enhancements" chainguard-demo#1 — decouple the demo from the hard-coded smalls.xyz org so anyone can run it against their own Chainguard org (or the public chainguard catalog) without editing source files. Single knob: CHAINGUARD_ORG in jenkins/.env (gitignored, bootstrapped from the new .env.example, default smalls.xyz). docker-compose auto-loads .env and propagates the value four ways: * As a build ARG to Dockerfile.jenkins (controller image FROM lines) * As a runtime env var on the controller container * Through JCasC: ${CHAINGUARD_ORG} interpolation in systemMessage, plus a globalNodeProperties.envVars block so pipelines see it via env.CHAINGUARD_ORG (env vars set on the controller process do NOT auto-flow into pipeline env — JCasC bridge is required) * To setup.sh, which now sources .env so its chainctl pull-token call uses the same org as the rest of the stack Per-app changes follow the existing print-events/Dockerfile pattern: each app Dockerfile declares ARG CHAINGUARD_ORG=smalls.xyz and uses ${CHAINGUARD_ORG} in FROM lines; the corresponding Jenkinsfiles pass --build-arg CHAINGUARD_ORG="$CHAINGUARD_ORG" to docker build, and their `agent { docker { image '...' } }` blocks switch from single- to double-quoted strings interpolating ${env.CHAINGUARD_ORG}. Default of smalls.xyz preserves zero-friction continuity for the current demo flow. Out of scope: ttl.sh push tag prefixes (smalls-pytest, smalls-nodetest) — those go away with PLAN.md enhancement chainguard-demo#3 (push to Harbor). Verified end-to-end: corretto-java17-maven and python314-uv-flask both pass with default org; controller image rebuild against CHAINGUARD_ORG=chainguard correctly resolves cgr.dev/chainguard/... in FROM lines. Build console output for build chainguard-demo#5 of python pipeline shows the substitution flowing through every layer: + docker inspect cgr.dev/smalls.xyz/python:3.14-dev + docker build --build-arg 'CHAINGUARD_ORG=smalls.xyz' -t ... Step 1/12 : ARG CHAINGUARD_ORG=smalls.xyz Step 2/12 : FROM cgr.dev/${CHAINGUARD_ORG}/python:3.14-dev AS build Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipelines used to write the full cgr.dev/<org>/<image>:<tag> path
inline. Now they call cgImage('<token>') (e.g. 'corretto-java17',
'python-3.14') from a Jenkins shared library that resolves the token
to a Map of build/test/runtime image strings, with the <org> segment
coming from env.CHAINGUARD_ORG.
Implementation:
* shared-libraries/cg-images/vars/cgImage.groovy — the library entry
point. Catalog of 7 tokens covering all current sample apps. To add
a new token, append a row.
* jenkins/casc/jenkins.yaml — globalLibraries config registering the
library as `cgImages` with implicit: true (no @Library annotation
needed in pipelines), retriever=legacySCM(filesystem(path:...)).
* jenkins/plugins.txt — adds filesystem_scm so the library can load
from a bind-mounted directory rather than a Git remote.
* docker-compose.yml — bind-mounts ./shared-libraries into the
controller (read-only) at /tmp/cgjenkins-home/shared-libraries.
* Per-app Jenkinsfiles (×7) — `def img = cgImage('<token>')` at the
top of each, then `image img.build` / `image img.test` in agent
docker blocks. Zero cgr.dev hardcodes left in apps/*/Jenkinsfile.
* shared-libraries/cg-images/README.md — usage docs plus a caveats
section covering live-reload behavior, syntax-error blast radius,
and the implicit-loading vs explicit-@Library trade-off.
Verified: corretto-java17-maven build chainguard-demo#8 (11s) and python314-uv-flask
build chainguard-demo#6 (13s) both succeeded with the shared library resolving image
strings correctly. Confirmed empirically that filesystem-SCM library
edits propagate to the next pipeline run without a Jenkins restart
(probe-marker test).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each catalog entry in vars/cgImage.groovy now uses repo:tag@sha256:...
format. The tag is retained for human readability; the digest provides
immutable identity so re-runs of a pipeline pull the same image bytes
even if an upstream :dev tag is later repointed. Docker accepts the
combined repo:tag@digest form natively.
Adds refresh-digests.sh to re-resolve every catalog entry against the
configured CHAINGUARD_ORG (read from ../../.env, falling back to
smalls.xyz) and rewrite the digests in place. Use it when:
- Adopting newer image versions (security patches, etc.)
- Switching CHAINGUARD_ORG (digests are technically org-specific
even when content is byte-identical, so re-pin on org change)
- Adding a new catalog entry with no initial digest
Sed-based rewrite — initial perl version eats the `@` sign because
perl interpolates `@sha256` as an array variable in the replacement
string.
README adds a "pinned by digest" section, a refresh walkthrough, and
two new caveats:
- Digests are org-specific; re-pin when changing CHAINGUARD_ORG
- Pins are frozen in time and don't auto-update; for production
use digestabotctl/Renovate/etc. instead of leaving stale pins
Verified end-to-end: corretto-java17-maven chainguard-demo#11 (11s) and
python314-uv-flask chainguard-demo#8 (10s) both succeeded; build console shows
cgr.dev/smalls.xyz/<image>:<tag>@sha256:<digest> being inspected and
pulled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A new ops pipeline that runs refresh-digests.sh inside a Chainguard
crane container every 4 hours, picking up upstream tag movements
without manual intervention. The schedule (H H/4 * * *) randomizes
the minute and starting hour to avoid registry stampedes.
Implementation:
* jenkins/ops/refresh-cgimages-digests/Jenkinsfile — uses
`agent { docker { image cgr.dev/${CHAINGUARD_ORG}/crane:latest-dev } }`
and lets the docker-workflow plugin's automatic --volumes-from
inheritance pull in the controller's mounts. That gives us the
pull-token docker config (for crane auth) and the rw shared-
libraries dir (where the script writes) for free, no extra -v.
* jenkins/casc/jobs.groovy — a separate ops-jobs block so this lives
alongside the seven sample-app pipelines without polluting the
`apps` list.
* docker-compose.yml — flips ./shared-libraries from :ro to :rw so
the spawned crane container can write back, plus adds ./ops:ro
so the seed can load the Jenkinsfile from disk.
The bind-mount-rewrite story: when the job rewrites
vars/cgImage.groovy, the change shows up on the host filesystem
immediately. The cgImages library is filesystem-SCM live-loaded, so
the next sample-app pipeline run picks up the new digests with no
Jenkins restart (verified empirically earlier in this branch).
Two minor refresh-digests.sh fixes baked in:
* sed-based digest substitution (perl interpolated `@sha256` as an
array variable, eating the @ — bug fixed previously)
* Skip the trailing `git diff` when running in an environment
without git (the crane container) instead of falling through to
git's help text
Top-level README mentions the auto-refresh; ops/refresh-cgimages-
digests/README.md covers the design and a few caveats (writes are
visible on the host filesystem, digest refresh costs one HEAD per
entry, schedule edits require a controller restart).
Verified: build chainguard-demo#2 SUCCESS in 9s, all 14 digests resolved via crane
inside cgr.dev/smalls.xyz/crane:latest-dev, vars/cgImage.groovy
rewritten in place. Several digests had drifted since yesterday's
manual pin — exactly the case this job exists to handle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminates the long-lived pull-token (.secrets/docker-config.json) in
favor of per-build short-lived chainctl sessions. Each pipeline opens
with a new `Auth` stage that calls `cgLogin()` (a sibling shared-
library var to cgImage), which exchanges a per-build OIDC token
issued by Jenkins itself for a fresh ~30-min Chainguard session.
Architecture:
* Jenkins-as-OIDC-issuer via the new oidc-provider plugin.
JCasC configures buildClaimTemplates to set sub to a fixed string
(jenkins-cgimages-puller) and the credential's audience to
Chainguard's https://issuer.enforce.dev.
* Chainguard identity uses the `static` block — Jenkins' JWKS is
uploaded directly at terraform-apply time. Chainguard's IAM
never has to reach our local controller, so the demo stays
self-contained (no cloudflared/ngrok). The trade-off: subject is
fixed (single identity in audit logs), since `static` doesn't
support subject_pattern.
* Setup.sh refactor: polls Jenkins until it serves /oidc/jwks,
fetches the JWKS into iac/jenkins-jwks.json, runs terraform apply
against jenkins/iac/ (chainguard_identity + rolebinding to
registry.pull), and writes the resulting UIDP into
shared-libraries/cg-images/IDENTITY (gitignored). cgLogin reads
the file at build time, so a Jenkins restart does NOT need to
propagate it.
Per-build flow:
1. Auth stage (agent any) calls cgLogin() — withCredentials wraps
the oidc-provider plugin's per-build JWT, then chainctl auth
login + chainctl auth configure-docker write a fresh docker
config.json to $DOCKER_CONFIG.
2. Subsequent agent { docker { image cgImage(...).build } } stages
reuse the freshly-written config to pull from cgr.dev.
3. Token expires after ~30min; covers any single build comfortably.
New images required in smalls.xyz: chainctl (multi-stage-copied into
the controller alongside docker-cli). Already added by the user.
Several non-obvious wrinkles surfaced during build-out, all noted in
either jenkins.yaml comments or the cgLogin.groovy comments:
* jenkins.location.url is set to https://localhost:8080/ (lying
about the scheme) so the OIDC `iss` claim is HTTPS — chainguard's
static block validator requires HTTPS. The plugin still serves
JWKS at the real http://localhost:8080/oidc/jwks; static-mode is
offline-verification, the URL never has to resolve.
* Don't override `issuer` per-credential — the plugin then refuses
to serve JWKS on its default endpoint (Keys.java explicitly
skips creds with custom issuers).
* configure-docker creates a /usr/local/bin/docker-credential-cgr
symlink. uid 1000 can't write there, so we pre-create the symlink
in Dockerfile.jenkins as root.
* configure-docker also needs --identity --identity-token to skip
its fallback browser-login path.
* The cgImages library config now uses `clone: true` so
filesystem_scm checks out a fresh copy each build — without it,
edits to cgLogin.groovy don't propagate.
* After ANY restart, the oidc-provider plugin regenerates its
signing key, invalidating the uploaded JWKS. setup.sh must be
re-run after every controller restart. Documented in the README.
Files dropped: the .secrets/docker-config.json bind-mount in
docker-compose.yml is gone (no more pull tokens anywhere).
Verified end-to-end: corretto-java17-maven build chainguard-demo#23 (15s) succeeded
— Auth stage logged in as the assumed identity, agent docker stages
pulled cgr.dev/smalls.xyz/maven and amazon-corretto-jre using the
short-lived chainctl session. python314-uv-flask chainguard-demo#8 (10s) confirmed
the OCI-image-push flow still works (push to ttl.sh is unaffected).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the analysis of which parts of the OIDC assumed-identity work become redundant vs reusable when the Harbor pull-through and push- retarget enhancements (chainguard-demo#2 + chainguard-demo#3) land. Future-me will thank past-me. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…B and C)
Implements both halves of PLAN.md's Harbor enhancements as opt-in modes
selected at setup.sh time. The user is asked three questions and the
script dispatches to one of:
Mode A — direct cgr.dev (current default)
pulls: cgr.dev/<org> (via OIDC chainctl session, per-build)
push: ttl.sh/<prefix> (anonymous, default; user can override)
Mode B — Harbor pull-through, push to ttl.sh
pulls: localhost/cgr-proxy/<org> (anonymous, Harbor proxy cache)
push: ttl.sh/<prefix> (default; user can override)
Mode C — Harbor for both
pulls: localhost/cgr-proxy/<org>
push: localhost/library (Harbor admin/Harbor12345 baked in
by cgLogin)
Harbor deployment is lifted (with light adaptations) from the
chainguard-demo/cs-workshop "operations-track/harbor" tutorial:
jenkins/harbor/
deploy.sh Adapted from the tutorial's deploy-harbor.sh.
Env-var-driven (CHAINGUARD_ORG, PULL_USER,
PULL_PASS) so setup.sh can call it
non-interactively. Idempotent.
teardown.sh `kind delete cluster --name jenkins-harbor`.
kind/config.yaml Single-node kind cluster with host port
80/443 mappings for ingress.
cg/helm/values.template Harbor Helm values, all images parameterized
on $REGISTRY_URL = cgr.dev/$CHAINGUARD_ORG.
cg/manifests/... ingress-nginx static manifest, parameterized.
terraform/ Harbor IAM: registers cgr.dev as upstream and
creates the public 'cgr-proxy' project.
Dropped the tutorial's replication-mirror
project; we don't use replication.
Demo-side wiring:
setup.sh (rewritten) Three interactive prompts; updates .env;
rebuilds Jenkins; dispatches to either the
OIDC bootstrap (Mode A) or harbor/deploy.sh
(Modes B/C). Re-run to switch modes.
docker-compose.yml New env vars passed to controller:
PULL_REGISTRY, PUSH_REGISTRY, HARBOR_ENABLED.
jenkins.yaml Same env vars exposed to pipelines via
globalNodeProperties.
cgImage.groovy reg = env.PULL_REGISTRY ?: cgr.dev/<org>.
cgLogin.groovy Branches by HARBOR_ENABLED + PUSH_REGISTRY:
Mode A → OIDC chainctl flow (existing).
Mode B → no-op (anonymous everywhere).
Mode C → write Harbor admin creds for
localhost in $DOCKER_CONFIG.
apps/{python,node}*/ OCI-image Jenkinsfiles' IMAGE env now uses
"${env.PUSH_REGISTRY}/<app-name>:<tag>"
(slash-separated; works for both ttl.sh and
Harbor).
Verified end-to-end against all three modes:
Mode A — corretto-java17-maven chainguard-demo#24 SUCCESS in 13s, python314-uv-flask
chainguard-demo#10 SUCCESS in 15s, push to ttl.sh/smalls/pytest:3-14.
Mode B — corretto-java17-maven chainguard-demo#26 SUCCESS in 14s, image refs in build
console show pulls from localhost/cgr-proxy/smalls.xyz/...
Mode C — python314-uv-flask chainguard-demo#12 SUCCESS in 12s, push landed in Harbor's
library project (confirmed via Harbor API:
projects/library/repositories returns
library/pytest with artifact_count=1).
PLAN.md "Future enhancements" section updated — all three are done.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One script to clean up the whole demo:
1. harbor/teardown.sh — delete the kind cluster (no-op if not present)
2. iac/ terraform destroy — release the Chainguard assumed identity
3. docker compose down --rmi local --remove-orphans — Jenkins out
4. rm -rf /tmp/cgjenkins-home — JENKINS_HOME bind mount
5. Remove .secrets/, harbor/.pull-token, IDENTITY, terraform state,
rendered manifests/values
Prompts before doing anything destructive. Idempotent — re-running on a
clean slate reports each step as a no-op.
Pass --wipe-env to also delete .env (otherwise it's kept so re-running
setup.sh remembers the user's CHAINGUARD_ORG choice).
Step 4 tries plain rm first and only falls back to sudo when needed —
on macOS + OrbStack the bind-mount is owned by the host user (no sudo);
on Linux it may be owned by uid 1000 mapped from inside the container.
README's Teardown section now points to the script and keeps the manual
steps as a fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…in check Harbor 2.12.3+ pulled in gorilla/csrf v1.7.3, which hardcodes the request scheme to "https" inside its origin check. Harbor's middleware doesn't wrap the request as plaintext, so a browser POST /c/login over HTTP is rejected with 403 "origin invalid" before the password is checked. Upstream tracking issue: goharbor/harbor#22010. Fix: enable TLS on the chart with an auto-generated self-signed cert, keep externalURL as http://localhost, and disable ssl-redirect. Browser users hit https://localhost/harbor (one-time cert warning), the registry path stays HTTP so `docker push localhost/library/...` keeps working (Docker daemon refuses HTTPS to 127.0.0.0/8 by default). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four OCI-image pipelines (python314, python312, node22, node25) now run Sign and Verify stages after Push, so every demo build proves the sign-then-verify loop end-to-end. Setup.sh generates a static cosign keypair on first run (cached at /tmp/cgjenkins-home/.secrets/) and exports the passphrase for docker compose. JCasC wires three credentials into Jenkins's encrypted store at boot — cosign-private-key (Secret file), cosign-public-key (Secret file), cosign-password (Secret text) — using the new plain-credentials plugin. The cgSign and cgVerify shared-library vars pull credentials through withCredentials so the password is masked in build logs and the key material never lives in plaintext outside the Jenkins-managed store. Cosign itself runs as a one-shot --network host sibling container so Mode C's `localhost` registry references resolve to the host ingress (the controller container's loopback can't reach it). cgLogin's Mode C DOCKER_CONFIG now writes auth for both `localhost` and `localhost:80` — cosign's reference parser rejects bare `localhost` and we rewrite digest refs to the explicit-port form before invoking it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ght image probe
setup.sh now prompts for CHAINGUARD_ORG on first run if .env doesn't supply
one and persists the answer for re-runs. All ${CHAINGUARD_ORG:-smalls.xyz}
fallbacks throughout the codebase are gone — Dockerfiles, docker-compose.yml,
shared-library Groovy, refresh-digests.sh, terraform variables, and docs all
require the org to be set explicitly. docker-compose now uses the ${VAR:?msg}
form so unset CHAINGUARD_ORG produces a clear "run ./setup.sh to be prompted"
error rather than silently building against the wrong catalog.
setup.sh also adds a preflight probe before bringing anything up: parallel
`docker manifest inspect` calls against every image the demo will pull
(controller deps + cgImage catalog + Harbor microservices in Modes B/C).
Missing images are listed with auth/ org-typo hints, and setup aborts
before docker compose / terraform / helm waste time. Bypass with
SKIP_PREFLIGHT=1 ./setup.sh when you know what you're doing.
App READMEs that previously wrote `cgr.dev/smalls.xyz/...` now show
`cgr.dev/$CHAINGUARD_ORG/...` to make the parameterization obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch the preflight from "log only the missing ones" to a per-image status list. Parallel probes still run via xargs -P 8, but results land in a tempfile and are read back in input-array order so the printed list matches the curated CORE_IMAGES + HARBOR_IMAGES sequence rather than whatever order the parallel probes happened to finish. Each line gets a colored marker — green ✓ for accessible, red ✗ for missing — and the abort path stays the same: any failure prints the auth/org-typo hint and exits non-zero. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
setup.sh used to fall back to ttl.sh/smalls when the user hit enter without typing a push target — bakes one user's prefix into the demo. Replace with: - if PUSH_REGISTRY was persisted from a prior setup.sh run, offer that value as the suggested default (empty input keeps it) - otherwise loop until the user enters something explicit, with a generic ttl.sh/your-prefix hint Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous {{index .RepoDigests 0}} grabbed whichever digest was first in
the local image cache — fine on a fresh build but wrong when the same image
content had been pushed to multiple registries across mode switches. Reproed
on a Mode-A re-deploy where python312-pip-django's image already had a
stale localhost/library RepoDigest from a prior Mode-C session, which then
got rewritten to localhost:80 and tried to dial a registry that wasn't
running. Filter RepoDigests to the entry whose repo matches the IMAGE we
just pushed and use that.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The stock Stage View is fine for short linear pipelines but starts wrapping once a build has 7+ stages, which all the OCI-image samples do. pipeline-graph-view renders a tighter left-to-right graph in the build's "Pipeline Overview" tab. No JCasC config needed — the plugin auto-mounts its UI on existing pipeline jobs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three problems compounded after the recent refactors: 1. The crane container in the agent didn't see CHAINGUARD_ORG (the docker-workflow plugin only forwards env vars listed in `args`), so refresh-digests.sh's required-var check tripped immediately. Forward CHAINGUARD_ORG and PULL_REGISTRY explicitly. 2. The script hardcoded `crane digest cgr.dev/$ORG/...` for lookups, which needs cgr.dev creds in the agent's DOCKER_CONFIG. In Modes B/C cgLogin only writes localhost auth, so crane fell back to anonymous and 401'd. Route lookups through PULL_REGISTRY instead — Harbor proxy serves the upstream manifest verbatim, anonymously, so the digests come back identical to what cgr.dev would return. 3. crane in the agent container couldn't reach the host's localhost:80 (different network namespace) — same `--network host` workaround we already use for cosign. Also rewrite `localhost/...` to `localhost:80/...` so go-containerregistry's parser stops sending the request to Docker Hub instead of the local registry. The refreshed cgImage.groovy reflects today's tag-tip digests (python:3.x-dev and a couple others have rolled forward since the previous pin). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Eric Smalling <eric.smalling@chainguard.dev>
ericsmalling
changed the title
ericsmalling
changed the title
Jenkins's anti-symlink-traversal check rejects untar of any tar that writes a regular file inside a directory that's a symlink. pnpm's node_modules/.pnpm/ tree is built almost entirely out of those, so any stash that captured node_modules trips the check on the next unstash. The node25 Checkout stage was doing `cp -R /sources/... .` followed by `stash includes: '**'` against a workspace Jenkins reuses across builds. Once a build had run pnpm install, the next build's Checkout overlaid fresh source on top of the leftover node_modules and stashed all 3016 files together. Subsequent unstash failed. Two fixes: - cleanWs() before the cp so the workspace starts empty regardless of prior-build leftovers, - excludes: 'node_modules/**' on the stash as a safety net for any future stage ordering that ends up with node_modules in the workspace before stash runs. Apply the same to node22 defensively. npm's flat node_modules layout doesn't trigger the symlink check today, but cleaning the workspace and trimming the stash is good hygiene either way. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+97
to
+101
| // Mode A: OIDC chainctl flow. | ||
| def identity = readFile('/tmp/cgjenkins-home/shared-libraries/cg-images/IDENTITY').trim() | ||
| if (!identity) { | ||
| error('cgLogin: shared-libraries/cg-images/IDENTITY is empty — run setup.sh first (or set HARBOR_ENABLED=true to use the Harbor proxy cache).') | ||
| } |
Comment on lines
+274
to
+278
| if [[ -f .env ]]; then | ||
| while IFS= read -r line || [[ -n "$line" ]]; do | ||
| if [[ "$line" == "${key}="* ]]; then | ||
| printf '%s=%s\n' "$key" "$value" >> "$tmp" | ||
| found=1 |
Four Copilot findings from the tenth pass on PR chainguard-demo#330: - cgLogin: wrap the IDENTITY readFile in a fileExists guard so a missing file produces the friendly "run setup.sh first" error instead of a low-level Groovy exception. - teardown.sh: stop wiping iac/terraform.tfstate when TF_DESTROY_FAILED. Without state there's no handle on the remote assumed identity, so the previous behaviour orphaned IAM resources on failed/skipped destroy. Preserve the iac state for retry; harbor's terraform state is still wiped unconditionally (the kind cluster gets blown away wholesale by harbor/teardown.sh, so the state is moot). - setup.sh update_env: reject values containing a newline. A multi- line value would silently break .env (the value terminates at the embedded newline and the remainder is parsed as a separate key). - setup.sh pull-token parsing: ask chainctl for -o json and parse with python3 (already used for the JWKS validator, no new dep) rather than scraping the human-readable `--username "X"` flag-hint text with grep+sed. Accept a couple of alternative JSON key names for forward-compat; raw response is dumped on failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+100
to
+106
| if (( TF_DESTROY_FAILED == 0 )); then | ||
| rm -rf iac/.terraform iac/terraform.tfstate iac/terraform.tfstate.backup iac/jenkins-jwks.json | ||
| else | ||
| echo " Preserving iac/terraform.tfstate so a future ./teardown.sh can retry the destroy." | ||
| fi | ||
| rm -rf harbor/terraform/.terraform harbor/terraform/terraform.tfstate harbor/terraform/terraform.tfstate.backup harbor/terraform/terraform.tfvars | ||
| rm -f harbor/cg/helm/values.yaml harbor/cg/manifests/deploy-ingress-nginx.yaml |
Comment on lines
+47
to
+55
| if [[ -z "${CHAINGUARD_ORG:-}" ]]; then | ||
| echo "==> No Chainguard org configured." | ||
| echo " Examples: 'chainguard' (public catalog) or 'your-org.example.com'." | ||
| while [[ -z "${CHAINGUARD_ORG:-}" ]]; do | ||
| read -rp " Enter your Chainguard org: " CHAINGUARD_ORG | ||
| done | ||
| echo | ||
| fi | ||
| ORG="$CHAINGUARD_ORG" |
Comment on lines
+77
to
+86
| PUSH_REGISTRY_DEFAULT="${PUSH_REGISTRY:-}" | ||
| PUSH_REGISTRY="" | ||
| while [[ -z "$PUSH_REGISTRY" ]]; do | ||
| if [[ -n "$PUSH_REGISTRY_DEFAULT" ]]; then | ||
| read -rp "Where should pipelines push their built images? [last used: ${PUSH_REGISTRY_DEFAULT}]: " PUSH_REG_INPUT | ||
| PUSH_REGISTRY="${PUSH_REG_INPUT:-$PUSH_REGISTRY_DEFAULT}" | ||
| else | ||
| read -rp "Where should pipelines push their built images? (e.g. ttl.sh/your-prefix): " PUSH_REGISTRY | ||
| fi | ||
| done |
…e cleanup Five Copilot findings from the eleventh pass on PR chainguard-demo#330: - setup.sh: trim + validate user-supplied CHAINGUARD_ORG and PUSH_REGISTRY (sanitize_env_value + validate_env_value helpers) before persisting. Rejects empty, internal whitespace, and quotes — all of which would silently break .env parsing by docker compose / JCasC / bash `source`. Also re-validates values inherited from .env so a hand-edited corruption surfaces at setup.sh entry rather than mid-`terraform apply`. - cgSign + cgVerify: switch the digest-resolution sh blocks from `set -eu` to `set -eu -o pipefail` so a failing `docker image inspect` mid-pipeline surfaces as the script's exit status rather than being masked by the trailing `head -1` returning 0. - teardown.sh: also remove iac/.terraform.lock.hcl and harbor/terraform/.terraform.lock.hcl on cleanup. They're generated by terraform init and gitignored, so a "full teardown" leaving them behind contradicts the script's stated goal. (Preserved alongside iac/terraform.tfstate when TF_DESTROY_FAILED so the user can retry.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+342
to
+351
| # Persist the Harbor admin password if Harbor is enabled. Default to the | ||
| # chart's own default ("Harbor12345") so an unmodified first-run works, but | ||
| # allow user override by pre-setting HARBOR_ADMIN_PASSWORD in .env. Same | ||
| # value drives the Helm chart, the Terraform harbor provider, and cgLogin's | ||
| # Mode C push-auth — keeping them all wired off one env var means there's | ||
| # only one knob to rotate. | ||
| if [[ "$HARBOR_ENABLED" == "true" ]]; then | ||
| HARBOR_ADMIN_PASSWORD="${HARBOR_ADMIN_PASSWORD:-Harbor12345}" | ||
| update_env HARBOR_ADMIN_PASSWORD "$HARBOR_ADMIN_PASSWORD" | ||
| fi |
Comment on lines
+39
to
+44
| def call(String token) { | ||
| // PULL_REGISTRY is set by JCasC globalNodeProperties (driven by setup.sh). | ||
| // Defaults to cgr.dev/<org> for the no-Harbor case; switches to | ||
| // localhost/cgr-proxy/<org> when Harbor is the active pull-through cache. | ||
| def reg = env.PULL_REGISTRY ?: "cgr.dev/${env.CHAINGUARD_ORG}" | ||
| def catalog = [ |
Two Copilot findings from the twelfth pass on PR chainguard-demo#330: - setup.sh: also apply sanitize_env_value + validate_env_value to HARBOR_ADMIN_PASSWORD before persisting. Same risks as the other prompted values — whitespace or quotes would break .env sourcing, envsubst into the Helm values template, the harbor Terraform tfvar, and the base64 auth cgLogin builds for Mode C. - cgImage: error when both PULL_REGISTRY and CHAINGUARD_ORG are empty. Otherwise the fallback evaluates to the literal "cgr.dev/null" (Groovy stringifies a null reference inside the GString) and pipelines fail with confusing manifest-not-found errors instead of a clear "re-run setup.sh" message. Same guard pattern cgLogin / cgSign / cgVerify already use. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+301
to
+360
| echo "==> Writing mode flags to .env..." | ||
| [[ -f .env ]] || cp .env.example .env | ||
| update_env() { | ||
| local key="$1" value="$2" | ||
| # Reject values containing a newline — they'd produce a .env line that | ||
| # docker-compose / JCasC env-var parsers would silently misinterpret | ||
| # (the value would terminate at the embedded newline and the remainder | ||
| # would be parsed as a separate key). docker-compose's dotenv format | ||
| # supports escaped/quoted multi-line values but JCasC's `${VAR}` resolver | ||
| # doesn't, so the safest rule is no newlines at all. | ||
| if [[ "$value" == *$'\n'* ]]; then | ||
| echo "ERROR: update_env value for ${key} contains a newline — refusing to corrupt .env." >&2 | ||
| exit 1 | ||
| fi | ||
| # Rewrite .env line-by-line in pure bash rather than passing $value through | ||
| # sed. A user-supplied registry/org could contain &, \, |, or other sed | ||
| # special chars (the s/// replacement side treats & as the matched string | ||
| # and \N as a backref), which would silently corrupt the .env. Building | ||
| # the file in a temp and atomically renaming avoids that whole class of bug. | ||
| local tmp | ||
| tmp=$(mktemp) | ||
| local found=0 | ||
| if [[ -f .env ]]; then | ||
| while IFS= read -r line || [[ -n "$line" ]]; do | ||
| if [[ "$line" == "${key}="* ]]; then | ||
| printf '%s=%s\n' "$key" "$value" >> "$tmp" | ||
| found=1 | ||
| else | ||
| printf '%s\n' "$line" >> "$tmp" | ||
| fi | ||
| done < .env | ||
| fi | ||
| if (( found == 0 )); then | ||
| printf '%s=%s\n' "$key" "$value" >> "$tmp" | ||
| fi | ||
| mv "$tmp" .env | ||
| } | ||
| update_env CHAINGUARD_ORG "$ORG" | ||
| update_env HARBOR_ENABLED "$HARBOR_ENABLED" | ||
| update_env PULL_REGISTRY "$PULL_REGISTRY" | ||
| update_env PUSH_REGISTRY "$PUSH_REGISTRY" | ||
| # Persist the Harbor admin password if Harbor is enabled. Default to the | ||
| # chart's own default ("Harbor12345") so an unmodified first-run works, but | ||
| # allow user override by pre-setting HARBOR_ADMIN_PASSWORD in .env. Same | ||
| # value drives the Helm chart, the Terraform harbor provider, and cgLogin's | ||
| # Mode C push-auth — keeping them all wired off one env var means there's | ||
| # only one knob to rotate. | ||
| if [[ "$HARBOR_ENABLED" == "true" ]]; then | ||
| HARBOR_ADMIN_PASSWORD="${HARBOR_ADMIN_PASSWORD:-Harbor12345}" | ||
| # Same sanitize/validate rules as the other prompted values — whitespace | ||
| # or quotes here would break .env sourcing, envsubst into the Helm | ||
| # values template, the harbor Terraform provider tfvar, and the base64 | ||
| # auth string cgLogin builds for Mode C. The demo's threat model | ||
| # doesn't need passphrase-strength characters here (chart's default | ||
| # is `Harbor12345` and the cluster is localhost-only), so the | ||
| # restricted character set is acceptable. | ||
| HARBOR_ADMIN_PASSWORD=$(sanitize_env_value "$HARBOR_ADMIN_PASSWORD") | ||
| validate_env_value HARBOR_ADMIN_PASSWORD "$HARBOR_ADMIN_PASSWORD" || exit 1 | ||
| update_env HARBOR_ADMIN_PASSWORD "$HARBOR_ADMIN_PASSWORD" | ||
| fi |
Comment on lines
+14
to
+20
| settings.configure( | ||
| DEBUG=False, | ||
| SECRET_KEY="demo-not-secret", | ||
| ROOT_URLCONF=__name__, | ||
| ALLOWED_HOSTS=["*"], | ||
| INSTALLED_APPS=["django.contrib.contenttypes", "django.contrib.auth"], | ||
| MIDDLEWARE=[], |
Three Copilot findings from the thirteenth pass on PR chainguard-demo#330: - harbor/deploy.sh: pass an explicit allowlist to every `envsubst` call. Without it, envsubst expands every `$VAR` sequence in the template, so any future secret containing a `$`-prefixed substring (HARBOR_ADMIN_PASSWORD, PULL_PASS) could be silently mangled. Now each call lists just the variables it actually templates. - setup.sh: `chmod 600 .env` after persisting config. Once HARBOR_ADMIN_PASSWORD lands here it's secret material; a default umask of 022 would leave it group/world-readable. - python312-pip-django/app.py: read SECRET_KEY from DJANGO_SECRET_KEY with a demo-only fallback, and narrow ALLOWED_HOSTS from `["*"]` to `["localhost", "127.0.0.1", "testserver"]`. The Test stage's Django Client uses Host: testserver, so the smoke test still passes (verified locally against an in-memory venv). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot re-flagged the JCasC config path in the cgSign/cgVerify error messages: the round-6 "./jenkins/casc/jenkins.yaml under the jenkins/ subdir" wording was ambiguous about which root the path is relative to. Use the unambiguous repo-relative path "jenkins/jenkins/casc/jenkins.yaml" to match cgImage.groovy's matching error message from round 13. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+85
to
+90
| docker run --rm --network host \ | ||
| -v "$COSIGN_KEY_FILE:/cosign.key:ro" \ | ||
| -v "$DOCKER_CONFIG:/jenkins-docker:ro" \ | ||
| -e "COSIGN_PASSWORD=$COSIGN_PASSWORD" \ | ||
| -e DOCKER_CONFIG=/jenkins-docker \ | ||
| --entrypoint=/usr/bin/cosign \ |
…ling, pub 644 Three Copilot findings from the fifteenth pass on PR chainguard-demo#330: - cgSign: pass COSIGN_PASSWORD by name (-e COSIGN_PASSWORD) so docker inherits it from this shell's env rather than embedding the value on the docker-run command line (where `ps` and the docker event log would briefly see it). withCredentials already exported the variable into the shell env. - teardown.sh: capture `docker compose down` failure instead of letting `set -e` abort the rest of teardown. Otherwise a transient daemon issue would leave /tmp/cgjenkins-home and generated state files behind. Continue cleanup, surface the failure at the end via exit code, same pattern as TF_DESTROY_FAILED. - setup.sh: make cosign.pub chmod 644 (key + password stay 600). The public key is used outside Jenkins for manual verify (per the README example), so the host user needs to read it without sudo or a container helper — and it's not secret material. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+69
to
+73
| REPO="${IMAGE%:*}" | ||
| DIGEST=$(docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE" | grep -F "${REPO}@" | head -1) | ||
| if [ -z "$DIGEST" ]; then | ||
| echo "cgSign: could not resolve digest for $IMAGE under repo $REPO (was it pushed?)." >&2 | ||
| exit 1 |
Comment on lines
+46
to
+51
| REPO="${IMAGE%:*}" | ||
| DIGEST=$(docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE" | grep -F "${REPO}@" | head -1) | ||
| if [ -z "$DIGEST" ]; then | ||
| echo "cgVerify: could not resolve digest for $IMAGE under repo $REPO (was it pushed?)." >&2 | ||
| exit 1 | ||
| fi |
Comment on lines
+20
to
+33
| set -euo pipefail | ||
|
|
||
| cd "$(dirname "$0")" | ||
|
|
||
| # Pick up CHAINGUARD_ORG (and any prior choices) from .env if present. | ||
| if [[ -f .env ]]; then | ||
| set -a | ||
| # shellcheck disable=SC1091 | ||
| source .env | ||
| set +a | ||
| fi | ||
|
|
||
| JENKINS_URL="${JENKINS_URL:-http://localhost:8080}" | ||
| JENKINS_OIDC_ISSUER="${JENKINS_OIDC_ISSUER:-https://localhost:8080/oidc}" |
Three Copilot findings from the sixteenth pass on PR chainguard-demo#330: - cgSign + cgVerify: round 12 added `set -eu -o pipefail` but kept the `docker inspect | grep | head -1` pipeline intact — if grep finds no matching RepoDigest, the pipeline exits non-zero and set -e abandons the script before the friendly "could not resolve digest" message runs. Split the inspect from the filter so a grep-no-match is tolerated (`|| true`), then validate $DIGEST and emit the actionable error. Real `docker inspect` failures still abort (the inspect call is its own statement now, no pipe). - setup.sh: add an up-front dependency check for docker, curl, openssl, python3, terraform, chainctl. Otherwise a missing tool produces a confusing `command not found` partway through bootstrap. Harbor-only tools (kind, kubectl, helm, envsubst) stay in harbor/deploy.sh — no point demanding them up front if the user picks the non-Harbor path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+45
to
+48
| else | ||
| echo "==> kind cluster '$KIND_CLUSTER_NAME' already exists, reusing it." | ||
| kubectl config use-context "kind-${KIND_CLUSTER_NAME}" | ||
| fi |
Comment on lines
+3
to
+6
| set -euo pipefail | ||
| KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-jenkins-harbor}" | ||
| if kind get clusters | grep -qx "$KIND_CLUSTER_NAME"; then | ||
| echo "==> Deleting kind cluster '$KIND_CLUSTER_NAME'..." |
Comment on lines
+3
to
+4
| Resolves logical Chainguard image tokens (e.g. `corretto-java17`, `python-3.14`) into the concrete `cgr.dev/<org>/<image>:<tag>` strings that pipelines hand to `agent { docker { image '...' } }` blocks. The `<org>` segment comes from `env.CHAINGUARD_ORG`, so org overrides flow through automatically. | ||
|
|
Comment on lines
+31
to
+32
| The `Auth` stage runs `cgLogin` — a sibling shared-library var that exchanges a per-build Jenkins OIDC token for a short-lived Chainguard session and writes a fresh docker config to `$DOCKER_CONFIG`. It must precede any `agent { docker { } }` stage so the docker-workflow plugin picks up the new creds when it pulls the agent image. | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
... i'll be deleting the PR, squashing the myriad of commits, and opening a more sane PR soon