Skip to content

fix(conformance): certify gates fail closed (deps install, whole-tree contract drift via SDK regeneration), keep the GitHub token out of app code across certify and the CI reusables, and move conditional shell out of run: blocks [FND-4244] - #4731

Merged
cmgrote merged 22 commits into
mainfrom
christophergrote/fnd-4244
Nov 9, 2026

Conversation

@cmgrote

@cmgrote cmgrote commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes FND-4244, FND-4538, FND-4539 FND-4700, FND-4739, FND-4740 and FND-4744. #4738 (FND-4538), #4736 (FND-4539 + FND-4700, with #4757 squashed in), #4787 (FND-4739), #4790 (FND-4740) and #4792 (FND-4744) are merged into this branch. Do not merge until the rollout prerequisites are done: callers use build-and-publish-app.yaml@main and the reusables and composites @main, so everything here goes live for about 71 app repos the moment it lands.

Summary

Issue Change
FND-4244 A test guards the "no conditional shell in run:" rule. The three worst files are migrated into tested Python scripts.
FND-4538 Certify fails closed when an app's dependencies don't install. It also pins Python to the app image's version and adds an apt_packages input.
FND-4539 The contract-drift gate regenerates with the SDK's own regeneration, not the app's poe generate, and fails closed when the contract doesn't regenerate.
FND-4700 The contract-drift gate checks the whole working tree, including untracked output, not just generated/.
FND-4739 The GitHub token reaches only the dependency-install process, never app code. The certify job token is read-only.
FND-4740 The same for the CI reusables (tests, pre-commit, conformance, version bump): a read-only App token, passed only to the command that fetches the private dependency.
FND-4744 The image builds' git_token BuildKit secret is a read-only App token, not ORG_PAT_GITHUB.

FND-4244: guard the no-conditional-shell rule and migrate the worst workflows

docs/standards/ci.md bans branching shell in run: blocks, but nothing checked it.

Guard: .github/scripts/tests/test_no_conditional_run_blocks.py

  • It reads every workflow and every composite action.y*ml. actionlint skips composite actions, so they had no check before.
  • It fails on any if/for/while/until/case/select, and on a test chained with &&/|| ([ … ] && x, test -n "$X" || { …; }).
  • Comments, quoted strings and ${{ }} are blanked before matching. Inline heredoc Python is scanned too.
  • It runs in the existing CI script tests workflow, which has no path filter, so it runs on every PR.

Allowlist: .github/scripts/conditional_run_allowlist.txt

  • Seeded with the 72 existing conditional blocks; 39 remain after this PR.
  • It can only shrink. A stale entry fails, and ALLOWLIST_CEILING caps the size.

Migrations (stdlib Python; every external command goes through a stub-able wrapper, and every branch has a test):

File Blocks Now in
build-and-publish-app.yaml 18 .github/scripts/build_and_publish/{publish,unit_tests_gate,certify,leak_scan,prepare,manifest}.py
.github/actions/sdr-e2e/action.yaml 10 .github/actions/sdr-e2e/{sdr_steps,sdr_report}.py
check-dapr-version.yaml 5 .github/scripts/sync_dapr_version.py
  • The sdr-e2e scripts live in the action's own directory. The action copies that directory to /tmp/sdr-e2e before setup-deps wipes the workspace.
  • repin-application-sdk.sh now has a single copy, in connector-integration-tests/. A test enforces that.

FND-4538: fail the certify gate closed when an app's dependencies don't install

When uv sync failed, contract drift and unit tests were skipped and the verdict still passed. Only a "Certification incomplete" warning appeared.

  • certify.py verdict now blocks publish when the app install fails, in certify or in any certify-unit leg. An app with no tests/unit still passes.
  • certify_unit_shards.py verdict reports install_failed instead of folding it into skipped.
  • certify and certify-unit set UV_PYTHON from container_python_version.py print-version, which matches the app image's Python. The runner's system Python is newer, and wheel-less pins (for example pyarrow==20.0.0) built from source and failed or timed out.
  • New apt_packages input (default empty), installed before uv sync in both jobs. The guard is byte-identical to tests-reusable's.

FND-4539: drift regenerates with the SDK, never the app's poe generate

The gate used to run each app's own poe generate. That made it a second generator, separate from the one the generated-freshness check and Renovate's toolkit bump use, so apps could pass one check and fail the other. It also ran app-defined shell in a job that holds write tokens.

The gate now:

  • calls renovate_pkl_sync.regenerate, the exact function check_generated_freshness.py uses: pkl eval, then contract/post-generate.sh if present, then ruff on generated Python. A test pins that it is the same function.
  • never runs poe, and doesn't require it. A wiring test fails if any certify step runs poe.
  • fails closed when the contract doesn't regenerate, or when the regeneration can't run.
  • runs even if the app's dependencies didn't install, because it only needs pkl. The pkl CA-bundle seed step existed only for poe tasks, so it's removed.

Any post-processing an app does to generated output must now live in the contract's output paths or in contract/post-generate.sh.

FND-4700: the drift gate checks the whole working tree

The stale check was git diff --exit-code -- generated/ from the app root, which missed two things:

  • The pathspec matched only a top-level generated/, not app/generated/ or root files like atlan.yaml.
  • git diff ignores untracked files.

After regeneration, the gate now fails when git status --porcelain --untracked-files=all is non-empty. The ::error:: lists the paths (capped at 50), never their contents. Gitignored files don't count.

FND-4739: keep the GitHub token out of app code (squashed from #4787)

certify and certify-unit used to write the fleet App token into the runner's global git config, falling back to ORG_PAT_GITHUB when the App token was missing. It stayed there for the rest of the job, so unit tests and coverage could read it. certify also inherited packages, actions and pull-requests: write, and its checkouts left the job token in .git/config.

  • The install step is the only step with git auth, and only git sees the token. certify.py install-deps runs through the shared .github/scripts/git_auth.py: it mirror-clones the app's private deps with the token, then runs uv sync against the local mirrors with no token in uv's env (see Token never reaches uv). Nothing is written to git config, and the separate "Configure git auth" steps are gone.
  • The install syncs --all-groups. Tests and coverage run uv run --frozen, so nothing after the install fetches.
  • No ORG_PAT_GITHUB fallback for the install credential. Build backends run during the install, and the PAT is org-admin. The SDK checkout keeps its fallback, because no app code runs there.
  • The App token is minted with permission-contents: read. certify's job token is contents: read, and both its checkouts use persist-credentials: false.

Inputs and secrets are unchanged for callers.

FND-4740: keep the GitHub token out of app code in the CI reusables (squashed from #4790)

For an app with a private atlanhq git dependency, the unit, integration and e2e jobs in tests-reusable.yaml, pre-commit in checks-reusable.yaml, the conformance suite and release-version-bump.yaml wrote ORG_PAT_GITHUB (org-admin) into ~/.gitconfig before uv sync. Every later step could read it, including tests, pre-commit hooks and conformance series that import app code. All six jobs also exposed the PAT as job-level env: GIT_TOKEN, opted in or not.

  • Read-only App token, no PAT fallback. Each job mints a fleet App token with permission-contents: read, only when the caller opts in. If it doesn't mint, a guard step fails naming FLEET_APP_ID and FLEET_APP_PRIVATE_KEY.
  • Only git sees it. .github/scripts/git_auth.py mirror-clones the private deps with the token and runs the fetching command against the mirrors with no token in its env (see Token never reaches uv). Nothing is persisted. Once one authenticated sync has filled the uv cache and venv, later uv sync / uv run / uv lock --check fetch nothing (checked against a blocked github.com).
  • Composites. setup-deps, connector-unit-tests, connector-integration-tests and sdr-e2e take an optional git-auth-token input. A caller without one installs exactly as before.
  • Conformance runs a plain uv sync before the D series, so the vendored detect action never sees the token.
  • release-version-bump moves uv lock into its own step; the "drop rewrite before push" step is gone.
  • docs/standards/ci.md gets a section on handling a token for a private git dependency, and the bootstrap templates name the App secrets instead of ORG_PAT_GITHUB.

Out of scope: the bump job's checkout keeps its repo-scoped token in .git/config. The Docker build secret is covered by FND-4744 below.

FND-4744: read-only App token for the image build's git_token secret (squashed from #4792)

The Docker builds passed ORG_PAT_GITHUB as the git_token BuildKit secret, which a Dockerfile reads to clone a private atlanhq git dependency during uv sync. The mount keeps it out of the image, but not out of the RUN: every build backend that uv sync runs could read the org-admin PAT.

  • Each image build mints a read-only fleet App token (owner: atlanhq, permission-contents: read, continue-on-error, no PAT fallback) and passes it as the secret. This covers tests-reusable.yaml's build-e2e-image and e2e (git-token), build-and-publish-app.yaml's build, and both attempts in build-and-scan.yaml's build.
  • Minted on every build, not behind the opt-in flag. build-and-scan.yaml has no flag, and several callers build private-dependency images without setting private_git_auth, which gates certify only. A Dockerfile without the secret mount ignores it. If a mint fails, docker/build-push-action skips the empty secret with a warning, so only a build that needs the token fails, at its clone.
  • GHCR login and push stay on ORG_PAT_GITHUB. Fleet packages are org-level, and the App only has repo-level Packages permission.

Inputs and secrets are unchanged for callers.

Token never reaches uv or a build backend (lens rounds 5-6)

uv sync and uv lock run PEP 517 build backends as the job's user. They inherit uv's environment and can read every ancestor's (/proc/<pid>/environ). So the token must be in neither. Now:

  • A mirror-only step. git_auth.py mirror is the only step given the token. It mirror-clones each private atlanhq repo the app names into $RUNNER_TEMP/git-mirrors and exits. A bare mirror clone runs no repository code, and git stores the remote URL as written, so the token isn't left in the mirror.
  • Then a token-free run step. git_auth.py run -- <cmd> runs in a later step with no token, pointing those remotes at the mirrors with a token-free url.file://<mirror>.insteadOf. It refuses to start if GIT_TOKEN is set. This covers setup-deps, the lock checks, the conformance pre-sync, the version-bump re-lock and certify's install. The composites' SDK repins reuse setup-deps' mirrors. Public atlanhq repos (e.g. application-sdk as a git source) are fetched from GitHub as before.
  • Scoped mint, inline. Each mint checks out git_auth.py at job.workflow_sha, runs git_auth.py repos (every pyproject.toml and uv.lock in the checkout, minus repos an unauthenticated git ls-remote can read), removes the checkout from the workspace, and mints a read-only App token scoped to exactly that list. It skips the mint when the list is empty, because create-github-app-token with no repositories grants the whole installation. Every private-deps and image-build mint uses this shape (_scoped_mint.py pins it). It's inline rather than a composite because callers pin composites @main, so a new one can't run before merge, and composite steps can't continue-on-error. certify's SDK-checkout token is scoped to application-sdk.
  • Allowlist (lens round 7). A PR controls its own pyproject.toml / uv.lock, so the scope is also capped by ALLOWED_PRIVATE_DEPS in git_auth.py, read from the workflow's own commit: atlan-query-intelligence-app (redactors, 9 connectors), sqlglot-for-lorien (Atlan's SQLGlot fork, QI app only), and two marked for near-term migration: server-sdk (to atlan-application-sdk-api) and ai-application-sdk (the metering client, 3 apps on 3 different raw commits). repos and mirror both fail closed on any other private repo. atlanhq/common-app-server is the one aggregator allowed to name any, matched on GITHUB_REPOSITORY. Checked against 111 fleet repos: none refused.
  • release-version-bump runs plain uv lock when private_git_auth is off, so a caller pinning sdk_scripts_ref to a revision without git_auth.py keeps working.
  • Checked end to end with real uv 0.13 and git: uv lock and uv sync --frozen install a git dep (hatchling build) from the mirror over both ssh:// and https:// forms, and the lock keeps the original URL.

Things to review

  • Security-relevant (FND-4739):
    • A caller with private_git_auth: true but no FLEET_APP_* secrets used to fall back to the PAT. It now fails certify. Every known caller uses secrets: inherit.
    • Tests now run with uv run --frozen against the locked dependencies. Before, a stale uv.lock could be re-locked during the test step.
    • Build backends that run during uv sync no longer see the token, in uv's env or any ancestor's (lens rounds 5-6): only the mirror step holds it, and it exits before uv runs.
  • Security-relevant (FND-4740):
    • A reusable caller with the private-git-deps flag but no FLEET_APP_* secrets now fails instead of falling back to the PAT. All 9 callers that set the flag pass the App secrets.
  • Security-relevant (FND-4744):
    • Every image build mints a read-only App token scoped to the private repos the app names, whether or not the Dockerfile uses it, and none when it names none. It lives for one hour and only reaches the build secret. This is the one place caller code (the Dockerfile's RUN) can read a token.
    • If the mint fails, a build whose Dockerfile needs the secret fails at its clone with a git auth error. It used to get the PAT.
  • Security-relevant: release_tag in "Compute image tags" is now passed through env: instead of being written into the script. This removes a script-injection path.
  • New checkouts: validate-channel and unit-tests-gate get the same sparse .sdk-entrypoint checkout the file already uses. certify-unit's sparse checkout now includes /Dockerfile.
  • Small, deliberate behaviour changes:
    • check-dapr escapes the version before matching. The old sed treated . as a wildcard.
    • sdr-e2e's restore step only runs if the stash step succeeded.
    • The gitleaks Slack alert still fires when the report can't be read (count ?).
    • Three except Exception: pass blocks in the sdr report code now print one line naming the exception.

Blast radius (measured 2026-11-05)

Contract drift (FND-4539 + FND-4700). Certify's new drift check (SDK regeneration from a depth-1 clone, then a whole-tree git status) was run against the default branch of all 106 caller apps with a pkl contract. The app's dependencies aren't needed, so none were installed.

Result Apps Status
✅ Clean 92 —
🟡 Stale formatting a Renovate bump kept 3: mysql, discovery, search atlanhq/atlan-mysql-app#969, atlanhq/atlan-discovery-app#20, atlanhq/atlan-search-app#110
🔴 Post-processing only in poe generate 3: generic-miner (second root + merge.py), auth (hand-authored atlan.yaml), notification (contract/generated/ layout) notification: moved into contract/post-generate.sh with no output change, atlanhq/atlan-notification-app#28. generic-miner: FND-4741 (merged), which folds the credential root into contract/app.pkl. auth: FND-4742, which generates atlan.yaml from app.pkl.
🔴 Contract emits only atlan.yaml, and app/generated/ holds files it no longer emits 2: csa-uber, propagator-uber The regeneration refuses and names the stale files (#4788, FND-4736). App fixes: FND-4752 (csa-uber), FND-4753 (propagator-uber).
➖ Not publishing since 2026-07 6: conversational-ai, cube-assets-builder, notification-tester, se-apps, csa-uber-import, wdcustom No PR

bigquery and redshift are clean only because their post-generate.sh runs. It calls python, so it needs python on PATH; GitHub runners have it.

Install credential (FND-4739 + FND-4740). No app changes left. 9 fleet repos set the private-git-deps flag: six already passed the App secrets, and the other three gained them in atlanhq/atlan-bigquery-app#712, atlanhq/atlan-snowflake-app#872 and atlanhq/atlan-tableau-app#356, all merged. A caller without the FLEET_APP_* secrets now fails instead of falling back to the PAT.

Image build secret (FND-4744). No app changes. 15 fleet repos have a Dockerfile that mounts git_token, and every one of their callers of build-and-publish-app.yaml / build-and-scan.yaml uses secrets: inherit. Each one's Dockerfile uses the token only for uv sync, so the scoped token covers it: the private repos come from the app's pyproject.toml and uv.lock files, including sub-packages (memory-app's server/ names server-sdk).

Rollout

This PR is the switch: the gates go live for every caller when it merges.

  1. Before merging:
  2. Merge this PR.
  3. Straight after: merge atlanhq/atlan-hive-app#492. It passes apt_packages for pykerberos. It can't merge earlier, because that input doesn't exist on main until this PR lands.

Testing

  • pytest .github/scripts/tests:
    • FND-4244's branch passed in full (7184 passed).
    • The certify suites (test_build_and_publish_certify.py, test_build_and_publish_certify_wiring.py, test_certify_unit_shards.py) pass on the combined tree: 78 passed.
    • The full suite hasn't been run locally on the combined tree. CI's pytest .github/scripts/tests check covers that.
    • Run locally, tests that create tags need tag.gpgsign=false, because the author machine's global git config signs tags.
  • Red/green against the parent of each change:
    • FND-4244: the regex-escape test fails without the escape.
    • FND-4538: 29 new or changed tests fail against the parent.
    • FND-4539: 2 new assertions fail against the parent.
    • FND-4700: 6 fail, including the stale app/generated/ file, untracked file and root atlan.yaml cases.
    • FND-4740: test_private_git_deps_wiring.py has 51 tests; 38 fail on the old workflows. test_git_auth.py runs the mirror path against real git and a real child process, which fetches through the rewrite and finds no token in its env. The lens round 5 tests (scoped mint, mirror, release-bump split) all fail on the previous commit.
    • FND-4744: test_image_build_git_token.py has 23 tests; 10 fail on the old workflows.
  • The composites are pinned @main, so the reusables' end-to-end token path can only be exercised after merge.
  • Pre-commit is clean on every changed file, including actionlint and pyright.

Not done here: the remaining 39 allowlisted conditional blocks, and turning shellcheck back on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AbKqGboFK2UACRjz2eg4FV

@linear

linear Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

@cmgrote

cmgrote commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

/lens

@atlan-app-fleet

atlan-app-fleet Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

lens summary · updated after round 10 (re-review · only commits since 0c8e21d)

✅ Ready to merge — every finding is resolved

Open findings: 🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)

lens reads this PR as — From the intent and changed workflow/script symbols, this PR addresses fail-open certification when dependency installation fails, contract-drift checks that could miss generated changes, unguarded conditional shell in CI, and overly broad GitHub credentials reaching dependency-build paths. It also applies the credential change to reusable CI workflows and image builds. How: It makes install outcomes part of the certification verdict, regenerates contracts through the SDK’s shared regeneration function and checks the full working tree, adds a CI guard while moving branching workflow logic into Python scripts, and replaces broad tokens with read-only App tokens scoped to declared private repositories. The mirror-clone step is separated from dependency installation so app build backends and later test steps do not inherit that token.

Approach check — sound.

✔️ Resolved (21)
id severity where finding found resolved how
F-04dbed 🟠 high .github/scripts/build_and_publish/leak_scan.py:169 Fail closed when the Gitleaks report is unavailable round 1 round 2 its code was removed or rewritten
F-23961e ⚪ low .github/scripts/tests/test_no_conditional_run_blocks.py:55 The detector misses test chains split after a closing bracket round 1 round 2 verified fixed
F-3efbb9 🟡 medium .github/scripts/build_and_publish/publish.py:236 Keep the publish token out of curl's argv round 1 round 2 its code was removed or rewritten
F-66280f 🟡 medium .github/actions/sdr-e2e/sdr_report.py:62 Duplicate scenario names collapse multi-class grouping round 1 round 2 its code was removed or rewritten
F-a1d718 🔴 critical .github/scripts/build_and_publish/leak_scan.py:104 Keep the Slack webhook out of curl's argv round 1 round 2 its code was removed or rewritten
F-c3f000 🟡 medium .github/scripts/sync_dapr_version.py:41 Bound network-backed subprocesses with a timeout round 1 round 2 verified fixed
F-d56f6a ⚪ low .github/scripts/tests/test_no_conditional_run_blocks.py:160 The allowlist can replace migrated entries without shrinking round 1 round 2 its code was removed or rewritten
F-e4173f 🟡 medium docs/standards/ci.md:58 The ceiling does not enforce a shrinking allowlist round 1 round 2 its code was removed or rewritten
F-f28147 🟡 medium .github/scripts/sync_dapr_version.py:153 Empty PR listings are treated as a real PR round 1 round 2 its code was removed or rewritten
F-f78d4f ⚪ low .github/scripts/tests/test_no_conditional_run_blocks.py:62 Trailing shell comments can be misreported as conditional code round 1 round 2 its code was removed or rewritten
F-24f867 🟠 high .github/scripts/build_and_publish/certify.py:79 Do not pass the enforced drift gate when poe is unavailable round 1 round 3 its code was removed or rewritten
F-5069cc 🟠 high .github/scripts/build_and_publish/certify.py:158 Treat skipped enforced certification checks as blocking round 1 round 3 its code was removed or rewritten
F-1d73a0 🔴 critical .github/workflows/conformance-reusable.yaml:865 The build backend still inherits the private-read token round 5 round 6 verified fixed
F-3d63a7 🔴 critical .github/scripts/git_auth.py:98 Build hooks inherit the private Git token round 5 round 6 verified fixed
F-6613be 🔴 critical .github/actions/setup-deps/action.yaml:168 Build hooks can read the private-dependency token round 5 round 6 verified fixed
F-843ef1 🔴 critical .github/workflows/tests-reusable.yaml:1346 Build backends can read the private-dependency token round 5 round 6 verified fixed
F-9cd936 🟠 high .github/workflows/build-and-scan.yaml:351 Scope the read token before sending it into the PR build round 5 round 6 its code was removed or rewritten
F-e8ef5d 🟠 high .github/workflows/release-version-bump.yaml:201 This unconditional step breaks callers pinned to older SDK refs round 5 round 6 its code was removed or rewritten
F-ebff0b 🔴 critical .github/scripts/git_auth.py:319 The token-bearing wrapper remains readable by app code round 6 round 7 verified fixed
F-6e4829 🔴 critical .github/workflows/checks-reusable.yaml:209 A PR can expand the token to any installed private repo round 7 round 8 re-checked at @cmgrote's request: merge-base copy of .github/workflows/checks-reusable.yaml could not be parsed — The current workflow limits the token's repository input: code_now lines 209–13 state that git-deps-repos.outputs.repositories is capped by ALLOWED_PRIVATE_DEPS from the workflow's own commit, and that the precedin
F-b29afb 🟠 high .github/workflows/build-and-publish-app.yaml:223 SDK helper checkouts use the caller token and fail round 9 round 9 dismissed by @fyzanshaik-atlan: False positive: application-sdk is public; the identical tokenless checkout lens quotes (L197-204, build-decision) is unchanged from main and succeeds in consumer runs (atlan-mysql-app 42560901721, atlan-bigquery-app 42258810961).
🕘 Round history (17)
round run commits new resolved blocking open cost log
1 first review d8a2e5c..434662a 12 0 4 $0.165 run
1 recheck · F-24f867, F-5069cc by @cmgrote 434662a 0 0 4 $0.003 run
2 re-review · only commits since 434662a 434662a..88e9a66 0 10 2 $0.038 run
3 re-review · only commits since 88e9a66 88e9a66..8c77ec8 0 2 0 $0.050 run
4 re-review · only commits since 8c77ec8 8c77ec8..5e910c0 0 0 0 $0.020 run
5 re-review · only commits since 5e910c0 5e910c0..7f661c7 6 0 6 $0.148 run
6 re-review · only commits since 7f661c7 7f661c7..5b9b967 1 6 1 $0.163 run
7 re-review · only commits since 5b9b967 5b9b967..21bbd0a 1 1 1 $0.120 run
8 re-review · only commits since 21bbd0a 21bbd0a..4f69f82 0 0 1 $0.029 run
8 recheck · F-6e4829 by @cmgrote 4f69f82 0 1 0 $0.003 run
9 re-review · only commits since 4f69f82 4f69f82..0c8e21d 1 0 1 $0.041 run
9 recheck · F-b29afb by @cmgrote 0c8e21d 0 0 1 $0.002 run
9 recheck · F-b29afb by @cmgrote · ⚠️ incomplete 0c8e21d 0 0 1 $0.004 run
9 recheck · F-b29afb by @cmgrote · ⚠️ incomplete 0c8e21d 0 0 1 $0.004 run
9 recheck · F-b29afb by @cmgrote 0c8e21d 0 0 1 $0.007 run
after 9 dismiss · F-b29afb by @fyzanshaik-atlan 0c8e21d 0 1 0 $0.000 run
10 re-review · only commits since 0c8e21d 0c8e21d..57e37fb 0 0 0 $0.010 run

round 10 · $0.806 of $1.00 · 302 model calls · 0 failed requests · 75% prompt cache hits · gpt-6-luna

…tion

Keep the Slack webhook and publish token out of curl argv, fail the
gitleaks gate closed on an unreadable report, return no PR number when
none is open, bound sync_dapr_version's commands with timeouts, group
SDR workflow links by per-class summary file, and tighten the guard's
comment, continuation and ceiling handling.
@cmgrote

cmgrote commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck F-24f867 F-5069cc Both behaviours are unchanged from the merge base. This PR moves inline shell into scripts and does not change behaviour. On main, .github/workflows/build-and-publish-app.yaml:753 warns and exits 0 when poe is unavailable, and :867 and :893 warn on a failed app install and exit 1 only when CD or UT is "failure". certify.py reproduces this exactly. Failing closed here would block publishing for several fleet apps today. It is a deliberate gate-policy change, tracked separately in Linear, and is not a regression introduced by this PR.

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · re-check after round 1 · F-24f867, F-5069cc by @cmgrote — 4 blocking: F-a1d718, F-04dbed, F-24f867, F-5069cc

Open findings: 🔴 1 critical · 🟠 3 high · 🟡 5 medium · ⚪ 3 low (nit)

Approach check — ⚠️ concerns (advisory)

  • Problem: The CI standard forbids branching logic in inline workflow and composite-action run: blocks, but no automated check enforced that rule; the three targeted files still held substantial branching shell.
  • How the PR solves it: The PR adds an always-run pytest guard over workflow and composite-action run: blocks, grandfathering the remaining exceptions, and moves the larger branch-heavy sections into tested Python drivers with stub-able command wrappers. That fits the repository’s existing CI-script convention.
  • ⚠️ The allowlist is capped, but not guaranteed to shrink (A1) — The description says the allowlist “can only shrink,” but .github/scripts/tests/test_no_conditional_run_blocks.py::test_allowlist_only_shrinks only enforces len(entries) <= 39. A change can remove a now-stale key and replace it with a new conditional-block key at the same count; the stale-entry and unlisted-block checks would both pass without lowering the ceiling. Instead: Track the grandfathered keys so any newly added key is rejected even when another exception is removed, or describe the guarantee as a size cap rather than strict shrinkage.

$0.168 of $1.00 · 58 model calls · 0 failed requests

Full summary · Run log

  • ❌ F-24f867 still stands: The evidence says certify.py and contract_drift are absent from the merge base, with no in-file/import callers; that rules out a pre-existing contract, but not this code-level failure. In code_now, when contract/app.pkl exists and poe --help fails, contract_drift() warns and returns 0, so the enforced drift check can report success without running.
    • absent on merge base: .github/scripts/build_and_publish/certify.py (this PR adds the file)
    • public: module .github.scripts.build_and_publish.certify, symbol contract_drift
    • not listed in docs/agents/sdk-capabilities.md
    • callers of contract_drift at the PR head (0 in its file or importing it): none
  • ❌ F-5069cc still stands: The evidence likewise says certify.py and verdict are absent from the merge base, with no in-file/import callers; that does not cure the result produced by the new code. In code_now, verdict() labels contract drift and unit tests enforced, but returns 0 for CD=UT=skipped; the stated app-install-failure path can therefore pass without either enforced check running.
    • absent on merge base: .github/scripts/build_and_publish/certify.py (this PR adds the file)
    • public: module .github.scripts.build_and_publish.certify, symbol verdict
    • not listed in docs/agents/sdk-capabilities.md
    • callers of verdict at the PR head (0 in its file or importing it): none

@cmgrote

cmgrote commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

/lens

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · round 2 · re-review · only commits since 434662a — 2 blocking: F-24f867, F-5069cc

Open findings: 🔴 0 critical · 🟠 2 high · 🟡 0 medium · ⚪ 0 low (nit)
Resolved this round: 10

Approach check — ✅ concerns addressed

  • Problem: The CI standard forbids branching logic in inline workflow and composite-action run: blocks, but no automated check enforced that rule; the three targeted files still held substantial branching shell.
  • How the PR solves it: The PR adds an always-run pytest guard over workflow and composite-action run: blocks, grandfathering the remaining exceptions, and moves the larger branch-heavy sections into tested Python drivers with stub-able command wrappers. That fits the repository’s existing CI-script convention.
  • ✔️ The allowlist is capped, but not guaranteed to shrink (A1) — addressed in round 2

$0.206 of $1.00 · 75 model calls · 0 failed requests

Full summary · Run log

@cmgrote cmgrote changed the title ci: guard the no-conditional-shell rule and migrate the worst workflows (FND-4244) fix(ci): certify gates fail closed (deps install, missing poe, whole-tree contract drift) and conditional shell leaves run: blocks [FND-4244, FND-4538, FND-4539, FND-4700] Nov 5, 2026
@cmgrote cmgrote changed the title fix(ci): certify gates fail closed (deps install, missing poe, whole-tree contract drift) and conditional shell leaves run: blocks [FND-4244, FND-4538, FND-4539, FND-4700] ci: certify gates fail closed (deps install, missing poe, whole-tree contract drift) and conditional shell leaves run: blocks [FND-4244, FND-4538, FND-4539, FND-4700] Nov 5, 2026
The pyright baseline from main (FND-4245) fails any finding in a file it
does not list, and every script and test this PR adds was new to it.

- Tests import build_and_publish modules through the package under
  TYPE_CHECKING, so pyright resolves them; the runtime imports are unchanged.
- The sdr-e2e tests load their modules with importlib: the action directory
  has a hyphen and cannot be a package.
- Annotate test parameters and type the parsed JSON/YAML data.
@cmgrote

cmgrote commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

/lens

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

✅ lens · round 3 · re-review · only commits since 88e9a66 — ready to merge — every finding resolved · $0.26

Open findings: 🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)
Resolved this round: 2

Approach check — ✅ concerns addressed

  • Problem: The CI standard forbids branching logic in inline workflow and composite-action run: blocks, but no automated check enforced that rule; the three targeted files still held substantial branching shell.
  • How the PR solves it: The PR adds an always-run pytest guard over workflow and composite-action run: blocks, grandfathering the remaining exceptions, and moves the larger branch-heavy sections into tested Python drivers with stub-able command wrappers. That fits the repository’s existing CI-script convention.
  • ✔️ The allowlist is capped, but not guaranteed to shrink (A1) — addressed in round 2

$0.255 of $1.00 · 92 model calls · 0 failed requests

Full summary · Run log

atlan-ci
atlan-ci previously approved these changes Nov 5, 2026

@atlan-ci atlan-ci left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lens: ready to merge — every finding at every level is resolved (round 3).

…e generate

The certify contract-drift check ran each app's own `poe generate`, so it
was a second generator that disagreed with the generated-freshness check
and Renovate's toolkit bump, and it ran app-defined shell in a job that
holds write tokens. It now calls renovate_pkl_sync.regenerate, the
function the freshness check already uses (pkl eval, then
contract/post-generate.sh, then ruff), and still requires a clean
working tree afterwards.

- No poe requirement: an app is no longer failed for lacking poethepoet.
- A contract that does not regenerate, or a regeneration that cannot
  run, fails closed.
- Drift needs pkl, not the app's dependencies, so it no longer waits on
  the app install; the pkl CA-bundle seed step (only for poe tasks) goes.
- Tests pin that certify uses the freshness check's exact function and
  that no certify step runs poe.

Claude-Session: https://claude.ai/code/session_01AbKqGbjx7uUk6aM6jzSxNF
@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · round 8 · re-review · only commits since 21bbd0a — 1 blocking: F-6e4829

Open findings: 🔴 1 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The certification and CI gates could pass or continue with an incomplete result when app dependencies failed to install, could miss contract drift by using app-defined generation or checking only generated paths, and exposed broader GitHub credentials to code run during dependency installation or image builds. A separate CI-standard gap allowed conditional shell logic in workflow run: blocks to go unchecked.
  • How the PR solves it: The PR moves the risky inline workflow logic into tested Python scripts and adds a guard for conditional shell; certification uses the SDK’s shared renovate_pkl_sync.regenerate and checks Git’s whole working-tree status, while install failures now block the verdict. It also pins certification to the image’s Python, adds an allowlisted native-package input, and uses scoped read-only App tokens with a separate mirror-fetch step for private dependencies (and a read-only token for BuildKit).

$0.736 of $1.00 · 273 model calls · 0 failed requests

Full summary · Run log

Lens F-6e4829 is anchored at the mint's repositories: line, and its verify step only sees the lines around it, not git_auth.py where ALLOWED_PRIVATE_DEPS is enforced (4f69f82). Say at every mint what caps the scope, and name the repos step for what it does.
@cmgrote

cmgrote commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck F-6e4829

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

✅ lens · re-check after round 8 · F-6e4829 by @cmgrote — ready to merge — every finding resolved · $0.74

Open findings: 🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)
Resolved this round: 1

Approach check — ✅ sound

  • Problem: The certification and CI gates could pass or continue with an incomplete result when app dependencies failed to install, could miss contract drift by using app-defined generation or checking only generated paths, and exposed broader GitHub credentials to code run during dependency installation or image builds. A separate CI-standard gap allowed conditional shell logic in workflow run: blocks to go unchecked.
  • How the PR solves it: The PR moves the risky inline workflow logic into tested Python scripts and adds a guard for conditional shell; certification uses the SDK’s shared renovate_pkl_sync.regenerate and checks Git’s whole working-tree status, while install failures now block the verdict. It also pins certification to the image’s Python, adds an allowlisted native-package input, and uses scoped read-only App tokens with a separate mirror-fetch step for private dependencies (and a read-only token for BuildKit).

$0.738 of $1.00 · 274 model calls · 0 failed requests

Full summary · Run log

  • ✅ F-6e4829 resolved: merge-base copy of .github/workflows/checks-reusable.yaml could not be parsed — The current workflow limits the token's repository input: code_now lines 209–13 state that git-deps-repos.outputs.repositories is capped by ALLOWED_PRIVATE_DEPS from the workflow's own commit, and that the preceding repo-discovery step fails for an unlisted private repository. Token minting is gated on that output being nonempty (line 201) and passes only that output as the scope (line 213). This blocks the alleged PR-controlled expansion to arbitrary private repositories. The merge-base evidence says the workflow copy could not be parsed there; that does not establish a current bypass.

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

lens: nothing to review — round cap (5) reached; comment /lens force to review anyway.

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens force

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · round 9 · re-review · only commits since 4f69f82 — 1 blocking: F-b29afb

Open findings: 🔴 0 critical · 🟠 1 high · 🟡 0 medium · ⚪ 0 low (nit)
New this round: 1

🟠 F-b29afb · high · bug — SDK helper checkouts use the caller token and fail

Where: .github/workflows/build-and-publish-app.yaml:223

On a consumer-app run with publish: true, validate-channel reaches this cross-repository checkout without a token, so actions/checkout uses the caller app's GITHUB_TOKEN. This workflow explicitly notes that token cannot read application-sdk from a reusable workflow (and supplies an App-token/PAT fallback on its later SDK checkout), so this step fails before the channel gate runs. The same unauthenticated checkout pattern is used by other active helper-checkout steps, also blocking their jobs.

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.779 of $1.00 · 292 model calls · 0 failed requests

Full summary · Run log

atlanhq/application-sdk is public, so a consumer app's GITHUB_TOKEN can clone it from the reusable workflow; build-decision's identical tokenless checkout already succeeds in consumer runs. Correct the certify comment that claimed otherwise.
@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck F-b29afb

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · re-check after round 9 · F-b29afb by @cmgrote — 1 blocking: F-b29afb

Open findings: 🔴 0 critical · 🟠 1 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.781 of $1.00 · 293 model calls · 0 failed requests

Full summary · Run log

  • ❌ F-b29afb still stands: <code_now> checks out atlanhq/application-sdk without a token, so actions/checkout uses its default github.token; the supplied evidence does not establish that the caller token can read that cross-repository checkout. The evidence that this workflow module is public and absent from the SDK capability manifest does not establish access to the target repository. <at_merge_base> shows the same checkout pattern, and the merge-base parse failure provides no evidence of a fix, so the reported failure is not shown to be wrong or impossible.
    • merge-base copy of .github/workflows/build-and-publish-app.yaml could not be parsed
    • public: module .github/workflows/build-and-publish-app.yaml
    • not listed in docs/agents/sdk-capabilities.md

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck F-b29afb Public repo: GET /repos/atlanhq/application-sdk gives visibility=public, and any token clones a public repo. Main already has this identical tokenless checkout in build-decision, and it succeeds in consumer Build & Publish runs (atlan-mysql-app 42560901721, atlan-bigquery-app 42258810961).

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

⚠️ lens · re-check after round 9 · F-b29afb by @cmgrote — review incomplete — re-check: no verdict came back

Open findings: 🔴 0 critical · 🟠 1 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.785 of $1.00 · 294 model calls · 0 failed requests

Full summary · Run log

  • ❌ F-b29afb still stands: no verdict came back
    • merge-base copy of .github/workflows/build-and-publish-app.yaml could not be parsed
    • public: module .github/workflows/build-and-publish-app.yaml
    • not listed in docs/agents/sdk-capabilities.md

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck F-b29afb Public repo: GET /repos/atlanhq/application-sdk gives visibility=public, and any token clones a public repo. Main already has this identical tokenless checkout in build-decision, and it succeeds in consumer Build & Publish runs (atlan-mysql-app 42560901721, atlan-bigquery-app 42258810961).

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

⚠️ lens · re-check after round 9 · F-b29afb by @cmgrote — review incomplete — re-check: no verdict came back

Open findings: 🔴 0 critical · 🟠 1 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.789 of $1.00 · 295 model calls · 0 failed requests

Full summary · Run log

  • ❌ F-b29afb still stands: no verdict came back
    • merge-base copy of .github/workflows/build-and-publish-app.yaml could not be parsed
    • public: module .github/workflows/build-and-publish-app.yaml
    • not listed in docs/agents/sdk-capabilities.md

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens recheck verify-budget F-b29afb Public repo: GET /repos/atlanhq/application-sdk gives visibility=public, and any token clones a public repo. Main already has this identical tokenless checkout in build-decision, and it succeeds in consumer Build & Publish runs (atlan-mysql-app 42560901721, atlan-bigquery-app 42258810961).

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

❌ lens · re-check after round 9 · F-b29afb by @cmgrote — 1 blocking: F-b29afb

Open findings: 🔴 0 critical · 🟠 1 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.796 of $1.00 · 296 model calls · 0 failed requests

Full summary · Run log

  • ❌ F-b29afb still stands: code_now lines 197–204 check out atlanhq/application-sdk without an explicit token, so the action defaults to the caller's github.token. The supplied evidence marks the workflow module as public and says it is not in the SDK capability manifest, but it does not establish the visibility or caller-token access of the separate application-sdk checkout target, nor verify the consumer-run results asserted in author_claim; the merge-base parse failure supplies no such confirmation either. The claim that this checkout cannot read the target therefore remains possible.
    • merge-base copy of .github/workflows/build-and-publish-app.yaml could not be parsed
    • public: module .github/workflows/build-and-publish-app.yaml
    • not listed in docs/agents/sdk-capabilities.md

@fyzanshaik-atlan

Copy link
Copy Markdown
Contributor

/lens dismiss F-b29afb False positive: application-sdk is public; the identical tokenless checkout lens quotes (L197-204, build-decision) is unchanged from main and succeeds in consumer runs (atlan-mysql-app 42560901721, atlan-bigquery-app 42258810961).

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

✅ lens · dismissal after round 9 · F-b29afb by @fyzanshaik-atlan — ready to merge — every finding resolved · $0.80

Open findings: 🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: The PR addresses several CI trust gaps: certification could treat failed dependency installation as a passing gate, contract drift relied on app-owned generation and a narrow generated-file check, and private-dependency credentials could be exposed to code running during installs or CI. It also adds enforcement for the repository’s ban on conditional shell in workflow run: blocks.
  • How the PR solves it: It makes install failure block certification, reuses the SDK’s existing renovate_pkl_sync regeneration and checks the whole working tree, and separates private-repository mirror cloning (with a scoped read-only App token) from token-free installs and app execution. It also moves the targeted branching logic into tested Python scripts and adds a shrinking allowlist-backed guard.

$0.796 of $1.00 · 296 model calls · 0 failed requests

Full summary · Run log

@cmgrote

cmgrote commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

/lens force

@atlan-app-fleet

Copy link
Copy Markdown
Contributor

✅ lens · round 10 · re-review · only commits since 0c8e21d — ready to merge — every finding resolved · $0.81

Open findings: 🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low (nit)

Approach check — ✅ sound

  • Problem: From the intent and changed workflow/script symbols, this PR addresses fail-open certification when dependency installation fails, contract-drift checks that could miss generated changes, unguarded conditional shell in CI, and overly broad GitHub credentials reaching dependency-build paths. It also applies the credential change to reusable CI workflows and image builds.
  • How the PR solves it: It makes install outcomes part of the certification verdict, regenerates contracts through the SDK’s shared regeneration function and checks the full working tree, adds a CI guard while moving branching workflow logic into Python scripts, and replaces broad tokens with read-only App tokens scoped to declared private repositories. The mirror-clone step is separated from dependency installation so app build backends and later test steps do not inherit that token.

$0.806 of $1.00 · 302 model calls · 0 failed requests

Full summary · Run log

@atlan-ci atlan-ci left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lens: ready to merge — every finding at every level is resolved (round 10).

@cmgrote
cmgrote added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit c33607d Nov 9, 2026
48 checks passed
@cmgrote
cmgrote deleted the christophergrote/fnd-4244 branch November 9, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants