Release workflow hardening — lessons from 1.47.9/1.47.10

[christian-byrne]

Problem

The core 1.47.9 → 1.47.10 releases surfaced three concrete friction points in the release automation. Each cost manual recovery or produced a false signal. Fixes below are minimal and verified against the current workflow YAML on main.

Reference runs / PRs

• Biweekly run that failed on the tag-wait, then recovered via gh run rerun … --failed: https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/29979988134
• release-draft-create run for v1.47.10 (cosmetic failure — see Configure vite to copy from src to dist #2): https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/29983428805
• Frontend version-bump PR 1.47.10 #14029 ("1.47.10" → core/1.47)
• ComfyUI pin PR Bump comfyui-frontend-package to 1.47.10 ComfyUI#15045 (requirements.txt 1.45.21 → 1.47.10, base master)

---

1. publish-pypi tag-wait timeout takes down the whole release, with no auto-recovery

Where: release-biweekly-comfyui.yaml → publish-pypi job → step "Wait for release PR to be created and merged".

The step polls for tag v${TARGET_VERSION} for 60 × 30s = 30 min, then exit 1. That tag is only created by release-draft-create when a human merges the Release-labeled bump PR (#14029) — so the poll depends on a human action inside a hard 30-min window. Because create-comfyui-pr has needs: publish-pypi + if: … needs.publish-pypi.result == 'success', a timeout skips create-comfyui-pr too — a single run loses both the PyPI publish and the ComfyUI pin PR.

Observed (1.47.10): run 29979988134 failed exactly this way; recovery was a manual gh run rerun 29979988134 --failed after the tag existed (which reuses the resolved target_version — no re-resolve, no spurious next-patch bump).

Suggested fixes (any one; first is cleanest):

• (Preferred) Decouple publish from the poll — make it release-triggered. Move the PyPI build+publish into release-draft-create.yaml, or a new workflow with on: release: { types: [published] } / on: push: tags: ['v*'], so PyPI publishes exactly when the tag/release is created — no polling, no window. The biweekly then only resolves + triggers the bump; the pin PR keys off the same release event.
• (If keeping the poll) raise the timeout (a human merge realistically isn't 30 min), and on timeout end the job neutral rather than hard-fail, or gate create-comfyui-pr on the tag existing (not on publish-pypi.result == 'success') so a late publish doesn't also drop the pin PR.
• At minimum: print "recover with gh run rerun <run-id> --failed once the tag exists" in the timeout branch and job summary.

---

2. comment_release_summary crash marks a fully-successful release as failure

Where: release-draft-create.yaml → comment_release_summary job → step "Post release summary comment" (./.github/actions/comment-release-links). Job is if: success(); the step has no continue-on-error.

Observed (1.47.10): run 29983428805 — build, draft_release, and publish_types all succeeded (tag cut, GH release created, npm types published), but the comment step crashed (Node error) and marked the whole run failure. A cosmetic step produced a false "release failed" signal.

Suggested fix (one line): add continue-on-error: true to the "Post release summary comment" step (or the job). Optionally keep a ::warning:: so the crash stays visible without failing the run. A comment must never gate release success.

---

3. No automated "release-done" assertion — the 1.47.9 stranded-commits case went undetected

Where: nothing in the four release workflows asserts the release actually reached users after publish.

Observed (1.47.9): v1.47.9 published to PyPI while the QA fix set (19 commits) sat unreleased past the tag — desktop/self-host users got a version advertised as fixed that wasn't, forcing the 1.47.10 re-cut. Only a manual git rev-list caught it.

Suggested fix: add a release-done assertion job (after create-comfyui-pr, or a small workflow on release publish) that hard-fails + pings if any of:

• git rev-list v${TARGET}..origin/${TARGET_BRANCH} --count != 0 (commits stranded past the tag)
• PyPI .info.version of comfyui-frontend-package != ${TARGET}
• ComfyUI master:requirements.txt pin != comfyui-frontend-package==${TARGET} (warn-only if the pin PR is intentionally left for a human to merge)

---

4. Smaller observations

• publish-pypi's "Wait" step also blocks a manual dispatch (needs_release == true) on a human merge inside the same run — same coupling as Bump braces from 3.0.2 to 3.0.3 in /tests-ui #1; decoupling fixes it too.
• The biweekly job summary should print the resolved target_version/target_branch and the recovery command up front, so a failed run is self-documenting.
• make_latest semantics are correct (core core/* base → non-latest, so a core patch can't steal latest from e.g. v1.48.x) — worth a one-line inline comment in release-draft-create.yaml to prevent a future "why isn't the core release latest?" edit.

Priority

1. Configure vite to copy from src to dist #2 — one-line continue-on-error; kills false release-failure alarms. Do now.
2. Bump braces from 3.0.2 to 3.0.3 in /tests-ui #1 — decouple publish from the tag-wait poll (or make it tolerant + recoverable). Removes the failure class that lost work in 1.47.10.
3. Migrate domWidget from js to ts #3 — release-done assertion job; auto-catches the 1.47.9 stranded-commits class.
4. Migrate widgets.js #4 — polish / comments.

[coderabbitai]

🔗 Related PRs

#13203 - ci: run lint/knip verification on push to main and release branches [open]
#13454 - fix: keep queued batches bound to starting workflow [open]
#13847 - Correlate frontend preview readiness [open]
#13905 - fix: recover workflow draft persistence after transient quota errors [open]
#14022 - feat(ci): auto-assign backport and version-bump PRs to the release sheriff [open]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[christian-byrne]

Companion observability issue filed: #14038 — Slack alerting on backport/release failures + a release-done assertion that all needs-backport PRs actually landed. That's the alerting half of the release-automation friction described here (this issue covers the reliability fixes); the two should land together.

[christian-byrne]

All three of these fired again this week, and §1 and §3 together cost users a working build for seven days.

§3 (no release-done assertion) — recurred exactly as predicted. v1.47.10 was tagged 2026-07-22 22:54 PT. The templates dropdown fix (#14065) merged onto core/1.47 19h later. No patch was cut, ComfyUI 0.29.0 pinned 1.47.10 on 07-29, and the fix reached nobody until today — five user reports in the meantime (#14063, #14131, #14212, #14351, #14397), with people hand-editing z-index in shipped CSS as a workaround.

At the point of investigation git rev-list v1.47.10..origin/core/1.47 --count was 6, four of them fix commits. Exactly the assertion proposed here.

§1 (tag-wait coupling) — fired again on 2026-07-27. The scheduled run failed with publish-pypi: failure / create-comfyui-pr: skipped, targeting 1.48.6, whose bump PR #14150 is still unmerged. That is twice now that a single unmerged bump PR has dropped both the publish and the pin PR. Six Release-labelled bump PRs are open right now, oldest from 07-15 — the human-merge step is where releases routinely die.

§2 (cosmetic failure) unchanged.

PRs:

• feat(cicd): fail when fixes sit unshipped on a consumed release line #14429 implements §3 — daily + on-release guard, scoped to consumed lines only, Slack on failure. Against pre-1.47.11 state it exits 1 and names all four stranded fixes.
• fix(ci): stop cosmetic failures and slow merges from losing a release #14428 implements §2 and mitigates §1 (four-hour window + self-documenting rerun --failed recovery).

§1's preferred fix — decoupling publish from the poll so it is release-triggered — is deliberately not in #14428. It restructures a pipeline that cannot be dry-run, and getting it wrong breaks every release. That remains the open item here.

One more defect found while cutting 1.47.11, not currently tracked in this issue: release-draft-create.yaml passes make_latest: false for a core/* base, but GitHub still surfaced v1.47.11 as Latest, displacing v1.49.1. I restored it by hand via the API. The release-done check should also assert releases/latest still points at the newest published line.