Commit a7196d6
test(first-run-tour): assert every pinned template is served, not just three (#14670)
## Summary
The first-run tour's role-pin drift guard could pass while every curated
template was unserved. It is the only thing standing between a template
rename and a silently degraded first-run experience, so a guard that
tolerates five 404s out of eight is close to no guard at all.
Closes #14620
> [!IMPORTANT]
> **This PR is red on purpose, and the failure is the finding.** The
tightened guard immediately caught two unserved pins in CI — see "What
CI found" below. It needs a maintainer decision before it can merge; I
did not loosen it back to hide them.
## Changes
- **What**: `firstRunTourRolePins.spec.ts` now asserts the identity of
the unserved set rather than its arity.
The spec collects `unserved` template ids as it walks `TOUR_ROLE_PINS`,
then backstopped it with `expect(pinnedTemplates.length -
unserved.length).toBeGreaterThan(2)` — against 8 pinned templates, that
permits 5 to 404. The four _curated_ ids are the ones the Getting
Started grid prefers, so exactly the set real users hit could go
entirely unverified while CI stayed green.
```ts
expect(
unserved,
'every pinned template must be served, or its card vanishes from the Getting Started grid and its pins go unchecked'
).toEqual([])
```
The `test.info().annotations` block above it is removed: it existed to
surface unserved ids past a _passing_ run, and a failing assertion
prints the array itself.
## What CI found
Shard 4/16 fails with:
```
- Array []
+ Array [
+ "image_krea2_turbo_t2i",
+ "templates-qwen_multiangle.app",
+ ]
```
Two of eight pins are unserved by the e2e backend, and
**`image_krea2_turbo_t2i` is a curated id** — the first entry in
`CURATED_TEMPLATE_IDS` and the thumbnail for the `inpaint` tutorial
card. Under the old `> 2` check this was invisible: 6 served > 2, green.
I chased the cause rather than assuming a bad pin:
- Both ids resolve upstream on `Comfy-Org/workflow_templates@main`
(`templates/image_krea2_turbo_t2i.json`, 58 KB;
`templates/templates-qwen_multiangle.app.json`, 57 KB), so **the pins
are correct**.
- The e2e job runs `ghcr.io/comfy-org/comfyui-ci-container:0.0.21`
(ComfyUI v0.19.3), whose bundled `comfyui-workflow-templates` predates
both.
- `0.0.21` is the newest published tag — it is what `latest` points at —
so this **cannot be fixed by bumping the image here**.
So the guard is correct and the environment is stale. That is worth
knowing on its own: the e2e suite currently cannot verify the curated
first-run templates at all, which is the same blind spot from the other
direction.
## Decision needed
1. **Republish the CI container** with a current template package, then
this merges green. Preferred — it restores real coverage of the curated
set.
2. **Merge red / hold** until that happens.
3. **Scope the assertion** to templates the backend advertises in
`/templates/index.json`, so version lag stops counting as drift. Still
far tighter than `> 2`, but it reintroduces a quiet pass whenever a
curated template is simply absent — the hole this issue is about. I did
not do this unasked; say the word and I will.
I have left the assertion strict and added a comment at the call site
naming both ways to land there (upstream rename → fix the pin; stale
backend → the coverage gap), so the next person to hit this red does not
have to redo the investigation.
## Verification
- `pnpm exec playwright test --list
browser_tests/tests/firstRunTourRolePins.spec.ts` — compiles and
collects (1 test).
- `pnpm format:check`, `pnpm lint` (oxlint covers `browser_tests`),
`pnpm typecheck` — all clean.
- CI ran the spec for real: it fails, with the two ids above. That is
the only execution of it — **I could not run this spec locally**, it
needs a live backend serving `/templates/*.json`.
- The upstream existence check was `gh api
repos/Comfy-Org/workflow_templates/contents/templates/<id>.json` against
`main`; I did not verify which template-package _release_ first
contained them, only that `main` has them and the pinned container does
not serve them.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent ce2f58f commit a7196d6
1 file changed
Lines changed: 4 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
| 76 | + | |
81 | 77 | | |
82 | | - | |
83 | | - | |
84 | | - | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
85 | 81 | | |
86 | 82 | | |
0 commit comments