ci: `:testing` promotion frozen — `run-e2e / GNOME 50 — smoke-a` failing since 2026-06-25 (firefox.feature AT-SPI)

[castrojo]

Summary

promote-to-testing in .github/workflows/post-testing-e2e.yml has been skipped
on essentially every run since 2026-06-22, because the matrix leg
run-e2e / smoke,common / GNOME 50 — smoke-a fails. The failure is not
6 days old — it is ~46 days old and 100% reproducible.

Filed because this has had no tracking issue.

1. Blast radius

gh run list --workflow post-testing-e2e.yml --limit 300 (back to 2026-06-16):

conclusion	runs
failure	209
skipped	89
success	2

The only two successes were 27922429853 and 27922740150, both on
2026-06-22. Every run since has been failure (or skipped when the
upstream build didn't qualify). promote-to-testing has therefore been dead
for ~46 days.

Registry state (skopeo inspect docker://ghcr.io/projectbluefin/bluefin:testing):

Digest:  sha256:bf615b200faefc44b50232ecc8a3eb21490e88dd9316b528b27f54472122611d
Created: 2026-08-04T16:32:43Z
Tags on this version: testing, testing-20260804, testing-44.20260804

So :testing is ~2.5 days stale as of 2026-08-07.

⚠️ Secondary finding — the gate is being bypassed

sha256:bf615b20… is the exact digest that run
30931237054
tested and FAILED (IMAGE: ghcr.io/projectbluefin/bluefin@sha256:bf615b20…),
and in that run promote-to-testing was skipped.

build-image-testing.yml passes publish_stream_tag: "false", and
projectbluefin/actions/.github/workflows/reusable-build.yml@v1 (compute-push-tags,
~L375–392) is supposed to exclude the bare stream tag for exactly this reason.
Yet the bare testing tag landed on that digest anyway. No other workflow in
this repo writes :testing (promote-testing-to-main.yml does not; nightly.yml
and pr-validation.yml only read it).

This needs investigation in projectbluefin/actions: either the
compute-push-tags output is not honored by the push step (e.g. just tag-images
tags DEFAULT_TAG locally and a later push publishes all local tags), or something
else re-tags. Net effect: :testing is both stale and pointing at an
e2e-failing digest — the worst of both worlds.

2. The exact error

Failing leg: run-e2e / smoke,common / GNOME 50 — smoke-a.
Identical across every run sampled (30931237054 / 30833806016 / 30758889324,
and as far back as 28148474893 on 2026-06-25).

The whole firefox.feature file fails, and survives both @retry passes:

Failing scenarios:
  bluefin-tests/tests/smoke/features/firefox.feature:17  Address bar is present and focusable
  bluefin-tests/tests/smoke/features/firefox.feature:21  Navigating to about:blank loads successfully
  bluefin-tests/tests/smoke/features/firefox.feature:26  New tab opens via Ctrl+T and tab count increases
  bluefin-tests/tests/smoke/features/firefox.feature:30  Closing tab via Ctrl+W reduces tab count
  bluefin-tests/tests/smoke/features/firefox.feature:35  Navigating to a real URL loads the page
  bluefin-tests/tests/smoke/features/firefox.feature:40  Firefox closes cleanly via Ctrl+Q
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 6 failed, 2 skipped

Traceback:

  File "bluefin-tests/tests/smoke/features/steps/firefox_steps.py", line 103, in address_bar_is_present
    context.firefox_address_bar = _address_bar(context)
  File "bluefin-tests/tests/smoke/features/steps/firefox_steps.py", line 63, in _address_bar
    assert matches or bars, "Firefox address bar not found"
AssertionError: Firefox address bar not found

and, for the Ctrl+T/Ctrl+W scenarios:

AssertionError: Firefox tab list not found     (_tab_count, roleName == "page tab list")
AssertionError: Firefox is still visible in the AT-SPI tree   (Ctrl+Q teardown)

Interpretation: Firefox starts and is present in the AT-SPI tree, but its
internal accessible tree is never populated — no entry, no page tab list.
The preceding scenario "Firefox launches and main window is accessible" passes,
but that is a false pass: _firefox_window() accepts roleName in {"frame", "filler"},
and a bare filler stub satisfies it even when Firefox's ATK bridge never activated.
Consistent with Firefox not having accessibility enabled at process start
(GNOME_ACCESSIBILITY=1 / accessibility.force_disabled=0) in the headless
QEMU GNOME 50 session. The suite launches it as a plain firefox command
(FIREFOX_LAUNCH_TARGETS[0]) with no a11y env.

AT-SPI itself is up in the VM — from the same journal:

at-spi2-registryd[6001]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Started org.gnome.SettingsDaemon.A11ySettings.service - GNOME accessibility service.

which is why the separate gnome_accessibility.feature:33 AT-SPI registry daemon is running
first-pass failure also looks like a bad assertion (it is dbus-activated as
dbus-:1.19-org.a11y.atspi.Registry@0.service, not a statically named unit).

Other first-pass-only failures that clear on retry (i.e. flake, not the blocker):
bluefin_desktop.feature:18 Dash to Dock is visible and
gnome_apps.feature:6 Ptyxis terminal launches successfully.
Only firefox.feature fails deterministically through all retries.

3. First bad run

• Last green: 27922740150 — 2026-06-22T00:38:22Z
• First red after it: 28148474893 — 2026-06-25T05:14:07Z
  (job 83360626356), already failing the identical six firefox.feature
  scenarios. Everything between was skipped.

Nothing in bluefin's own tree explains it. projectbluefin/testsuite
tests/smoke/features/steps/firefox_steps.py has been untouched since
774f7696 (2026-06-21), and firefox.feature since f66fb71f (2026-06-22T04:23Z) —
i.e. the last change to the Firefox feature file landed between the last green
run and the first red one. ~50 other testsuite commits landed in that same window
(2026-06-22 → 2026-06-25), including a wave of GNOME 50 quarantine/AT-SPI fixes
(3f2d10cd, 6a289153, 6b48898c, 984d2eaa), and 51238645
"docs(ops): document common suite runner model and pre-existing smoke failures".

4. Root-cause classification

Broken test in projectbluefin/testsuite, not a bluefin image regression.

• Bluefin ships Firefox as an RPM (build_files/packages/base.toml), unchanged.
• The failure is a missing AT-SPI subtree, not a Firefox launch failure —
  Firefox does start.
• No bluefin-side change correlates with 2026-06-22 → 2026-06-25.
• The fix (enable Firefox accessibility before launch, and stop treating a bare
  filler node as a healthy main window) belongs in
  projectbluefin/testsuite/tests/smoke/features/steps/firefox_steps.py.

Plus the separate projectbluefin/actions stream-tag leak described in §1.

5. Unblock criteria

1. testsuite: launch Firefox with GNOME_ACCESSIBILITY=1 (and/or set
   accessibility.force_disabled=0 in the profile) before the smoke run, and
   tighten _firefox_window() so a filler-only tree fails "main window is
   accessible" instead of falsely passing.
2. testsuite: fix gnome_accessibility.feature:33 to accept the
   dbus-activated org.a11y.atspi.Registry unit name.
3. testsuite: stabilise or @retry-tag the Dash-to-Dock and Ptyxis
   first-pass flakes.
4. actions: confirm publish_stream_tag: "false" actually prevents the bare
   :testing tag from being pushed; add a regression guard.
5. bluefin: once (1)–(3) land, one green Post-Testing E2E run with
   promote-to-testing: success closes this.

Explicitly NOT the fix

Do not add always(), continue-on-error, or drop run-e2e from
promote-to-testing's needs:. That promotes an unverified digest and is the
failure mode a sibling PR was already closed for.

Related

• ci: restore upgrade-test (lifecycle) as a promotion gate once GNOME 50 AT-SPI is stable #400 — run-upgrade-test (lifecycle) is already disabled as a gate for the
  same class of GNOME 50 AT-SPI breakage, and is blocked behind this same
  instability.
• ci: stable promotion has not completed since July 20 #929 — stable promotion has not completed since 2026-07-20 (likely downstream
  of this).

_{Filed by an automated diagnosis pass; evidence gathered from
gh run view --log on runs 30931237054, 30833806016, 30758889324, 29736070134,
28148474893, and skopeo inspect of ghcr.io/projectbluefin/bluefin:testing.}

[castrojo]

Fix in flight: projectbluefin/testsuite#692 — fix(smoke): enable Firefox AT-SPI accessibility and stop the filler false pass.

Two root causes for the ~46-day post-testing-e2e freeze (last green 27922740150, 2026-06-22; 0 successes in the last 100 runs, promote-to-testing always skipped):

1. Firefox AT-SPI tree never populated. e2e.yml sets toolkit-accessibility session-wide, which only drives the GTK atk-bridge. Firefox renders its own chrome and gates its a11y bridge on GNOME_ACCESSIBILITY/ACCESSIBILITY_ENABLED at process start, and it was launched with a bare nohup firefox inheriting neither. launch_background() now takes an env= mapping applied on the SSH, local-Popen, and flatpak run --env= paths. Contrast evidence: gnome_extensions_steps.py already launches with GNOME_ACCESSIBILITY=1 and passes.
2. False pass in _firefox_window(). It accepted a bare roleName "filler" node, so "Firefox main window is accessible" passed against a window with no a11y tree and the real failure surfaced later as "address bar not found". filler is load-bearing on GNOME 50 (commit 12bd892e) so it is kept, but a candidate must now expose real chrome widgets, and the failure message names GNOME_ACCESSIBILITY as the likely cause.

Verified locally: ruff, 898 unit tests (19 new), behave --dry-run, docs validation. The AT-SPI behaviour itself needs a live GNOME session and can only be confirmed by a real e2e run — verification steps are in the PR body.

PR is left open for human review, not enqueued.