Skip to content

ci: stop a formula that fails to build from reporting a green job - #21

Merged
ilitteri merged 1 commit into
mainfrom
ci/fail-on-ignored-test-bot-steps
Aug 26, 2026
Merged

ci: stop a formula that fails to build from reporting a green job#21
ilitteri merged 1 commit into
mainfrom
ci/fail-on-ignored-test-bot-steps

Conversation

@MegaRedHand

Copy link
Copy Markdown
Contributor

What

Add a step after brew test-bot --only-formulae that fails the job if test-bot ignored a failing step.

Why

ethrex 25.0.0 did not build on Linux and all three jobs of that run were still green. The build error was reported as:

==> Skipped or failed formulae:
lambdaclass/tap/ethrex
Warning: 1 failed step ignored!
brew install --verbose --formula --build-bottle lambdaclass/tap/ethrex

test-bot decides whether a failure is fatal with (Homebrew/test_bot/formulae.rb):

ignore_failures = !args.test_default_formula? && !bottled_on_current_version && !new_formula

An ignored step becomes :ignored instead of :failed, and run! returns failed_steps.empty?, so the command exits 0.

bottled_on_current_version is bottled?(formula, no_older_versions: true), so every version bump lands in this branch: the bottle block still points at the previous version. That is exactly when the build most needs checking, and it is the one time the result is silently discarded. Upstream homebrew/core can live with this (an unbottled formula failing is not a blocker there); a single-formula tap cannot.

--fail-fast does not help: it only exits on steps already marked failed.

How

test-bot writes its summary to steps_output.txt. All steps passed! when clean, otherwise a Warning: N failed steps ignored! and/or Error: N failed steps! block. The new step greps for failed step and fails the job, printing the summary for context. A missing file also fails, so the check cannot silently stop working.

Verified against fixtures for the four states:

steps_output.txt exit
All steps passed! 0
Warning: 1 failed step ignored! 1
Error: 1 failed step! 1
absent 1

Notes

  • Merge fix: declare the OpenSSL dependency needed on Linux #20 first. With this check in place and the OpenSSL dependency still missing, every PR touching the formula goes red on ubuntu-24.04 (correctly, but there is no reason to sit through it).
  • Formulae that are skipped rather than failed stay green on purpose. macos-15-intel skips unconditionally: rust and rustup have no sequoia (x86_64) bottles, so deps_without_compatible_bottles is never empty there. That job has tested nothing for some time and is worth its own discussion, but failing on skips would just make it permanently red.

`brew test-bot` marks a failing step as ignored, rather than failed, when
the formula has no bottle for the platform it is running on
(`ignore_failures = !test_default_formula? && !bottled_on_current_version
&& !new_formula`). It prints `Warning: N failed steps ignored!` and exits
0, so the job goes green.

The tap hits that branch on every version bump: the bottle block only
covers the platforms bottles were built for, and a bump replaces the
version those bottles belong to. Run 32897389813 is the result: `ethrex`
25.0.0 did not build on Linux at all, and all three jobs were green.

`--fail-fast` does not help, it only exits on steps that are actually
marked failed. So read the `steps_output.txt` that test-bot writes and
fail the job whenever a step failed, ignored or not.
@ilitteri
ilitteri merged commit 8a228ab into main Aug 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants