Skip to content

Commit ad53918

Browse files
ausimianclaude
andcommitted
ci: watch the coverage floor on the newest Elixir
The floor was enforced in only one place that ran it - mix precommit, on whatever Elixir the contributor happened to have - and nothing in CI checked it at all. The test matrix ran a plain mix test and the precommit job is pinned to the one version that agreed with the number, so CI stayed green while a clean tree failed the gate on Elixir 1.20. A drift nothing watches is a drift the next contributor finds. The newest cell of the test matrix now exports coverage and checks the floor. It is the lowest reporter, so it trips first on attribution drift. Every other cell stays on a plain mix test: seven readings that disagree by version would be noise rather than signal. The include entry augments the existing 1.20/29 cell rather than adding a job. Two steps rather than one mix test --cover, so the step that goes red names what broke without opening the log - exporting cannot fail on the threshold, and mix test.coverage cannot fail on a test. The job name carries the same information. Verified on 1.20.3/OTP 29: export exits 0, the check reports 88.40% and exits 0, and with the floor raised to 89 the check exits 3 - so it is a gate and not decoration. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 234cbc7 commit ad53918

2 files changed

Lines changed: 67 additions & 8 deletions

File tree

.github/workflows/ci.yml

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ jobs:
3939
run: mix precommit
4040

4141
test:
42-
name: Test (Elixir ${{ matrix.elixir }} / OTP ${{ matrix.otp }})
42+
name: Test (Elixir ${{ matrix.elixir }} / OTP ${{ matrix.otp }})${{ matrix.cover && ' + coverage floor' || '' }}
4343
# Only Linux for now. Castle is pure Elixir talking to :release_handler,
4444
# with no shell scripts of its own and no suite that boots a release, so
4545
# there is nothing platform-sensitive to run. Add macos-15 alongside the
@@ -60,6 +60,27 @@ jobs:
6060
# Elixir 1.19 supports Erlang/OTP 26-28
6161
- elixir: '1.19'
6262
otp: '29'
63+
include:
64+
# **The one cell that checks the coverage floor, and the only place in
65+
# CI that checks it at all.** `mix precommit` enforces the threshold in
66+
# `mix.exs`, but the `precommit` job below is pinned to a single
67+
# Elixir, so on its own it says nothing about the rest of the
68+
# `~> 1.18` range that `mix.exs` declares - and cover's line
69+
# attribution is not constant across that range. Elixir 1.20 counts
70+
# one more relevant line in `Castle.Peer` than 1.19 does, which is how
71+
# a floor set from the pinned toolchain came to fail a clean tree for
72+
# contributors on a current Elixir.
73+
#
74+
# This is the newest Elixir in the matrix, so it reports the lowest
75+
# figure and trips first: a canary for exactly that drift. Every other
76+
# cell stays on a plain `mix test` deliberately - seven readings that
77+
# disagree by version would be noise, not signal.
78+
#
79+
# Keep this on whichever cell is newest when the matrix moves on. The
80+
# threshold note in `mix.exs` explains what to do if it goes red.
81+
- elixir: '1.20'
82+
otp: '29'
83+
cover: true
6384

6485
steps:
6586
- name: Checkout code
@@ -91,4 +112,23 @@ jobs:
91112
run: mix compile --warnings-as-errors
92113

93114
- name: Run tests
115+
if: ${{ !matrix.cover }}
94116
run: mix test
117+
118+
# Split in two rather than one `mix test --cover`, so that the step which
119+
# goes red names the thing that broke without anyone opening the log:
120+
# exporting cannot fail on coverage, and checking cannot fail on a test.
121+
- name: Run tests (exporting coverage)
122+
if: ${{ matrix.cover }}
123+
run: mix test --cover --export-coverage ci
124+
125+
- name: Check the coverage floor
126+
# Red here means the total fell under the threshold in `mix.exs` while the
127+
# tests passed. Either coverage genuinely dropped, or this Elixir
128+
# attributes relevant lines differently from the pinned toolchain - read
129+
# the threshold note in `mix.exs` before changing the number, because it
130+
# is a floor across the whole supported range and not a reading from one.
131+
if: ${{ matrix.cover }}
132+
env:
133+
MIX_ENV: test
134+
run: mix test.coverage

AGENTS.md

Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1538,13 +1538,32 @@ list is module names rather than a pattern: a regex over `Stub` would swallow a
15381538
production module spelled that way, which is the one thing an exclusion must not
15391539
do.
15401540

1541-
**It is part of `mix precommit`**, which is what makes the threshold a gate
1542-
rather than a number in a comment. Nothing else runs it: CI's `test` matrix runs
1543-
a plain `mix test`, and CI's `precommit` job is pinned to Elixir 1.19 / OTP 28.
1544-
So the *only* place this number is checked against the rest of the `~> 1.18`
1545-
range is a contributor's own machine, in the gate this project makes mandatory —
1546-
which is exactly where a false failure does the most damage, because the lesson
1547-
it teaches is to stop running the gate.
1541+
**Three things check the floor, and they are not checking the same thing.**
1542+
1543+
* **`mix precommit`** runs `test --cover`, which is what makes the threshold a
1544+
gate rather than a number in a comment. It runs on whatever Elixir the
1545+
contributor happens to have, so this is where the floor meets the `~> 1.18`
1546+
range in practice — and where a false failure does the most damage, because
1547+
the lesson it teaches is to stop running the gate.
1548+
* **CI's `precommit` job** runs the same alias pinned to Elixir 1.19 / OTP 28.
1549+
That is the reading the rest of this section quotes.
1550+
* **One cell of CI's `test` matrix** — the newest Elixir, currently 1.20 /
1551+
OTP 29 — exports coverage and then runs `mix test.coverage`. It is the lowest
1552+
reporter, so it trips first: a canary for attribution drift rather than a
1553+
second opinion on the figure. Every other cell stays on a plain `mix test`
1554+
deliberately, because seven readings that disagree by version are noise.
1555+
1556+
That cell is **two steps rather than one `mix test --cover`**, so the step that
1557+
goes red names what broke without anyone opening the log: exporting coverage
1558+
cannot fail on the threshold, and `mix test.coverage` cannot fail on a test. Its
1559+
job name carries `+ coverage floor` for the same reason. Move the `cover: true`
1560+
flag when the matrix's newest cell moves on.
1561+
1562+
None of that existed until the floor had already failed a clean tree on 1.20 and
1563+
CI had stayed green through it, because the matrix ran a plain `mix test` and the
1564+
one job that checked the threshold was pinned to the version that agreed with
1565+
it. A gate checked only on the maintainer's toolchain is a gate that discovers
1566+
its own bugs through other people.
15481567

15491568
**The figure is not the same on every supported toolchain, and the threshold has
15501569
to be a floor across them rather than a reading from one.** Measured, one cell

0 commit comments

Comments
 (0)