Skip to content

Commit bc4329d

Browse files
ausimianclaude
andcommitted
build: set the coverage floor from the measurement and enforce it
The 85 threshold sat below the 87.95% it was meant to floor, so it ratcheted nothing and licensed a thirteen-line regression. It is now 88.58 - the measured figure, 419 of 473 relevant lines. One uncovered line added to lib gives 88.40% and fails; that was checked rather than assumed. And precommit runs test --cover, so the floor is a gate. Nothing else ran it, and a floor nothing enforces is decoration. CI test matrix stays on a plain mix test: cover line attribution can differ between Elixir versions, so the figure is enforced on the one pinned toolchain the precommit job uses. 90% would need 426 covered, seven more. Five of the seven can only come from the compiler generated clauses for defaulted arities nothing calls - measured, not assumed: calling the intermediate arities covers all five and reaches 89.64%. Each is a delegation whose defaults are a subset of an arity already called, so those cases would asserted nothing. That is the move this project does not make. The mechanism claim was also wrong. cover does not need distribution: :cover.start/0 works in a VM with no node name. What happens is that Mix instruments the modules loaded on this node and the peer loads the target own beams, which nothing instrumented - and cover only route to another VM is a distributed node, which the peer deliberately is not. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 54af6bb commit bc4329d

1 file changed

Lines changed: 43 additions & 21 deletions

File tree

mix.exs

Lines changed: 43 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -42,36 +42,52 @@ defmodule Castle.MixProject do
4242
# `mix test --cover` measures the shipped code, which is `lib` - the modules
4343
# under `test/support` are fixtures, and a fixture is covered by being run at
4444
# all. Left in, they moved the total without ever being the thing measured, and
45-
# two of them moved it *down* for a reason that is not about tests: `mix test`
46-
# compiles everything into one `cover` run on this node, while
45+
# two of them moved it *down* for a reason that is not about tests:
4746
# `Castle.PeerProviderStub` and most of `Castle.IoSink` execute inside the peer
48-
# VM, which is a separate node with no `cover` on it. Their code genuinely runs
49-
# and genuinely cannot be observed from here, so the figure they contributed was
50-
# an artefact of where they run rather than a gap in the suite - and raising it
51-
# would have meant calling them directly on the test node, which tests nothing.
47+
# VM, which nothing instruments - see the threshold below. Their code genuinely
48+
# runs and genuinely cannot be observed from here, so the figure they
49+
# contributed was an artefact of where they run rather than a gap in the suite,
50+
# and raising it would have meant calling them directly on the test node, which
51+
# tests nothing.
5252
#
5353
# Named module by module rather than matched by a pattern. A regex over `Stub`
5454
# or over `Castle.*Release` would quietly swallow a production module that
5555
# happened to be spelled that way, which is the one thing an exclusion list
5656
# must not do. Renaming a fixture makes the total drop, which is visible.
5757
#
58-
# The threshold is explicit because the default, 90, is unreachable here by
59-
# construction rather than for want of tests. The second half of
60-
# `Castle.Peer` - `resolve/1` and everything below the `## In the peer` comment -
61-
# runs in the peer VM, which has no node name and `is_alive() == false`, so
62-
# `cover` cannot be started on it. Around 7% of the shipped lines therefore
63-
# execute on every run of `Castle.PeerTest` and are counted as missed, which
64-
# puts the observable ceiling near 93%. Left at the default, `mix test --cover`
65-
# reports a failure that no test can fix and that says nothing about the suite.
58+
# **The threshold is the measured figure and nothing rounder.** 419 of 473
59+
# relevant lines, which is 88.58%; a single uncovered line added to `lib` takes
60+
# it to 88.37% and fails. It is set from a measurement rather than chosen, so a
61+
# refactor that legitimately removes covered lines will fail it too - and the
62+
# right answer then is to re-measure and edit this number deliberately, which
63+
# is the same rule every other claim in this project is held to. It replaced an
64+
# 85 that sat *below* the figure it was meant to floor and so licensed a
65+
# thirteen-line regression.
6666
#
67-
# It is deliberately not part of `mix precommit`. A percentage is the wrong
68-
# shape of gate for this project: every test here is justified by what it fails
69-
# against, and a merge-blocking number cannot tell a test that discriminates
70-
# from one written to raise it. See AGENTS.md for what is left uncovered and
71-
# why.
67+
# 90% would need 426 covered, seven more than there are. What is left is 33
68+
# lines in the peer's VM (below) plus 21 that are observable in principle: five
69+
# are the compiler's own default-argument clauses for arities nothing calls,
70+
# and the other sixteen need a file mode, a device node, or a config provider
71+
# sabotaging Castle's working directory. So the seven would have to include all
72+
# five of the default-argument clauses, whose only effect is on this number.
73+
# That is the move this project does not make. See AGENTS.md for the line-by-
74+
# line account.
75+
#
76+
# **What cannot be measured is the peer's VM, and the reason is where
77+
# instrumentation is applied rather than anything cover cannot do.**
78+
# `:cover.start/0` works perfectly well in a VM with no node name -
79+
# `is_alive() == false` is no obstacle to it. What happens here is that Mix
80+
# starts cover on *this* node and instruments the modules loaded here; the peer
81+
# is a separate VM that loads `Castle.Peer` from the target release's own beam
82+
# files on disk, which nothing has instrumented. Cover's only mechanism for
83+
# another VM is `:cover.start/1` over a *distributed* node, and this peer
84+
# deliberately has no distribution at all. So `resolve/1` and everything below
85+
# the `## In the peer` comment - 33 lines, about 7% of the shipped total - run
86+
# on every `Castle.PeerTest` and are counted as missed, which puts the
87+
# observable ceiling near 93%.
7288
defp test_coverage do
7389
[
74-
summary: [threshold: 85],
90+
summary: [threshold: 88.58],
7591
ignore_modules: [
7692
Castle.DeploymentStub,
7793
Castle.InitStub,
@@ -105,7 +121,13 @@ defmodule Castle.MixProject do
105121
"deps.unlock --unused",
106122
"format",
107123
"credo --strict",
108-
"test"
124+
# With `--cover`, so the threshold in `test_coverage/0` is a gate rather
125+
# than decoration: nothing else runs it, and a floor nothing enforces is
126+
# a number in a comment. This is the only place it is enforced - the CI
127+
# `test` matrix stays on a plain `mix test`, deliberately, because line
128+
# attribution can differ between Elixir versions and the figure is
129+
# measured on one toolchain.
130+
"test --cover"
109131
]
110132
]
111133
end

0 commit comments

Comments
 (0)