Skip to content

Commit 39826f6

Browse files
Revert PR-tier coverage; keep coverage canonical in nightly
The PR-tier coverage instrumentation introduced in b80b090 turned out to add ~600% wall to the unit run on the GitHub ubuntu-latest 2-vCPU runner, not the ~30% that motivated option 3. Coverage's sys.settrace collides with JAX's own tracing inside compute_mlt / compute_fluxes / the JIT-compiled RHS path; the ubuntu + py3.12 cell ran for >18 min versus 3 min for the same suite without --cov on py3.13. The fast-feedback rationale for PR-side coverage is dead at that wall. ci_tests.yml: drop pytest-cov from the pip install, drop the COV_FLAGS env var and the --cov-report flags from the pytest invocation, drop the "Upload coverage to Codecov" step, drop the "Download last nightly coverage artifact" step, drop the "Estimate combined coverage union" step, drop the actions:read permission (only needed for gh run download which is gone). PR tier returns to ~3 min wall on every cell. nightly.yml: drop --cov-report=json:coverage.json, drop the timestamp stamp, drop the upload-artifact step. The artifact had no consumer after removing the union-estimation step. Coverage to Codecov continues via --cov-report=xml under the `nightly` flag; the canonical 95% gate stays enforced via [tool.coverage.report].fail_under in pyproject.toml. Element 4 of the PROTEUS Ecosystem Testing Standard is satisfied for this repo by Codecov's own server-side patch coverage check (which compares each PR diff against the most recent successful nightly upload on main). The local union estimation step was redundant for repos where the PR tier is a strict subset of the nightly tier (PR runs unit only; nightly already includes the same unit tests in its full sweep).
1 parent f4ee181 commit 39826f6

2 files changed

Lines changed: 3 additions & 257 deletions

File tree

.github/workflows/ci_tests.yml

Lines changed: 2 additions & 236 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,6 @@ on:
3434

3535
permissions:
3636
contents: read
37-
# actions:read enables `gh run download` to fetch the previous
38-
# nightly's coverage artifact for the union estimation step.
39-
actions: read
4037

4138
# Cancel superseded runs on the same ref so a stack of feature-branch
4239
# pushes does not pile up macOS runners (free-tier concurrency is tight).
@@ -79,7 +76,7 @@ jobs:
7976
8077
- name: Install pip-only test deps (scikits-odes-sundials builds against conda SUNDIALS)
8178
run: |
82-
pip install pytest pytest-cov pytest-xdist pytest-dependency ruff
79+
pip install pytest pytest-xdist pytest-dependency ruff
8380
# Install scikits-odes-sundials directly (skip the scikits-odes
8481
# metapackage). This avoids scikits-odes-daepack, which is a
8582
# transitive Fortran dep with f2py incompatibility with numpy
@@ -180,21 +177,9 @@ jobs:
180177
PY
181178
182179
- name: Run unit tests
183-
env:
184-
# Coverage instrumentation slows pytest by ~30 %. The Codecov
185-
# upload + union-estimation steps below only run on the
186-
# ubuntu + py3.12 cell, so the other five matrix cells gain
187-
# nothing from emitting coverage. Gate --cov flags to that one
188-
# cell to keep the rest of the matrix on the original ~3 min
189-
# wall. --cov-fail-under=0 overrides the 95 % floor in
190-
# pyproject.toml for this tier (unit-only is around 83 %; the
191-
# canonical 95 % gate is enforced in nightly.yml where the
192-
# full unit + smoke + slow tier reaches ~96 %).
193-
COV_FLAGS: ${{ (matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12') && '--cov=src/aragog --cov-report=xml --cov-report=json:coverage.json --cov-fail-under=0' || '' }}
194180
run: |
195181
pytest -m "unit and not slow" -n auto -o "addopts=" \
196-
-o "junit_family=legacy" --junitxml=junit.xml \
197-
$COV_FLAGS
182+
-o "junit_family=legacy" --junitxml=junit.xml
198183
199184
- name: Upload test results to Codecov
200185
if: ${{ !cancelled() && matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12' }}
@@ -203,222 +188,3 @@ jobs:
203188
token: ${{ secrets.CODECOV_TOKEN }}
204189
slug: FormingWorlds/Aragog
205190
files: junit.xml
206-
207-
- name: Upload coverage to Codecov
208-
if: ${{ !cancelled() && matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12' }}
209-
uses: codecov/codecov-action@v5
210-
with:
211-
token: ${{ secrets.CODECOV_TOKEN }}
212-
slug: FormingWorlds/Aragog
213-
files: coverage.xml
214-
flags: ci
215-
fail_ci_if_error: false
216-
217-
- name: Download last nightly coverage artifact
218-
id: download-nightly
219-
if: ${{ !cancelled() && matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12' }}
220-
continue-on-error: true
221-
env:
222-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
223-
run: |
224-
# First-party-only path to cross-workflow artifact download:
225-
# query the latest successful nightly run on main, then use
226-
# `gh run download`. Failures here do not break the PR (the
227-
# union step downgrades to "nightly missing" warning).
228-
set +e
229-
mkdir -p nightly-coverage
230-
RUN_ID=$(gh run list \
231-
--workflow=nightly.yml \
232-
--branch=main \
233-
--status=success \
234-
--limit=1 \
235-
--json databaseId \
236-
--jq '.[0].databaseId' \
237-
-R ${{ github.repository }})
238-
if [ -z "$RUN_ID" ]; then
239-
echo "No successful nightly run found on main."
240-
echo "nightly_missing=true" >> "$GITHUB_OUTPUT"
241-
exit 0
242-
fi
243-
echo "Downloading artifact from nightly run $RUN_ID"
244-
gh run download "$RUN_ID" \
245-
--name nightly-coverage \
246-
--dir nightly-coverage \
247-
-R ${{ github.repository }}
248-
if [ ! -f nightly-coverage/coverage.json ]; then
249-
echo "Artifact missing coverage.json; treating as missing."
250-
echo "nightly_missing=true" >> "$GITHUB_OUTPUT"
251-
exit 0
252-
fi
253-
echo "nightly_missing=false" >> "$GITHUB_OUTPUT"
254-
echo "nightly_run_id=$RUN_ID" >> "$GITHUB_OUTPUT"
255-
256-
- name: Estimate combined coverage union
257-
id: coverage-union
258-
if: ${{ !cancelled() && matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12' }}
259-
env:
260-
NIGHTLY_MISSING: ${{ steps.download-nightly.outputs.nightly_missing }}
261-
run: |
262-
python - <<'PY'
263-
import json
264-
import os
265-
import pathlib
266-
import sys
267-
import tomllib
268-
from datetime import datetime, timezone
269-
270-
GRACE_PERIOD = 0.3 # percent; coverage drops below this do not block
271-
STALE_HOURS = 48
272-
273-
def read_totals(path):
274-
try:
275-
with open(path) as fh:
276-
data = json.load(fh)
277-
t = data.get('totals', {})
278-
return (
279-
t.get('percent_covered', 0.0),
280-
t.get('covered_lines', 0),
281-
t.get('num_statements', 0),
282-
)
283-
except Exception:
284-
return 0.0, 0, 0
285-
286-
def norm_path(p):
287-
p = p.replace('\\', '/')
288-
if 'src/' in p:
289-
return p.split('src/', 1)[-1]
290-
return p
291-
292-
def line_set_executed(data):
293-
out = set()
294-
for path, fd in data.get('files', {}).items():
295-
n = norm_path(path)
296-
for line in fd.get('executed_lines', []) or []:
297-
out.add((n, line))
298-
return out
299-
300-
def line_set_executable(data):
301-
out = set()
302-
for path, fd in data.get('files', {}).items():
303-
n = norm_path(path)
304-
exec_ = fd.get('executed_lines', []) or []
305-
miss_ = fd.get('missing_lines', []) or []
306-
for line in exec_ + miss_:
307-
out.add((n, line))
308-
return out
309-
310-
# Threshold: source of truth is pyproject.toml.
311-
full_threshold = 95.0
312-
try:
313-
data = tomllib.loads(pathlib.Path('pyproject.toml').read_text())
314-
full_threshold = float(
315-
data['tool']['coverage']['report']['fail_under']
316-
)
317-
except Exception as exc:
318-
print(f'WARN: could not read fail_under from pyproject.toml: {exc}')
319-
320-
# Unit-tier coverage from this PR.
321-
u_pct, u_covered, u_total = read_totals('coverage.json')
322-
u_data = {}
323-
if pathlib.Path('coverage.json').exists():
324-
try:
325-
u_data = json.loads(pathlib.Path('coverage.json').read_text())
326-
except Exception:
327-
pass
328-
329-
# Nightly full-suite coverage.
330-
n_path = pathlib.Path('nightly-coverage/coverage.json')
331-
n_pct, n_covered, n_total = read_totals(n_path)
332-
n_data = {}
333-
if n_path.exists():
334-
try:
335-
n_data = json.loads(n_path.read_text())
336-
except Exception:
337-
pass
338-
339-
# Staleness check.
340-
ts_path = pathlib.Path('nightly-coverage/nightly-timestamp.txt')
341-
stale = False
342-
stale_hours = None
343-
if ts_path.exists():
344-
try:
345-
ts_str = ts_path.read_text().strip().replace('Z', '+00:00')
346-
ts = datetime.fromisoformat(ts_str)
347-
age = datetime.now(timezone.utc) - ts
348-
stale_hours = age.total_seconds() / 3600.0
349-
stale = stale_hours > STALE_HOURS
350-
except Exception:
351-
pass
352-
353-
missing = os.environ.get('NIGHTLY_MISSING', 'false') == 'true'
354-
missing = missing or not n_data
355-
356-
# Union estimate.
357-
est_pct = None
358-
est_covered = 0
359-
est_total = 0
360-
if u_data.get('files') and n_data.get('files'):
361-
union_exec = line_set_executed(u_data) | line_set_executed(n_data)
362-
union_all = line_set_executable(u_data) | line_set_executable(n_data)
363-
if union_all:
364-
est_covered = len(union_exec)
365-
est_total = len(union_all)
366-
est_pct = min(100.0, 100.0 * est_covered / est_total)
367-
368-
# Status: ok / warn / fail relative to full threshold + grace.
369-
status = 'ok'
370-
drop = 0.0
371-
if est_pct is not None:
372-
drop = full_threshold - est_pct
373-
if drop > GRACE_PERIOD:
374-
status = 'fail'
375-
elif drop > 0:
376-
status = 'warn'
377-
378-
summary_lines = [
379-
'## Coverage estimate (PR vs nightly union)',
380-
'',
381-
f'- Unit (this PR): **{u_pct:.2f}%** ({u_covered}/{u_total} stmts)',
382-
]
383-
if missing:
384-
summary_lines.append('- Nightly artifact: **MISSING** (no successful nightly on main)')
385-
else:
386-
age_str = f' ({stale_hours:.1f}h old)' if stale_hours is not None else ''
387-
tag = ' [STALE]' if stale else ''
388-
summary_lines.append(
389-
f'- Nightly: **{n_pct:.2f}%** ({n_covered}/{n_total}){age_str}{tag}'
390-
)
391-
if est_pct is not None:
392-
summary_lines.append(
393-
f'- Estimated union: **{est_pct:.2f}%** ({est_covered}/{est_total})'
394-
)
395-
summary_lines.append(f'- Threshold: {full_threshold:.2f}% | drop: {drop:+.2f}% | status: **{status}**')
396-
else:
397-
summary_lines.append('- Union estimate unavailable (one or both JSONs missing).')
398-
if missing or stale:
399-
summary_lines.append('')
400-
summary_lines.append('> Nightly missing or stale; union number is informational only and does not gate this PR.')
401-
402-
summary_md = '\n'.join(summary_lines) + '\n'
403-
print(summary_md)
404-
step_summary = os.environ.get('GITHUB_STEP_SUMMARY')
405-
if step_summary:
406-
with open(step_summary, 'a') as fh:
407-
fh.write(summary_md)
408-
409-
out = os.environ.get('GITHUB_OUTPUT')
410-
if out:
411-
with open(out, 'a') as fh:
412-
fh.write(f'unit_pct={u_pct:.2f}\n')
413-
fh.write(f'nightly_pct={n_pct:.2f}\n')
414-
fh.write(f'est_pct={est_pct:.2f}\n' if est_pct is not None else 'est_pct=\n')
415-
fh.write(f'est_covered_union={est_covered}\n')
416-
fh.write(f'est_total_union={est_total}\n')
417-
fh.write(f'coverage_status={status}\n')
418-
fh.write(f'coverage_drop={drop:.2f}\n')
419-
fh.write(f'nightly_stale={"true" if stale else "false"}\n')
420-
fh.write(f'nightly_missing={"true" if missing else "false"}\n')
421-
422-
# Informational only on this branch; never fails the job.
423-
# Future tightening can flip to sys.exit(1) when status == 'fail'.
424-
PY

.github/workflows/nightly.yml

Lines changed: 1 addition & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -98,27 +98,7 @@ jobs:
9898
# FAILED list entirely if there are no other final messages,
9999
# making post-mortem on cancelled runs near-impossible.
100100
pytest -m "unit or smoke or integration or slow" -n auto -o "addopts=" -ra \
101-
--cov=src/aragog --cov-report=term-missing --cov-report=xml \
102-
--cov-report=json:coverage.json
103-
104-
- name: Stamp nightly artifact timestamp
105-
if: ${{ !cancelled() }}
106-
run: |
107-
# ISO 8601 UTC; consumed by ci_tests.yml's staleness check (>48 h
108-
# since this stamp triggers a warning, not a hard fail).
109-
date -u +"%Y-%m-%dT%H:%M:%SZ" > nightly-timestamp.txt
110-
111-
- name: Upload nightly coverage artifact
112-
if: ${{ !cancelled() }}
113-
uses: actions/upload-artifact@v4
114-
with:
115-
name: nightly-coverage
116-
path: |
117-
coverage.xml
118-
coverage.json
119-
nightly-timestamp.txt
120-
retention-days: 14
121-
if-no-files-found: warn
101+
--cov=src/aragog --cov-report=term-missing --cov-report=xml
122102
123103
- name: Upload coverage to Codecov
124104
if: ${{ !cancelled() }}

0 commit comments

Comments
 (0)