pytest.ini is dead config that silently shadows pyproject.toml (--strict-markers, testpaths, markers all inert)

[jeremymanning]

Part of #108 · Phase 1 · Discovered while red-teaming #109.

Problem

pytest.ini uses the section header [tool:pytest]. That header is only valid in setup.cfg. In a file named pytest.ini, pytest requires [pytest].

pytest still selects the file as its config — and having selected it, stops looking, so pyproject.toml's [tool.pytest.ini_options] is never read.

Verified:

$ pytest tests/unit/test_billable_safety.py --co
rootdir: /Users/jmanning/clustrix
configfile: pytest.ini              <- not pyproject.toml

$ pytest --markers | grep -cE "^@pytest.mark.(expensive|real_world|dartmouth_network)"
0                                   <- no project markers registered at all

$ pytest <file with @pytest.mark.definitely_not_registered_xyz> --co -q
1 test collected                    <- --strict-markers is NOT active

Everything currently inert

Both files declare these; neither takes effect:

Setting	Intended	Actual
addopts (-v --tb=short --strict-markers, and -n 4 --dist loadfile in pyproject)	applied to every run	ignored
testpaths	limit collection to tests	ignored — bare pytest collects the whole repo
markers (13 of them)	registered	none registered
filterwarnings	suppress paramiko/crypto noise	ignored
--strict-markers	typo in a marker = error	inert

Why this matters beyond tidiness

1. It invalidates a premise of several other issues. pip install -e ".[dev]" produces an environment where pytest cannot start #110 reports that pip install -e ".[dev]" cannot start pytest because addopts requires xdist. That is true only on branches where pytest.ini is absent — the epic branch deleted it. On master, addopts never applies. Any statement about "the addopts" needs to say which config was live.
2. Marker-based safety is illusory. tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters #109's gate deliberately does not rely on markers, which turns out to have been necessary: applying pytest.mark.expensive today produces PytestUnknownMarkWarning, not a usable selector.
3. --strict-markers being inert hides typos. scripts/test_discovery.py (on the closed epic branch) found 1,532 marker-hygiene issues that a live --strict-markers would have caught at source.

Fix — needs care, do not just flip it

The obvious change ([tool:pytest] → [pytest], or delete pytest.ini so pyproject wins) will activate --strict-markers, testpaths, and possibly -n 4. Any test using an unregistered mark then becomes a hard error, and there are 1,532 known marker issues. Sequence it:

• Decide on ONE config source. Recommendation: delete pytest.ini, keep pyproject.toml (it has the fuller marker list — 13 vs 10 — and is the modern convention).
• Before activating, inventory every mark actually used: grep -rhoE "@pytest\.mark\.[a-z_]+" tests/ | sort -u and reconcile against the registered list.
• Confirm pytest-xdist is installed wherever addopts will now apply (ties to pip install -e ".[dev]" produces an environment where pytest cannot start #110).
• Verify testpaths activation does not pull tests/real_world/ into default runs.
• Land --strict-markers last, once the marker inventory is clean.
• Add a test asserting Config.inifile is the expected file, so a future stray config file cannot silently shadow it again.

Verification

pytest --co -q tests/unit/ 2>&1 | grep -i configfile
pytest --markers | grep -cE "^@pytest.mark."

[jeremymanning]

Handoff plan written — notes/handoff_130_pytest_config.md (commit de09f1a)

A fresh session can pick this up cold. Everything in it was verified on master at a9393b7; every fact carries the command that established it.

⚠️ Landmine discovered while writing the plan

Naively fixing this breaks bare pytest outright. Verified:

$ mv pytest.ini /tmp && pytest --co -q
ERROR: Refusing to run 'tests/integration': tests/integration provisions real,
billable cloud resources (AWS EKS/EC2). Set CLUSTRIX_ALLOW_BILLABLE=1 ...

Cause: master's pyproject sets testpaths = ["tests/unit", "tests/integration"], and pytest populates config.args from testpaths when no paths are given on the command line. The #109 guard inspects config.args, so the moment pyproject becomes live, every bare run aborts.

This is a trap I created in #109 and it must be defused first. The clean fix — verified — is to key the guard off config.invocation_params.args (what the user actually typed) instead:

invocation	config.args	config.invocation_params.args
pytest (testpaths live)	['tests/unit', 'tests/integration']	(no path entries)
pytest tests/unit/	['tests/unit/']	['tests/unit/', ...flags]

Corrections to this issue's original body

• pyproject declares 4 markers on master (real_world, slow, unit, integration), not 13. The longer list I cited was from the closed epic branch. pytest.ini declares 14.
• master's pyproject addopts is just -v --tb=short — no -n 4. So activating pyproject does not require xdist here. The xdist problem is specific to the epic branch, which deletes pytest.ini. pip install -e ".[dev]" produces an environment where pytest cannot start #110 needs the same correction.

Measured marker gap (the thing that gates --strict-markers)

marker	uses	files	declared in pyproject?
real_world	224	—	yes
dartmouth_network	11	4	no
slow	6	—	yes
expensive	5	5	no
performance	4	4	no
integration	2	—	yes
cleanup	1	1	declared nowhere — test_kubernetes_performance_benchmarks.py:1038, likely a typo

Plan shape

6 steps, one commit each, verification commands after every one:

0. Baseline the numbers
1. Defuse the landmine (guard → invocation_params.args) — must be first
2. Delete pytest.ini, set testpaths = ["tests"]
3. Declare the 3 missing markers; decide cleanup
4. Enable --strict-markers last
5. Add a regression test so a stray config can never shadow again
6. Full scripts/pre_push_check.py, then correct pip install -e ".[dev]" produces an environment where pytest cannot start #110, MIGRATION.md:80, .claude/commands/testing/prime.md:95

The regression test uses pytestconfig.inipath and getini("markers"); both were verified against pytest 8.4.2 in the broken and fixed states, so the snippet is known-working rather than assumed. It fails today and passes after Step 2 — write it first and watch it fail.

[jeremymanning]

Fixed in #134 (branch fix/130-pytest-config). Before/after evidence below; the issue closes when that PR merges.

Before — master @ a9393b7

$ head -1 pytest.ini
[tool:pytest]

$ pytest tests/unit/ --co 2>&1 | grep configfile
configfile: pytest.ini

$ pytest --markers | grep -c '^@pytest.mark.real_world'
0

$ printf 'import pytest\n@pytest.mark.bogus_xyz\ndef test_x(): assert True\n' > tests/unit/test_probe_tmp.py
$ pytest tests/unit/test_probe_tmp.py --co -q
1 test collected            # --strict-markers inert

$ pytest tests/ -m "not real_world" --co -q -o addopts= | tail -1
1214/1592 tests collected (378 deselected), 6 errors

After

$ ls pytest.ini tox.ini setup.cfg 2>/dev/null | wc -l
0

$ pytest --co 2>&1 | grep configfile
configfile: pyproject.toml

$ pytest --markers | grep -cE '^@pytest.mark.(real_world|slow|unit|integration|expensive|dartmouth_network|performance):'
7

$ pytest tests/unit/test_probe_tmp.py --co -q
ERROR ... Failed: 'bogus_xyz' not found in `markers` configuration option

$ pytest --co -q | tail -1
1670 tests collected        # bare pytest, 0 errors, no abort

$ pytest tests/unit/ -q | tail -1
75 passed

Checklist from this issue

• Decide on ONE config source — deleted pytest.ini, kept pyproject.toml
• Inventory every mark in use before activating — 7 in use, 4 were declared; added expensive, dartmouth_network, performance; removed the single dead cleanup
• Confirm pytest-xdist is installed wherever addopts applies — not needed: master's addopts has no -n. A comment above addopts now records that adding one requires moving xdist from [test] into [dev] first (pip install -e ".[dev]" produces an environment where pytest cannot start #110)
• Verify testpaths activation does not pull tests/real_world/ into default runs — it does collect them (testpaths = ["tests"]), which is correct and matches the previous no-config behaviour. Nothing runs them by default: CI uses pytest tests/unit/ -m "not real_world", and scripts/pre_push_check.py now does the same, which it previously did not
• Land --strict-markers last — Step 4, after the marker inventory was clean
• Add a test asserting the config file is the expected one — tests/unit/test_pytest_config.py, four assertions, all failing on master

Two corrections to this issue's own text

"13 markers" / "1,532 marker issues." pytest.ini declared 10 markers and pyproject.toml 4. The suite uses 7 distinct non-builtin markers across 253 applications. The 1,532 figure came from scripts/test_discovery.py on the closed epic branch and does not correspond to anything measurable on master.

The landmine was not in the issue. Deleting pytest.ini alone breaks pytest entirely, because testpaths then feeds config.args and the #109 guard refuses the run. That had to be fixed first, in its own commit. Details in #134 and in notes/handoff_130_pytest_config.md.

Found along the way

• The Step 0 baseline had 6 collection errors, not zero: an f-string backslash SyntaxError (fatal on Python <3.12 while requires-python is >=3.8) plus undeclared numpy/pandas. Fixed here, since Step 4's gate is "0 errors".
• scripts/pre_push_check.py cannot exit 0 for unrelated reasons, and never could — filed as pre_push_check.py can never pass: 92 flake8 findings, incl. a real f-string bug in test_direct_gpu_detection.py #133, which also documents a real f-string bug in tests/integration/test_direct_gpu_detection.py.
• pip install -e ".[dev]" produces an environment where pytest cannot start #110's addopts premise and its sklearn claim are both wrong; corrected there.