Issue #130: pytest.ini was dead config shadowing pyproject.toml

[jeremymanning]

Closes #130. Part of #108. Corrects a premise of #110. Surfaced #133.

The bug

pytest.ini opened with [tool:pytest] — a section header valid only inside setup.cfg. In a file named pytest.ini, pytest wants [pytest].

pytest still selected pytest.ini as its config file, and having selected one it stops searching. So pyproject.toml's [tool.pytest.ini_options] was never read either. Two config files, zero effective configuration: addopts, testpaths, filterwarnings, all 14 marker declarations and --strict-markers were declared and inert, with no warning from anything.

Why it could not just be flipped

Deleting pytest.ini alone breaks pytest completely:

$ mv pytest.ini /tmp/ && pytest --co -q
ERROR: Refusing to run 'tests/integration': tests/integration provisions real,
billable cloud resources (AWS EKS/EC2). Set CLUSTRIX_ALLOW_BILLABLE=1 ...

pyproject.toml had testpaths = ["tests/unit", "tests/integration"]. With no path on the command line pytest fills config.args from testpaths, and the #109 billable-resources guard inspected config.args. The moment the config became effective, the guard fired against the project itself.

So the guard is fixed first, in its own commit, before any config change.

Commits, in the order they must be read

9c18cb3	Clear 6 collection errors that blocked the --strict-markers gate
176d87e	Regression tests — TDD red, all 4 fail
9c85296	Guard reads invocation_params.args, not config.args
d1d6090	Delete pytest.ini; testpaths = ["tests"]
6d39bd4	Register all 7 markers actually in use
d93f933	Enable --strict-markers (last)
43e7bf3	MIGRATION.md, prime.md
14222d0	pre_push_check.py runs the command CI runs
5483917	gitignore test artifacts; session notes

Each step is independently revertable, which was the point of not squashing.

Two things the plan did not anticipate

The baseline was not clean. Measuring Step 0 turned up 6 collection errors that had been invisible while nothing was configured, and Step 4's gate is "0 errors under --strict-markers" — so they had to go first:

• test_container_registry_comprehensive.py:571 put a backslash inside an f-string expression. SyntaxError on every Python before 3.12, while requires-python is >=3.8. An AST parse over all of tests/ and clustrix/ confirmed it was the only one.
• numpy and pandas were imported at module scope by five modules and declared in no extra. Added to [dev].

pre_push_check.py ran a bare pytest while claiming to verify "GitHub Actions won't fail" — CI runs pytest tests/unit/ -m "not real_world". Harmless only while no config was effective; with testpaths live it resolved to all 1670 tests including the 224 network-bound ones in tests/real_world/ and stopped terminating.

Regression coverage

tests/unit/test_pytest_config.py asserts the loaded config is pyproject.toml, that no higher-precedence config file exists in the tree at all, that all 7 markers are registered, and that --strict-markers is live. All four fail on master.

test_bare_pytest_is_not_refused_by_the_guard covers the landmine. Its scope is stated honestly in its docstring — measured, each fix alone is sufficient and it catches only the combination:

guard reads	testpaths	bare pytest
invocation_params.args	["tests"]	ok
invocation_params.args	["tests/unit","tests/integration"]	ok
config.args	["tests"]	ok
config.args	["tests/unit","tests/integration"]	REFUSED

Verification

configfile                                       pyproject.toml
shadowing config files in repo                   0
bare pytest                                      1670 collected, no abort
pytest tests/integration/<file>                  refused (#109 holds)
tests/ -m "not real_world" integration node IDs  0
markers registered                               7 / 7
collection errors under --strict-markers         0
tests/unit/                                      75 passed
fresh venv, [dev] only, pytest tests/ --co       1670 collected, 0 errors

black and mypy pass. flake8 does not, on 92 findings that predate this branch (91 on it — the f-string fix removed one). CI runs the identical flake8 command with --exit-zero, so this has never been green anywhere. Filed as #133 rather than folded in, because it includes a genuine unrelated bug: test_direct_gpu_detection.py:42 builds a remote Python program inside an f-string without escaping its braces, so {torch.__version__}, {i}, {props.name} and {e} interpolate in the local scope and the module raises NameError before the subprocess starts. It survives precisely because the #109 guard keeps that directory out of CI.

Note on #110

Its addopts/xdist premise quotes the closed epic branch, not master; on master addopts was -v --tb=short and applied to nothing. Its sklearn collection-error claim is also wrong — every sklearn import under tests/ is inside a function body. Two of its acceptance criteria are met here. Corrected in a comment; the issue stays open for the remaining items.

🤖 Generated with Claude Code

https://claude.ai/code/session_012gTBDPK16HUZ3kHQ2QyjuU

---

Update: red-team pass (4 agents) — 2 self-inflicted bugs found and fixed

Per standing instruction to red-team and merge rather than hand over for review. Corrections to what I originally wrote in this PR:

The Step 1 fix was wrong, and made things less safe

Reading config.invocation_params.args unblocked bare pytest but opened two real money-safety holes. Verified against master, which refused both:

PYTEST_ADDOPTS=tests/integration/test_timeout_mechanism.py pytest   -> 2 tests collected
pytest -o testpaths=tests/integration/test_timeout_mechanism.py     -> 2 tests collected

What the operator types is not what pytest collects. testpaths, -o testpaths= and PYTEST_ADDOPTS all feed config.args without ever appearing in the typed argv.

The correct fix was in the config, not the guard: reading config.args is safe because Step 2 set testpaths = ["tests"]. The guard is back on the effective target list. Also fixed a pre-existing #109 hole: relative paths resolved only against rootdir, so from tests/ the path integration/test_x.py was invisible.

attack	before	now
explicit dir / file / node id	refused	refused
-o testpaths=	collected 2	refused
PYTEST_ADDOPTS=	collected 2	refused
--pyargs dotted	collected 2	refused
cwd-relative from tests/	collected 2 (also on master)	refused
bare pytest	ok	ok, 0 integration nodes
CLUSTRIX_ALLOW_BILLABLE=1	works	works

-p <dotted> cannot be blocked before the import it triggers — no conftest hook runs that early — but the run is now refused before any test executes, which is where the cost is. It also fails on its own here, since tests/ is not a package.

Four parametrized regression tests cover the indirect routes; three fail against the old guard.

Claims of mine that were wrong

• "the 224 in tests/real_world/" — wrong, and I had committed it as a code comment. 224 counts @pytest.mark.real_world decorator lines; the directory collects 388 tests.
• The pre_push_check.py rationale — I wrote that the bare pytest was "harmless while no config applied". It was not: on master it aborted in seconds on a collection error and gated nothing. Fixing that SyntaxError is what made the runtime problem appear.
• test_no_shadowing_config_file_exists forbade tox.ini and setup.cfg — pytest checks both after pyproject.toml, so neither can shadow it. The test also failed on an ordinary pytest-free setup.cfg holding flake8 config. Now limited to pytest.ini/.pytest.ini.
• coverage_detailed_report.txt in .gitignore — nothing in the repo writes it; justification was unsupported, entry removed.
• "18/18 CI checks passing" — incomplete. gh pr checks omitted the Fast CI workflow, which was running and was failing.

Unrelated CI bug found — filed, not fixed (#135)

I first added pytest-timeout to .github/workflows/fast_ci.yml and claimed it fixed a job "failing on every run for that reason." That diagnosis was wrong and the change is reverted — fast_ci.yml is now byte-identical to master in this PR.

The workflow is invalid YAML and has never run a single job. Every run, on master too, reports conclusion: failure with jobs: 0:

$ python -c "import yaml; yaml.safe_load(open('.github/workflows/fast_ci.yml'))"
yaml.scanner.ScannerError: while scanning a simple key
  in ".github/workflows/fast_ci.yml", line 89, column 1

Three steps open python -c " inside a run: | block and write the Python body at column 0, which terminates the block scalar. The missing pytest-timeout is real but latent — it only bites once the workflow can start.

Not fixed here: repairing it turns on a workflow whose jobs run real @cluster execution and a Docker build, and whose local-integration step may legitimately fail given #120. That is an unpredictable blast radius inside a PR scoped to pytest configuration. Filed as #135 with the full diagnosis and sequencing.

Also restored

filterwarnings from the deleted pytest.ini, never migrated (third-party paramiko/cryptography noise only). The five other markers it declared are deliberately not migrated — tests/real_world/conftest.py registers them with addinivalue_line where they are used.

Verified after all changes

configfile                   pyproject.toml
bare pytest                  1674 collected, 0 integration nodes
collection errors            0
markers registered           7 / 7
tests/unit/                  79 passed
flake8                       91 (master: 92)
mypy                         clean