Tests: cassette/HTTP-server mocking infrastructure#292
Open
jeandet wants to merge 41 commits into
Open
Conversation
Captures decisions on UV adoption, hatchling build backend, ruff/basedpyright tooling, and three-tier test strategy (unit/contract/e2e). Sequences the work as 17 small PRs ending with a mass reformat. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-task implementation plan for the first PR of the modernisation effort. Covers pyproject.toml updates, uv.lock generation, CI/RTD switch to uv, deletion of requirements*.txt / tox.ini / setup.cfg, and developer-doc updates to drop the PYTHONPATH=. pattern. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add SPEASY_CORE_HTTP_REWRITE_RULES env to PRs.yml non-3.10 pytest step (previously only on push/scheduled tests.yml — would have hit a non-existent server on PR builds for non-3.10 matrix entries). - Add --with wheel to PRs.yml build step for parity with tests.yml. - Scope flake8 to 'speasy tests' in both workflows (matches Makefile lint target). Avoids silently broadening lint to docs/conf.py and removes the .venv exclusion workaround that was needed when flake8 ran from repo root.
Without UV_PROJECT_ENVIRONMENT, uv creates .venv/ inside the project and RTD's sphinx step (which calls $READTHEDOCS_VIRTUALENV_PATH/bin/python directly) fails with 'python: not found'. Point uv at RTD's venv so the install lands where the runner looks for it.
Classified via devtools/apply_test_markers.py: - 12 files marked unit (pure-logic, no network) - 19 files marked contract (real-server, will be migrated to cassettes in PRs 4-9) Reclassifications during manual review: - test_cache.py: contract -> unit (pure cache-logic, no network or speasy provider use) - test_file_access.py: unit -> contract (uses HTTP via any_loc_open against live servers) test_wasm.py was manually adjusted to place pytestmark at module level (the file's body lives inside a try/except ImportError block, so the script's naive insertion landed at wrong indentation).
…le path and sample across the inventory The flat_inventories.generic_archive lookup uses module attribute access on an instance, not a submodule import. Also, the first N parameters in the flat inventory are clustered by mission, so a fixed time range can miss all of them; sample across the full list instead.
- test_e2e_smoke.test_generic_archive: fail loudly if every candidate raises (was silently skipping, defeating the e2e tier's purpose). - pyproject.toml: drop dead --ignore=setup.py from addopts and document the -m unit override semantics so future contributors don't trip on 'pytest tests/test_amda.py' silently collecting nothing. - contract.yml / e2e.yml: add concurrency groups so a manual run can't overlap with a cron run hammering the same upstream servers. - CONTRIBUTING.rst: add a short note explaining the three test tiers and how to invoke each from local dev.
CI failure (blocking): - unit.yml: 'make doctest' was using system Python (no sphinx in scope). Prefix with 'uv run' so make uses the project venv. Reviewer findings: - wasm_tests.yml: pytest tests/test_wasm.py without -m collected 0 tests under the new addopts default (test_wasm.py is contract-marked). Add -m '' to override. - CLAUDE.md: examples like 'uv run pytest tests/test_amda.py' silently collected 0 tests under -m unit default. Replaced with tier-aware examples and added -m '' for the all-tests case. - unit.yml: only sync --group docs on the coverage runner that needs it, not on every matrix entry.
nbsphinx requires the system pandoc binary (not the Python pandoc wrapper that's in the docs dependency group). PR 1's tests.yml had 'sudo apt install -y texlive pandoc' before make doctest; my unit.yml rewrite in PR 2 dropped that line, so the doctest job failed with 'nbsphinx.NotebookError: PandocMissing in examples/AMDA.ipynb'. Restored as a separate apt step on the coverage runner.
The doctest step's examples reference all data providers (cdpp3dview included) and live inventories. The job-level SPEASY_CORE_DISABLED_PROVIDERS='cdpp3dview' makes the inventory tree's cdpp3dview attribute missing during doctest, surfacing as 'types.SimpleNamespace object has no attribute cdpp3dview' and a chain of NameErrors for variables defined in earlier doctest blocks. Original tests.yml overrode SPEASY_CORE_DISABLED_PROVIDERS="" on the combined pytest+doctest step, plus set HTTP_REWRITE_RULES (re-routes the placeholder URL used in some examples to LPP's mirror) and USER_AGENT. My PR 2 rewrite dropped the env block; restoring it on the doctest step.
Pandas now prints its public name in type() repr ('pandas.DataFrame')
rather than the internal module path ('pandas.core.frame.DataFrame').
The user/numpy.rst doctest was written against the old form.
Surfaced now that uv.lock pins a recent pandas; pip-installed envs
were getting older pandas where the old form still applied.
- vcr_config: broaden scrub list (add bearer, x-api-key, sessionID, apikey/api_key) to harden cassette recording against accidental secret leaks once PRs 4-9 start capturing real traffic. - tmp_speasy_cache: rewrite docstring to honestly describe the module-level singleton limitation. The Cache instance is constructed at import time of speasy.core.cache._instance, so monkeypatching SPEASY_CACHE_PATH later has no effect on the live singleton. Cassette-migration PRs will introduce a sturdier isolation mechanism if needed. - test_infra_smoke: switch urllib.request -> requests for consistency with the rest of the codebase. - test_infra_smoke: add test_record_mode_none_blocks_unmocked_requests asserting vcrpy raises CannotOverwriteExistingCassetteException when an unmocked HTTP call is made under record_mode='none'. Guards against pytest-recording silently disengaging if a fixture is renamed or its scope changes.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if any("import pytest" in ln for ln in lines): | ||
| block = [f"\n{MARKER_LINE_PREFIX}{tier}\n", "\n"] | ||
| lines[insert_at:insert_at] = block | ||
| path.write_text("".join(lines)) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #292 +/- ##
===========================================
- Coverage 85.10% 68.61% -16.49%
===========================================
Files 69 82 +13
Lines 4834 5092 +258
Branches 668 693 +25
===========================================
- Hits 4114 3494 -620
- Misses 479 1374 +895
+ Partials 241 224 -17
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Two PR 3 bugs found while implementing PR 4:
1. tests/conftest.py: vcr_config dict set record_mode="none",
which unconditionally overrode the --record-mode CLI flag (per
pytest-recording/_vcr.py:82-83) and made re-recording impossible.
Drop the key — pytest-recording's session fixture already defaults
record_mode to "none", and --record-mode now works as expected.
2. devtools/apply_test_markers.py: the idempotency guard only
recognized single-form pytestmark ("pytestmark = pytest.mark.X"),
not list-form ("pytestmark = [pytest.mark.X, pytest.mark.Y]").
Re-running the script over a list-form file would re-add a
single-form marker. Recognize both.
Cassettes will not live in the git repo. Instead they are hosted at https://sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/ behind HTTP Basic auth, identified by sha256 of their uncompressed content. Components: - tests/cassettes_manifest.json: maps each cassette path under tests/cassettes/ to its sha256. Empty in PR 3; populated by PR 4 onwards when AMDA/CDA/etc. cassettes are recorded. - tests/conftest.py: pytest_configure hook reads the manifest at session start, downloads any missing cassettes from the storage server (auth via SPEASY_CASSETTE_FETCH_USER/PASSWORD env vars or ~/.netrc), verifies the sha, decompresses to tests/cassettes/. Uses an XDG_CACHE_HOME-rooted local cache to avoid re-downloading across runs. New --no-cassette-fetch CLI flag opts out of fetching. - devtools/publish_cassettes.py: maintainer-only staging tool. Walks tests/cassettes/, hashes each .yaml, gzips deterministically (mtime=0) into .publish_staging/<sha>.yaml.gz, and rewrites the manifest. Prints the rsync command for the maintainer to run; this script does not upload itself. - .gitignore: ignore tests/cassettes/* (keep .gitkeep) and .publish_staging/. - CI workflows (unit.yml, contract.yml, e2e.yml): inject SPEASY_CASSETTE_FETCH_USER and SPEASY_CASSETTE_FETCH_PASSWORD secrets so the conftest fetch hook works on GitHub runners. - CONTRIBUTING.rst: document the new flow for both contributors (set env vars / netrc) and maintainers (record, publish, rsync, commit manifest).
This was referenced May 10, 2026
Two infrastructure fixes that originally surfaced during PR 5 (CDA) and PR 9 (CDPP3DView) cassette migrations, backported here where they belong: 1. speasy/core/http.py Response.url: try/except AttributeError fallback for vcrpy's VCRHTTPResponse which delegates geturl() to http.client that reads self.url — an attribute the cassette response doesn't carry. Without this, every cassette-replayed response that triggers a debug-log of resp.url crashes. 2. tests/conftest.py _canonical_rewrite_rule_for_vcr autouse fixture: Pin speasy.core.url_utils._REWRITE_RULES_ (cached at module import time) to the recording-time policy for any vcr-marked test. Without this, a developer with a custom http_rewrite_rules entry in ~/.config/speasy/config.ini sees replay failures because the replay-side URL no longer matches the cassette.
vcrpy's filter_headers and filter_query_parameters only scrub the REQUEST side. Response headers like Set-Cookie (JSESSIONIDs from CSA, session tokens from AMDA) and certain response bodies (AMDA's auth.php returns a 32-char hex hash that may be derivable from credentials) were being committed verbatim into cassettes. Add before_record_response callback in vcr_config: - Drops Set-Cookie response headers - Replaces any 32-char hex body (matching AMDA auth.php response shape) with <SCRUBBED> This guards future recordings. Existing cassettes are scrubbed in a follow-up one-shot script (see PR description).
Sister script to the new before_record_response callback in conftest. Scrubs the same patterns (Set-Cookie response headers, 32-char hex auth.php response bodies) from cassettes that were recorded before the callback existed. Idempotent — safe to re-run. Used once to retroactively clean the existing AMDA + CSA cassettes on the modernisation/pr3-mocking-infra branch. Future recordings are automatically scrubbed at record time by the conftest callback.
The cassette hosting at sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/ is now public-read. Cassettes are content-addressed by sha256 — URLs are unguessable for outsiders and any tampering is caught on download via the existing hash verification in _fetch_cassette. Practical benefits: - Fork PRs can run the cassette-replaying unit tier (previously blocked: GitHub Actions doesn't expose repo secrets to fork PRs). - New contributors need no credential setup to run the tests. - CI workflows lose the SPEASY_CASSETTE_FETCH_USER/PASSWORD env injection (no longer needed). Cassettes are still scrubbed (Set-Cookie response headers and AMDA auth.php hash response bodies) by the before_record_response callback in vcr_config, so no session/credential material reaches the cassette content itself.
The conftest is imported by every pytest run, including the wasm_tests.yml workflow that uses a minimal Python env (it installs pytest-pyodide but not the project's full dev group). numpy is only needed by the speasy_variable_factory fixture's _make closure; importing it at module level breaks wasm test collection with ModuleNotFoundError. Move the import inside the fixture body so non-unit-tier callers don't need it.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third PR of the modernisation effort (spec:
docs/superpowers/specs/2026-05-08-speasy-modernisation-design.md, plan:docs/superpowers/plans/2026-05-09-pr3-mocking-infrastructure.md).Stacked on PR #291 (test markers + CI split), which itself stacks on PR #290 (UV foundation). This PR's diff includes both predecessors until they merge in order.
What this PR does
pytest-recording,pytest-httpserver,pytest-sugarto thedevdependency group.tests/conftest.pywith shared fixtures:vcr_config/vcr_cassette_dir: cassette replay config (defaultrecord_mode = "none"; cassettes per test module undertests/cassettes/<module>/). Scrub list coversauthorization,bearer,cookie,set-cookie,x-api-key(headers) andapikey/api_key/password/sessionID/token/userID(query params).tmp_speasy_cache: per-testSPEASY_CACHE_PATHsetter (with an honest docstring about the module-level singleton limitation).speasy_variable_factory: builds a minimalSpeasyVariablefor unit tests._disable_proxy_for_unit_tier(autouse): setsSPEASY_PROXY_ENABLED=falsefor unit-marked tests so they cannot be silently satisfied by a hot proxy hit.tests/cassettes/as the cassette directory.tests/test_infra_smoke.py(6 unit tests) so the infrastructure has its own regression coverage — includingtest_record_mode_none_blocks_unmocked_requestswhich asserts vcrpy actually intercepts traffic (catches regressions where pytest-recording silently disengages).CONTRIBUTING.rst.https://sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/behind HTTP Basic auth. Apytest_configurehook readstests/cassettes_manifest.jsonat session start, fetches missing cassettes by content hash, caches them under~/.cache/speasy-tests/, and decompresses them totests/cassettes/. CI usesSPEASY_CASSETTE_FETCH_USER/SPEASY_CASSETTE_FETCH_PASSWORDGitHub secrets.devtools/publish_cassettes.py(maintainer tool): hashes cassettes, gzips deterministically, stages them in.publish_staging/for the maintainer to upload via rsync, and updates the manifest.What this PR does NOT do
Test plan
unit.ymlgreen across full matrix (now 494 unit tests = 488 existing + 6 infra-smoke)lint.ymlgreenuv run pytest tests/test_infra_smoke.py— all 6 passuv run pytest -m unit— 494 pass