-
Notifications
You must be signed in to change notification settings - Fork 10
Restructure CI/CD pipeline: consolidate workflows, reorganize tests, fix PyPI publishing #680
Copy link
Copy link
Open
Description
Problem
- 9+ workflow files, many deprecated or broken
- Versioning workflow broken — PAT expired, preventing PyPI publishing
- Tests disorganized — unit and integration tests mixed together, some tests in wrong directory and never discovered by pytest
- No test coverage reporting
- Integration tests run on every PR (~3 hours) even for doc-only changes
Solution
Fix PyPI publishing
Migrate versioning workflow from expired PAT to GitHub App token.
Reorganize tests
Move all tests to top-level tests/ with unit/ and integration/ split. Integration tests named per-dataset. Merge sanity tests into per-dataset files. Fix broken tests. Remove test_reproducibility.py (imported nonexistent modules since July 2025).
Consolidate workflows
Replace 9 workflow files with 4:
pr.yaml— Fork check, lint, uv.lock, changelog, unit tests with Codecov, smoke test. Integration tests only withrun-integrationlabel.push.yaml— Single-container Modal data build with integration tests, manual approval gate, pipeline dispatch. PyPI publish on version bumps.pipeline.yaml— Dispatch-only H5 generation with scope filtering.versioning.yaml— Auto version bump with GitHub App token.
Add Codecov
Informational coverage reporting on PRs (non-blocking).
Prerequisites (manual setup after merge)
- Create
pipeline-approvalGitHub environment with required reviewers - Add
CODECOV_TOKENsecret from codecov.io
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels