Skip to content

Restructure CI/CD pipeline: consolidate workflows, reorganize tests, fix PyPI publishing #680

@anth-volk

Description

@anth-volk

Problem

  • 9+ workflow files, many deprecated or broken
  • Versioning workflow broken — PAT expired, preventing PyPI publishing
  • Tests disorganized — unit and integration tests mixed together, some tests in wrong directory and never discovered by pytest
  • No test coverage reporting
  • Integration tests run on every PR (~3 hours) even for doc-only changes

Solution

Fix PyPI publishing

Migrate versioning workflow from expired PAT to GitHub App token.

Reorganize tests

Move all tests to top-level tests/ with unit/ and integration/ split. Integration tests named per-dataset. Merge sanity tests into per-dataset files. Fix broken tests. Remove test_reproducibility.py (imported nonexistent modules since July 2025).

Consolidate workflows

Replace 9 workflow files with 4:

  • pr.yaml — Fork check, lint, uv.lock, changelog, unit tests with Codecov, smoke test. Integration tests only with run-integration label.
  • push.yaml — Single-container Modal data build with integration tests, manual approval gate, pipeline dispatch. PyPI publish on version bumps.
  • pipeline.yaml — Dispatch-only H5 generation with scope filtering.
  • versioning.yaml — Auto version bump with GitHub App token.

Add Codecov

Informational coverage reporting on PRs (non-blocking).

Prerequisites (manual setup after merge)

  • Create pipeline-approval GitHub environment with required reviewers
  • Add CODECOV_TOKEN secret from codecov.io

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions