Support installation through PyPI

[joshua-laughner]

Since we've added support for plain Python virtual environments in GGG2020.1, it would be useful to keep that option in 2020.2. Since uv is installable from PyPI, it must be possible. I suspect that compiling in the netCDF library will be the most annoying part, though that should be doable with a wheel. Gemini has provided an initial recommendation for a Github Action workflow that would handle this, including building for ARM64 just in case. Some notes from the discussion:

• The maturin-action step is supposed to include patching a wheel to ensure that all dynamic libraries are included and pointed to by the RPATH entries
• PyPI distributions can include source for non-Python code, and so if anyone does need to build from source, this should recognize that maturin is the correct build system to use
• Since we are bundling static libraries, we will need to verify that their licenses allow that.

Also note that I will want to modify this to pin specific commits for the actions as a protection against supply chain attacks.

name: CI/CD - Multi-Arch Release
on:
  release:
    types: [published]

permissions:
  contents: read

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }} (${{ matrix.target }})
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          # Linux x86_64
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
          # Linux ARM64 (Native Runner)
          - os: ubuntu-24.04-arm
            target: aarch64-unknown-linux-gnu
          # macOS (Maturin handles Universal binaries by default on macos-latest)
          - os: macos-latest
            target: aarch64-apple-darwin
          # Windows x86_64
          - os: windows-latest
            target: x86_64-pc-windows-msvc

    steps:
      - uses: actions/checkout@v4

      - name: Install Rust toolchain
        uses: dtolnay/rust-toolchain@stable
        with:
          targets: ${{ matrix.target }}

      # Crucial for 'netcdf' static build: needs cmake to compile C dependencies
      - name: Install Build Dependencies (Linux)
        if: runner.os == 'Linux'
        run: sudo apt-get update && sudo apt-get install -y cmake libclang-dev

      - name: Install Build Dependencies (macOS)
        if: runner.os == 'macOS'
        run: brew install cmake

      - name: Build wheels
        uses: messense/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --sdist
          # 'auto' ensures manylinux compatibility for Linux wheels
          manylinux: auto

      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: wheels-${{ matrix.os }}-${{ strategy.job-index }}
          path: dist

  publish:
    name: Publish to PyPI
    runs-on: ubuntu-latest
    needs: [build_wheels]
    permissions:
      id-token: write # Required for Trusted Publishing
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: wheels-*
          merge-multiple: true
          path: dist

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

[joshua-laughner]

This actually seems to work? At least as far as building and installing wheels. So far I've tested the ARM wheel in a Debian Docker container on my Mac, the x86_64 linux wheel on ccycle (no docker), and on my Mac Studio. All worked fine - even able to generate a public netCDF file. It would be good to test on a Mac without netCDF installed, I'll try to do that tonight.

The only note is that if you try to install the wheel with a Python < 3.8, it will say "not supported on this platform," which is the same error you get if the architecture is wrong. That's a little confusing, but hopefully won't matter once this is deployed to PyPI.

I'm holding off on actually doing that; I want more of a chance to think this through. Particularly who owns this on PyPI. It looks like we either have to request an organization, or I start as the owner and can invite a collaborator as a second owner. Organizations seem to be somewhat new, so I don't know how well they work yet. Also want to think about versioning, but I would probably do a parallel version to GGG itself, e.g. 2020.1.0 for this release. The patch number will be a real patch, bug fixes only. I think I would need to go to a "git flow" type structure for the repo then, so I can distinguish fixes from features.

So I still need to enable the upload to PyPI in the workflow, and we probably want it to happen on a release.

[joshua-laughner]

Tested installing from test-pypi, worked on an Ubuntu machine. Will want to switch the workflow over to the main pypi, and do it on releases.

As for installing into GGGPATH, I think we can use pip's --target option as a bridge. When I did pip install --target ./tgt ggg-rs (with ./tgt already created), the result was:

.
└── tgt
    ├── bin
    │   ├── add_nc_flags
    │   ├── apply_nc_fills
    │   ├── apply_tccon_airmass_correction
    │   ├── apply_tccon_insitu_correction
    │   ├── bin2nc
    │   ├── change_ggg_files
    │   ├── collate_tccon_results
    │   ├── create_ak_table
    │   ├── i2s_setup
    │   ├── list_spectra
    │   ├── plot_opus_spectra
    │   ├── query_output
    │   ├── strip_header
    │   └── write_public_netcdf
    └── ggg_rs-2020.1.0rc1.dist-info
        ├── INSTALLER
        ├── METADATA
        ├── RECORD
        ├── REQUESTED
        ├── sboms
        │   └── ggg-rs.cyclonedx.json
        └── WHEEL

Since only the ggg-rs programs end up in bin, we can use this to install them to some dummy directory first, then move the programs to $GGGPATH/bin. Alternately, we just set --target=$GGGPATH and clean up the dist info afterwards.

[joshua-laughner]

It would probably also be a good idea to try to get maturin building with cargo-auditable. There are instructions here, but it will take a little work to make that happen in a GitHub workflow. Presumably we would need to make the wrapper script as a step, then export CARGO=/path/to/cargo-wrapper before calling maturin build