Automate builds of source and docs artifacts with GitHub Actions

[sethmlarson]

This issue assumes that Release Managers are using the run_release.py script, the "current" ordering is derived from that script. Implementing the proposal below would require changes to both run_release.py and release.py. Expand the diagrams below to see a visualization of the proposal.

Current release process

• There are two stages where Sigstore is used to sign the artifacts.
• Platform experts are only tagged in once the source and docs build has been uploaded to python.org
• Source and docs builds can't verifiably originate from a specific git tag/commit.

Expand for diagram

flowchart TD
    pull_fork[git pull upstream -> fork]
    blurb[blurb+bump]
    tag[git tag]
    push_fork[git push fork]
    build_artifacts[Build release artifacts]
    gpg_and_sigstore[Sign with GPG and Sigstore]
    sigstore[Sign with Sigstore if needed]
    test_artifacts[Test release artifacts]
    upload_artifacts[Upload release artifacts]
    platform_builds[macOS and Windows builds]
    push_upstream[git push upstream]

    pull_fork --> blurb
    blurb --> tag
    tag --> build_artifacts
    build_artifacts --> gpg_and_sigstore
    gpg_and_sigstore --> test_artifacts
    test_artifacts --> upload_artifacts
    upload_artifacts --> push_fork
    push_fork --> platform_builds
    platform_builds --> sigstore
    sigstore --> push_upstream

Loading

Proposed release process

Differences:

• Platform builds can occur concurrently to source build+test+docs.
• Source build along with testing and docs builds occur concurrently.
• Only one GPG and Sigstore signing step, at the very end in add-to-pydotorg.py.
• Would require rebuilding and repushing a new git tag+commit if a problem is detected during source/docs builds. I don't know how frequently this is an issue.
• Isolates the docs and source builds from eachother because the docs build requires many more dependencies (both from the OS package manager and from Doc/requirements.txt).

Expand for diagram

flowchart TD
    pull_fork[git pull upstream -> fork]
    blurb[blurb+bump]
    tag[git tag]
    push_fork[git push fork]
    run_workflow[Run GitHub Workflow]
    build_artifacts[Build release artifacts]
    build_docs[Build docs]
    gpg_and_sigstore[Sign with GPG and Sigstore]
    test_artifacts[Test release artifacts]
    upload_artifacts[Upload release artifacts]
    platform_builds[macOS and Windows builds]
    push_upstream[git push upstream]

    pull_fork --> blurb
    blurb --> tag
    tag --> push_fork
    push_fork --> run_workflow
    push_fork --> platform_builds

    subgraph GitHub Actions
    run_workflow --> build_artifacts
    run_workflow --> build_docs
    build_artifacts --> test_artifacts
    build_docs
    end

    build_docs --> upload_artifacts
    test_artifacts --> upload_artifacts
    platform_builds --> upload_artifacts
    upload_artifacts --> gpg_and_sigstore
    gpg_and_sigstore --> push_upstream

Loading

In order to implement the above proposal, I would do the following:

• Create a new GitHub workflow which is manually triggerable using a git commit, tag, and repo which creates source and docs tarballs as artifacts.
• Move the GPG signing step to add-to-pydocorg.py (used by run_release.py). Remove Sigstore signing from release.py.
• Create some method of detecting the source and docs tarballs by the run_release.py script
• Move the source/docs upload steps in run_release.py to be after all artifacts are ready.

After the changes have been implemented, I can modify the process for PEP 101 as necessary. I'm hoping to get some clarifications where I have things wrong from release managers, let me know!