[Roadmap] Implement PEP 740

[woodruffw]

PEP 740 is in a final but not yet approved state. This issue is intended to lay out the dependencies/subcomponents of its implementing, including things that can be done in a preliminary manner.

Index side

• Support the attestations form field in the current legacy upload endpoint
  • Add support for uploading attestations in legacy API #15952
• Parse each received attestation from attestations and verify each against the uploading trusted publisher; fail the file upload if any attestations are invalid
  • This should be done in as generic a manner as possible, to ensure that future extension to email and key identities is possible.
  • Add support for uploading attestations in legacy API #15952
  • Dockerfile: put some XDG dirs under /tmp #16304
  • Store attestations for PEP740 #16302
• Persist verified attestations and expose them via the PEP 503 and Simple JSON API indices (via provenance objects, as specified in PEP 740)
  • Provide Trusted Publisher information pypi's API #14799
  • Project Details Sidebar Statistics behavior #14727
  • Reapply "Store attestations for PEP740 (#16302)" (#16545) #16546

Uploader/publish side

• Support attestation generation in gh-action-pypi-publish
  • Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740
  • The resulting bundle should be transformed into a PEP 740 attestation object
  • Expose PEP 740 attestations functionality pypa/gh-action-pypi-publish#236
  • requirements: re-compile requirements with latest twine pypa/gh-action-pypi-publish#245
• Support attestation uploading in twine: Proposal: preliminary support for PEP 740 pypa/twine#1094
  • twine/upload: attestations scaffolding pypa/twine#1095
  • upload: add attestations to PackageFile pypa/twine#1098
  • upload: prevent --attestations on non-PyPI indices pypa/twine#1099
  • upload: turn attestation error into a warning pypa/twine#1101
  • Update changelog for 5.1.0 pypa/twine#1107
• Once twine supports uploading, gh-action-pypi-publish should use that support to actually upload the attestation objects it generates above

Docs & UI

• Update all relevant docs (PyPI's docs, PyPUG)
  • PEP 740: initial attestation docs #16063
  • Initial PEP 740 documentation #16398
• Add a per-artifact view #16491
• PEP 740 blog post #17070

CC @di @webknjaz @facutuesca for visibility

[webknjaz]

I've gone ahead and filed a tracking issue in pip-tools as well: jazzband/pip-tools#2080.

[webknjaz]

• Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740

With the exciting changes suggested in pypa/gh-action-pypi-publish#230, it looks like we'll be able to leverage bits of composite actions and call the action even (if it makes sense to do so, as an alternative to using a Python lib within the container like the rest of stuff).

[webknjaz]

• Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740

Should this be done by the uploader, though? My initial feeling was that this should be done right after the job building the artifacts. Though, perhaps not by pypa/build (because of the privilege elevation concerns I often complain about). Are we going to treat the artifacts that the uploader gets as coming from trusted sources? Should we be emphasizing the security considerations and teaching the users more about the least privilege principle?

[woodruffw]

Should this be done by the uploader, though?

Hmm -- I suppose it doesn't have to be, although for targeting publish provenance (and getting provenance generation exposed to the largest number of users) I think it's the most straightforward path.

Long term, PEP 740 is designed to enable multiple attestations: the building step(s) can generate one, as can the publishing step. But I believe getting that into a workable state will require a decent bit more coordination, so I want to target an MVP of "the artifact is signed by the same identity that pushed it to the index" 😅

[webknjaz]

@woodruffw fair! I just wanted to make sure we're on the same page.

[sethmlarson]

@webknjaz Glad you're thinking about build provenance already, I'm interested in tackling build provenance for Python too 🚀

These initial attestations are similar to NPM's publish provenance so is best done by the uploader. If the attestation can show "this artifact was uploaded by this workflow which matches a Trusted Publisher configuration for the project" we can already build some interesting things like verified GitHub URLs, exposing which artifacts have Trusted Publishers configured to users, monitoring for changes in provenance, etc

[webknjaz]

@sethmlarson fair enough! I remember similar ideas being brought up during the initial private beta of trusted publishing. It's nice to have some more or less specified goals recorded and synchronized, though, which is why I posted my thoughts :)

Looking back, I'm glad that I encouraged implementing it in my action from the very beginning, months before it's gone GA so everybody could start using it right away with almost no changes. So I'm hoping that the Sigstore integration happens in a similar fashion, enabling any early adopters to start uploading signatures with zero changes to their workflows (if they already use trusted publishing, they have the OIDC access set up).

[di]

Should this be done by the uploader, though? My initial feeling was that this should be done right after the job building the artifacts.

I think your initial feeling is valid: since it's not guaranteed that the workflow using the publish action is also the workflow that built the artifacts, this isn't necessarily "build provenance".

That said, having an attestation ('publish provenance') from the uploader in the absence of any other attestation being provided to it would still be very valuable here I think, especially for projects that aren't using Trusted Publishing yet.

[facutuesca]

@woodruffw What behavior do we want when uploading attestations to a file already in PyPI? Currently, the behavior for duplicate files is either stop the upload process and return OK (if the new file is the same as the existing one), or fail (if the files don't match).
How do we want to deal with those cases when the request also includes attestations?

[woodruffw]

What behavior do we want when uploading attestations to a file already in PyPI? Currently, the behavior for duplicate files is either stop the upload process and return OK (if the new file is the same as the existing one), or fail (if the files don't match).

For the MVP, I think we want to skip if the distribution file already exists, for consistency with the existing upload API behavior.

Longer term, this probably falls under the "Upload 2.0 API" PEP -- we'll want a way to augment existing uploaded distributions with new attestations. But that's out of scope for now, IMO 🙂