Add TOML v1.1 -> v1.0 backwards compatibility for source distributions#18741
Open
Add TOML v1.1 -> v1.0 backwards compatibility for source distributions#18741
Conversation
konstin
added
the
enhancement
konstin
force-pushed
the
konsti/toml-backwards-compatibility
branch
2 times, most recently
from
f0d7ebe to
d580ff9
Compare
[TOML 1.1](https://github.com/toml-lang/toml/releases/tag/1.1.0) introduces support for new syntax that older tools with TOML 1.0 don't understand. Usually, the user is either in control of which tools need to read the TOML files or the TOML gets converted before publishing (`pyproject.toml` -> `METADATA` for wheels). The specific case where this doesn't work is when a user builds the source distribution of the package with a tool that only support TOML 1.0. Build tools need to parse `pyproject.toml` in source distributions to extract the `[build-system]` table, and if any other part of the file contains TOML 1.1 syntax, they fail to build. This generally doesn't trigger backtracking, so the user is left if a failure when any (transitive) dependency in their dependency tree has started using a single instance of TOML 1.1. Most package managers, including pip, are implemented in Python and use stdlib's tomllib, which only support TOML 1.0 up to including Python 3.14. To work around this, we do a best-effort rewrite of `pyproject.toml` to TOML 1.0 during source distribution builds. This approach is inspired by Cargo, which is successfully rewriting published `Cargo.toml`s for many versions. While the `toml` crate doesn't guarantee this downgrade is always done (toml-rs/toml#1088), this crate is also used by Cargo, and this best effort rewrite is sufficient currently. Similarly following Cargo, we also add a `pyproject.toml.orig` to the source distribution. https://discuss.python.org/t/adopting-toml-1-1/105624 went nowhere, but a best-in-class tool should do this transformation, so we're adding it.
konstin
force-pushed
the
konsti/toml-backwards-compatibility
branch
from
d580ff9 to
9edc16e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TOML 1.1 introduces support for new syntax that older tools with only TOML 1.0 support don't understand.
Usually, the user is either in control of which tools need to read the TOML files or the TOML gets converted before publishing (for wheels:
pyproject.toml->METADATA). The specific case where this doesn't work is when a package manager that only support TOML 1.0 tries to build the source distribution of a dependency. Build tools need to parsepyproject.tomlin source distributions to extract the[build-system]table, and if any other part of the file contains TOML 1.1 syntax, they fail to build. This generally doesn't trigger backtracking, so the user is left with a failure when any (transitive) dependency in their dependency tree has started using a single instance of TOML 1.1. Most package managers, including pip, are implemented in Python and use stdlib's tomllib, which only support TOML 1.0 up to including Python 3.14.To work around this, we do a best-effort rewrite of
pyproject.tomlto TOML 1.0 during source distribution builds.This approach is inspired by Cargo, which has been successfully rewriting published
Cargo.tomls for many versions. While thetomlcrate doesn't guarantee this downgrade always works (toml-rs/toml#1088), this crate is also used by Cargo, and this best effort rewrite handles the biggest failure case: Newlines and trailing commas in inline tables. Similarly following Cargo, we also add apyproject.toml.origto the source distribution.https://discuss.python.org/t/adopting-toml-1-1/105624 was inconclusive, but a best-in-class tool should do this transformation.