Skip to content

Update repodata to v3 syntax#273

Merged
danyeaw merged 10 commits intoconda:mainfrom
danyeaw:v3-syntax
Mar 17, 2026
Merged

Update repodata to v3 syntax#273
danyeaw merged 10 commits intoconda:mainfrom
danyeaw:v3-syntax

Conversation

@danyeaw
Copy link
Member

@danyeaw danyeaw commented Mar 12, 2026 •
edited
Loading

Description

Closes #247. In py-rattler 0.23.0, they updated support for the v3 repodata with conditionals/extras which is defined in:

  1. CEP XXXX: Conditional dependencies, extras and flags ceps#111
  2. CEP XXXX: A backwards-compatible repodata update strategy ceps#146

Currently the conditional/extras CEP draft defines the extras field to be called extras. py-rattler is using the field depends_extra, so I am going with that to match the implementation and I posted about this on Zulip.

This also enabled usage of conditional markers which requires quite a lot of conversion:

  • Convert Python markers to python... MatchSpec fragments, including python_version not in "x, y" -> (python!=x and python!=y).
  • Convert platform/os markers to virtual packages when feasible (__win, __linux, __osx, __unix).
  • Keep extras in extra_depends, with remaining non-extra marker logic encoded via [when="..."].
  • Drop unsupported marker dimensions (e.g. interpreter/machine-specific variants) including platform_release, platform_version, implementation_version. These markers are rarely used and difficult to map to equivalents.

One thing I'm not sure about is that I am using a private _marker method from the packaging library in one place. I tried to keep the usage isolated to one area, and it is for only test metadata generation. I think this is much better than using regex to parse the markers.

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@github-project-automation github-project-automation bot moved this to 🆕 New in 🔎 Review Mar 12, 2026
@danyeaw danyeaw mentioned this pull request Mar 12, 2026
Open
@danyeaw
Copy link
Member Author

danyeaw commented Mar 13, 2026

Hey @baszalmstra, I know you have spent some time thinking about conditional marker conversion. Would you mind please taking a quick peak at the generate_noarch_wheel_repodata script and make sure something doesn't stand out for you?

@baszalmstra
Copy link

baszalmstra commented Mar 13, 2026

I think the best way to test this is to have a large set of conditions/requirements convert them to conda requirements and snapshot them. That makes it easy to see what the conversion will do in many different cases, whether it makes, and whether it regressed. (You might already have that, I might have missed it)

@danyeaw
Copy link
Member Author

danyeaw commented Mar 16, 2026

Hey @baszalmstra, with the changes here: conda-incubator/conda-pypi-test#19, I ran python generate.py which generated the repodata for 512K packages. I didn't look at all of it myself of course, but it looked like it is working great for the 99%+ of packages. The few extra packages I added to this PR are specifically to test some of the edge cases I ran in to.

@danyeaw danyeaw marked this pull request as ready for review March 16, 2026 17:04
@danyeaw danyeaw requested a review from a team as a code owner March 16, 2026 17:04
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 16, 2026 •
edited
Loading

Merging this PR will degrade performance by 30.24%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 (👁 1) regressed benchmark
✅ 4 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
👁 test_build_conda[imagesize] 51.1 ms 73.3 ms -30.24%

Comparing danyeaw:v3-syntax (d94bc67) with main (836a0f3)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (609401a) during the generation of this report, so 836a0f3 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

danyeaw
danyeaw commented Mar 16, 2026
View reviewed changes
ryanskeith
ryanskeith approved these changes Mar 16, 2026
View reviewed changes
Copy link
Contributor

@ryanskeith ryanskeith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good to me. Sadly, the best way to get the parsed atoms from the markers for easy parsing is through a private member function. Another way would be to use another private function from packaging either

from packaging._parser import parse_marker

or

from packaging._parser import parse_requirements

Either would give the same parsed _markers result that is being used. Having no public programmatic access to the individual parts has has been marked as an issue for a few years in the packaging module.

A few comments that showed examples code be helpful for future reviewers to quickly understand the intent of the transform.

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Approved in 🔎 Review Mar 16, 2026
danyeaw and others added 2 commits March 17, 2026 08:16
@danyeaw danyeaw merged commit 22bf68d into conda:main Mar 17, 2026
40 checks passed
@github-project-automation github-project-automation bot moved this from ✅ Approved to 🏁 Done in 🔎 Review Mar 17, 2026
@danyeaw danyeaw deleted the v3-syntax branch March 17, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ryanskeith ryanskeith ryanskeith approved these changes

Assignees

No one assigned

Labels

cla-signed

Projects

🔎 Review
Status: 🏁 Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Updated repodata v3 support

4 participants

@danyeaw @baszalmstra @ryanskeith @conda-bot