pypy next steps

[ericdill]

TODO:

• (0) python_abi package
  • Add python_abi staged-recipes#10760
  • Use build number in string python_abi-feedstock#2
• (1) Fix conda-build
  • PyPy fixes conda/conda-build#3868
  • conda-build v3.18.12 conda-build-feedstock#124
  • Keep python pinning in hashing if there's a space conda/conda-build#3895
  • Add conda/conda-build#3895 conda-build-feedstock#126
• (2) Fix python package
  • Add _cpython to build string python-feedstock#309
  • noarch run_exports python-feedstock#310
  • Backport to 2.7 - Add python_abi for 2.7 branch python-feedstock#313
  • Backport to 3.6 - Add python_abi for 3.6 branch python-feedstock#312
  • Backport to 3.7 - Add python_abi for 3.7 branch python-feedstock#311
• (3) Hotfix metadata
  • Make existing python and python-dependent packages depend on python_abi conda-forge-repodata-patches-feedstock#27
  • Fix for conda-forge/python_abi-feedstock#2 conda-forge-repodata-patches-feedstock#28
  • Workaround for conda-build issues conda-forge-repodata-patches-feedstock#31
  • defaults - Add a constraint on python_abi AnacondaRecipes/repodata-hotfixes#71
  • defaults - revert Revert "Add a constraint on python_abi" AnacondaRecipes/repodata-hotfixes#76 (Workaround at Add strict priority for pypy builds conda-forge-ci-setup-feedstock#70)
• (4) Fix https://github.com/conda-forge/conda-forge-pinning-feedstock
  • Add _cpython pinning to python conda-forge-pinning-feedstock#394
• (5) Fix pypy
  • Fix pypy site-packages dir - Cleanup the recipe and fix site-packages location pypy3.6-feedstock#6
  • Fix ppc64le build and upload - Fix ppc64le build pypy3.6-feedstock#9
  • Add an aarch64 build
  • Send a PR to add a python output to pypy - Add pypy meta package staged-recipes#10828
  • Build the cache files for standard library Build pycache for standard library pypy3.6-feedstock#10, Add a timeout for testing pypy3.6-feedstock#11
• (6) Fix conda-smithy
  • Remove stars and spaces from config file names conda-smithy#1230
  • New release of conda-smithy - conda-smithy v3.6.8 conda-smithy-feedstock#178
  • Fixed variant comparisons when the variant has a space conda-smithy#1242
  • CFEP-09 migrations can't be updated conda-smithy#1243
• (7) Build wheel, setuptools, pip
  • Add pypy build setuptools-feedstock#134
  • Add pypy build certifi-feedstock#53
  • @conda-forge-admin, add noarch: python wheel-feedstock#36
  • @conda-forge-admin, add noarch: python pip-feedstock#43
• (8) Add docs
  • Add PyPy docs #988
  • Add a note about pypy #990

Then, start building packages

[mattip]

Hi. I believe this is a follow up to conda-forge/conda-forge-pinning-feedstock#225 which proposed (as far as I understand it, I may be wrong):

• modify recipe/conda_build_config.yaml to add pypy tags
• fix any breakage for parsing the longer tag
• add required pypy versions somewhere
• rebuild all packages with this tag.

[CJ-Wright]

I think there are some things on the migration/bot side that need to be done (not that this is blocking the other steps except for the rebuild).

• A way to have the bot update the conda_forge_config.yaml
• A way to scope migrations (in case we don't want all the packages to be rebuilt)

Aside from these the migration will look like the arch rebuild.

[mattip]

Once the Python 3.8 rollout activity dies down, I would love to move this issue forward.

[mattip]

@CJ-Wright @ocefpaf ping

[CJ-Wright]

@mariusvniekerk do you think it better to enhance the current cfep9 migration spec in the bot or make a bespoke migrator class?

[CJ-Wright]

@mattip do you have a list of packages that you want to build this out for? You don't need to list everything, we can get the needed deps for the packages you want to rebuild.

[mattip]

numpy, scipy, pandas, matplotlib would be a great start

[mbargull]

Say yes / no to the spec, point out potential pitfalls

Anyone got a link to the mentioned "spec"?

[mattip]

Perhaps the reference is to this meeting's minutes which reference this plan

[CJ-Wright]

Also: dask, streamz, and xonsh. Are there issues with the library search order and noarch support?

[msarahan]

I put together this slightly more fleshed-out plan: https://github.com/msarahan/pypy-python-structure

Compared to the earlier plan, it is much less disruptive in terms of hotfixes. However, renaming the python package is rather dramatic (though completely reasonable in my opinion) and this probably needs a lot of community review.

I hope it's enough to help you push forward.

[mattip]

I commented there. How does this affect user's experience? Would a user ever get into a situation where they have two interpreters in the same conda environment?

[isuruf]

A couple of questions.

1. Can we get rid of _python_impl (which enforces that two interpreters are not in the same environment) and use run_exports to do the same. This might reduce confusion as it results in only 2 packages instead of 3.
2. Do we want to hotfix existing packages to enforce cpython or allow existing packages work with cpyext?

[mattip]

allow existing packages work with cpyext

I am not sure what this means. Pure python packages will work as-is. C-extension packages will need to be rebuilt with the relevant PyPy interpreter each time PyPy releases a new major.minor version, since the c-extension modules built with CPython cannot be loaded into PyPy and visa-versa. In fact. each time PyPy changes its major.minor version, the modules will need rebuilding. Those changes should happen less frequently now that cpyext is beginning to stabilize. Note that PyPy versions have two components: the python side, which is compatible with CPython and reflected in sys.version_info, and the cpyext ABI side, which is reflected in sys.pypy_version_info. PyPy's last release was pypy_verision_info == 7.2.0 and version_info 2.7.13 and 3.6.9 (matching CPython2.7.13 and CPython3.6.9). The next, which should happen by the end of the year, will increment the pypy_version_info to 7.3.0.