Broken packages for all versions <2.8.0

[zklaus]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Due to some surprising behavior when using the jupyter_scheduler package from conda-forge, I noticed that most of its versions contain a lot of vendoring, i.e. the jupyter_scheduler package contains pypi installed versions of popular packages such as SQLAlchemy and Pydantic that, on installation, overwrite the legitimately installed versions from conda-forge. For an example, have a look at the file listing of jupyter_scheduler 2.5.2

That is problematic and means that all currently published packages of jupyter_scheduler <2.8.0 on conda-forge need to be marked as broken.

To prevent this in the future, we should also fix the build script to prevent installation of pypi dependencies in during the installation. I will open a PR for that.

Then, we can re-release older versions to give users flexibility.

@conda-forge/jupyter_scheduler, does that make sense to you?

What are the versions that we should re-release? (I've seen 2.5.2 used in the wild).

Installed packages

-

Environment info

-

[zklaus]

I have now sent the admin request to mark the old builds as broken.

@conda-forge/jupyter_scheduler, please weigh in there as you see fit.

If that PR gets merged, that means only three versions of jupyter_scheduler will be widely available: 2.8.0, 2.9.0, and 2.10.0, though people will always be able to install the old versions by explicit request from the broken label.

Do you reckon that's enough? Do we need to rebuild some more older versions?

[dlqqq]

@zklaus First, thank you so much for reporting this issue and contributing the PR to help fix our feedstock! I appreciate your contribution.

That is problematic and means that all currently published packages of jupyter_scheduler <2.8.0 on conda-forge need to be marked as broken.

I'm not sure I follow. While older releases are "double-installing" their dependencies, older Conda Forge releases should still work. IMO, a poorly-optimized installation sequence doesn't warrant yanking previous releases.

Have you tested to see if they are truly broken? If they are truly broken, then I agree. Otherwise, I think we should reconsider this.

[andrii-i]

@zklaus clarification: we understand that jupyter_scheduler packages <2.8.0 vendor dependencies. At the same time point still stands on if they need to be considered broken

[...] older Conda Forge releases should still work. IMO, a poorly-optimized installation sequence doesn't warrant yanking previous releases.

Have you tested to see if they are truly broken? If they are truly broken, then I agree. Otherwise, I think we should reconsider this.

[zklaus]

@zklaus First, thank you so much for reporting this issue and contributing the PR to help fix our feedstock! I appreciate your contribution.

That is problematic and means that all currently published packages of jupyter_scheduler <2.8.0 on conda-forge need to be marked as broken.

I'm not sure I follow. While older releases are "double-installing" their dependencies, older Conda Forge releases should still work. IMO, a poorly-optimized installation sequence doesn't warrant yanking previous releases.

I'm afraid the problem is really rather more severe. The issue is that those vendoring versions of jupyter_scheduler come with their dependencies embedded in the archive. That means that when jupyter_scheduler is installed, it will overwrite any other version of the dependency that was installed prior.

So say you have a project that depends on any version of jupyter_scheduler and needs sqlalchemy >=2.0.38 because they finally fixed that bug that was holding you back.
Conda determines that that is absolutely fine, as jupyter_scheduler states run requirements of sqlalchemy >=2.0,<3, and proceeds to install sqlalchemy 2.0.38 and then jupyter_scheduler 2.7.1; except in this step sqlalchemy gets overwritten with the version that is embedded in jupyter_scheduler---2.0.30, and the bug that you were sure was gone resurfaces again, leaving you dumbfounded.

This specific example is made up, but that is basically exactly what happened to my colleague Adam Lewis who mentioned this issue in the fix you can find linked higher up in the nebari repo.

That is actually what triggered my interest in the first place.

[andrii-i]

Thank you for detailed analysis @zklaus. I can reproduce this. Curious thing is that while content of sqlalchemy system package folder contains sqlalchemy 2.0.30, pip show sqlalchemy returns version 2.0.38. I think it's because both SQLAlchemy-2.0.30.dist-info and SQLAlchemy-2.0.38.dist-info package metadata folders are still present in system package folder and pip show defaults to showing metadata of the newer version of the package.

On the next steps: all jupyter-scheduler python versions were released within the 17 months and majority of them were released within the last year so we know that some environments are still using older versions of jupyter-scheduler and have not reported any problems with vendoring. So not to break these older environments all versions (for environments with exact version pin) or at least all minor versions (for environments with minor version pin) need to be available. Therefore to not disrupt users problematic versions should be rebuilt rather than marked broken.

I'm not clear on the rebuild process for the older versions, could you please point out the documentation or READMEs for it? Also for the sake of transparency let me mention that while this is needed, we don’t have resources to work on this right now. But we would love to accept contributions 🤗.

[andrii-i]

Reproduction steps:

conda create -n j-sch-t python=3.11 jupyterlab
conda activate j-sch-t
conda install sqlalchemy=2.0.38
pip show sqlalchemy
conda install jupyter_scheduler=2.7.1
pip show sqlalchemy

Note Version: 2.0.38 in the output
Note Location: folder in pip show sqlalchemy output and navigate to it
Note presence of sqlalchemy package folder, SQLAlchemy-2.0.30.dist-info and SQLAlchemy-2.0.38.dist-info package metadata folders
Inspect sqlalchemy/__init__.py:272 , note __version__ = "2.0.30"

It’s also possible to modify jupyterlab or jupyter-scheduler python packages directly and print or inspect sqlalchemy.__version__.

[zklaus]

Regarding the procedure, the instructions are a bit terse, but the long and short of it is that conda-forge will push anything that is built on any branch of a feedstock, so all we need to do is push the recipes for the old versions to any branch in the feedstock.

We could do this even with main, but that risks messing with other updates and version detection of the bot, so I would suggest to do it on a separate branch. For this, it would be helpful if a maintainer could create a new branch (broken-rebuilds?) that can serve as a target for PRs. Then we can create PRs with the old versions and merge them one at a time.

The only important bit then will be to make sure that the builds on the branch complete before merging the next PR.
As individual builds of a package (i.e. conda-forge/noarch::jupyter_scheduler-2.7.1-pyha770c72_0.conda) should be immutable, I think we should increase the build number for any rebuild. That leaves us with the following list of potential rebuilds

• 2.7.1 0
• 2.7.0 0
• 2.6.0 0
• 2.5.2 0
• 2.5.1 0
• 2.5.0 0
• 2.4.0 1
• 2.3.0 0
• 2.2.0 0
• 2.1.0 1
• 1.9.0 0
• 1.8.2 1
• 1.8.1 0
• 1.8.0 0
• 1.7.0 1
• 1.6.0 0
• 1.5.0 0
• 1.4.0 0

That is, the version 2.7.1, build-number 0 should be rebuild as version 2.7.1, build-number 1, etc.

I understand that time is precious, which is probably true for most of us. As the current packages must be removed, why don't we set a time limit (one week?) and start the rebuilds from the top. Then, what's in is in and what isn't isn't. People hard pinning to an older version (are you aware of good reasons to do so?) presumably would not have too hard a time to update to a newer version.

I can start with the most recent versions as a demonstrator and then we see how it goes.

In any case, we need the rebuild branch created by a maintainer.

[andrii-i]

@zklaus Thank you for the detailed reply. I've rebuilt latest patch of each minor version >2, <2.8.0. Non-rebuilt 2.x versions < 2.8.0 and all 1.x versions can be marked as broken.

Versions I've rebuilt:

• 2.7.1 Rebuild 2.7.1 #49
• 2.6.0 Rebuild 2.6.0 #50
• 2.5.2 Rebuild 2.5.2 #51
• 2.4.0 Rebuild 2.4.0 #52
• 2.3.0 Rebuild 2.3.0 #53
• 2.2.0 Rebuild 2.2.0 #54
• 2.1.0 Rebuild 2.1.0 #48

Versions OK to be marked as broken:

• 2.7.0
• 2.5.1
• 2.5.0
• All 1.x versions

Thank you for looking into this. Let's close this issue when it's done.

[zklaus]

Thanks for taking care of this, @andrii-i. Much appreciated!

Before re-activating the admin request PR, I want to clarify one last thing with you.
We don't mark versions as broken, but rather builds.
By this I mean that also some builds from, e.g. 2.7.1 will be marked broken, i.e., taking an excerpt from the list of builds to be marked broken in the admin request PR:

- noarch/jupyter_scheduler-2.7.0-pyha770c72_0.conda
- noarch/jupyter_scheduler-2.7.1-pyha770c72_0.conda

We see both 2.7.0 and 2.7.1 show up, but only build number 0 (as you can see from the _0 just before the .conda extension.
For the 2.7.0 version, that means that no build will be available and hence the version effectively becomes unavailable. But for the 2.7.1 version, the build number 0 vanishes, but now your new build number 1 (noarch/jupyter_scheduler-2.7.1-pyhd8ed1ab_1.conda) is available and thus the version remains accessible.

This is all a long winded way of saying: If we proceed as planned, more things than you listed as "OK to be marked broken" will be marked broken, but I believe the intent of what you want to achieve is matched.

Is that OK?