Discuss installable Python package support for Python 3.8 `plugin_host` and beyond

[kylebebak]

Hey @wbond @deathaxe @BenjaminSchaaf @FichteFoll

I'm creating this issue to discuss PC 4.0, specifically adding support for installing Python packages that plugins can depend on, and allowing the 3.8 plugin_host to be upgraded at some point.

I know very little about the PC source code, but I've read about design decisions and constraints, and how PC has evolved. From what I can tell:

• We'll probably always want to run a Python 3.3 plugin host
  • There are too many plugins that would just break if this host is removed
• There aren't a lot of plugins yet that depend on the 3.8 host, and there probably aren't many 3.8 plugins that depend on compiled code
  • If this is the case, replacing the 3.8 host with an e.g. 3.11 host wouldn't break too many plugins, but it would very likely break some
• One of the flagship features for PC 4 is support for installing Python packages, so plugins can depend on these without plugin devs having to vendor them, or having to rely on/create "dependencies" with vendored Python packages

Here are some thoughts on installable packages:

• They're a great feature!
  • Plugins with package deps would be much easier to write and maintain
  • Vendoring packages works, but it's a pain
    • It's what most plugin devs do, e.g. it's what I did with Requester
  • For packages that use shared objects, it makes the plugin dev responsible for vendoring these for every platform that ST supports
  • This is even more of a pain if the plugin host version changes
    • Package devs have to revendor compiled code for the new Python version
• So we should definitely add installable packages
• But we shouldn't support this for the 3.3 plugin host, because it would be a mess and place painful constraints on the design
  • That four-point-oh is 540 commits ahead of master suggests there are already painful constraints on the design

On upgrading the plugin host:

• The 3.8 plugin host will have to be upgraded at some point, because Python 3.8 will become obsolete just like 3.3 did
• At the same time, for the long-term success of PC it's probably best to not support a third plugin host
  • Not good to e.g. have 3.3, 3.8 and 3.11 hosts run in parallel
  • If we do this eventually we'll have 4 plugin hosts...
• We could instead replace 3.8 with 3.11 or 3.12 or whatever

On how to build installable packages in PC:

• Installing just wheels is a great idea
• But building our own pip to do this is not
• pip is solid, sophisticated, extremely well-tested, etc
  • We should just use pip, e.g. pip install --only-binary
• Just like ST ships a bundled Python for the plugin host, it should ship a bundled pip for the same Python version as the plugin host
• Plugins can specify package deps in something standard, like requirements.txt
• pip can install them and PC can put them on the plugin host's Python path, so they're importable by plugin code
• In the future, when we replace the 3.8 plugin host with e.g. 3.11, PC can remove site-packages or wherever the old pip installed its packages, then reinstall them with the new pip
  • With --only-binary, and a plugin host that doesn't use a bleeding edge Python, this is unlikely to break packages, even those that depend on compiled code
  • This "dynamic" plugin host can be upgraded every few years to a new Python version that's been in production for at least a year
  • E.g. later this year would be a reasonable time to upgrade the host to 3.11, but not to 3.12

On backwards compatibility with 3.3:

• Relying on pip means packages aren't supported for the 3.3 plugin host, but that's fine
• It's actually desirable to have a break between the 3.3 plugin host, which will have to stick around forever, and the "dynamic" plugin host (3.8 now, maybe 3.11 next, etc)
• Otherwise constraints in Python 3.3 (released 11 years ago!) prevent innovation on the 3.8 plugin host, and all future plugin host versions, and hence on all future ST plugins

In summary, I think installable packages and a "dynamic" plugin host dovetail nicely, and are key to the future of ST plugin development. New plugin devs don't want to write code against obsolete Python versions, and vendor & patch obsolete Python packages.

Either one of these features (installable packages in PC, new plugin host) could be shipped on its own. If we plan on doing both of them, we'd first need to ship a PC release with existing (legacy) code for 3.3 plugins, and code for dynamic 3.X plugins with installable packages.

Changing the plugin host version every few years is bound to break some plugins, but installable packages make this less likely and less painful. The alternatives, running more and more plugin hosts or freezing the second plugin host at 3.8, are worse.

four-point-oh notwithstanding, PC hasn't changed much in the last 3 years. If it stops getting better people will eventually stop writing plugins. I love ST, I've used it and pretty much only it for my entire career, but I would have switched to something else if it weren't for LSP support, and that's a plugin.

[wbond]

One of the most popular packages (SFTP, written by myself, and the impetus for Package Control in the fist place), used compiled 3.8.

Using pip itself is unlikely to be too much of a win, since you will need to:

1. Maintain a fork as they leave old versions behind
2. Handle constraint solving yourself anyway since you’ll have N packages that will request specific versions

There is existing code to do a lot of the actual package mechanics, and we already have all of the downloading code running for a long time. Trying to have package downloads using one code path and pip libraries use a completely different downloading mechanism isn’t a recipe for success IMO. There is new code to handle most of the package operations and it is probably 80% done. Finishing that should take less work than learning all about pip internals and trying to get it all to work.

I think aside from the many unique technical constraints of the ST Python environment, the big issue is that this is no longer a passion project for me, and there hasn’t been much work except by @deathaxe.

This is probably largely my fault, but I don’t see the way to really get out of this situation unless Sublime HQ decides to take over the project and decide how to do such things. They currently bankroll the vast majority of money for hosting, etc through a GitHub sponsorship. But really the package work could likely use someone who is working at least part time on it and is willing to get deep into the headspace of cross-platform, multi-version Python package management.

In terms of Python versions, that is 100% up to Sublime HQ. I personally wouldn’t ditch Python 3.8 any time soon, but I no longer work in such a capacity, so I don’t get to make such decisions.

[deathaxe]

I've played around with running pip directly from within ST's python 3.8 plugin host a while ago by copying pip folder from a system's python 3.8 instance and running CLI via ST's console as follows:

from pip._internal.cli.main import main
main(["install", "--only-binary", "-t", R"<st>\Data\Lib\python38", "pyyaml"])

If it works, it seems to work pretty well.

The downsides are:

1. it is very likely to crash ST's plugin_host!
2. pip is about 6.5MB in size (PC4.0 only used 0.5MB!).
3. as Will said, it would use a totally different download mechanism, which might provide its own caveats or challanges to setup.
4. it is an unmaintainable path without deeper knowledge about pip
5. using current pip would break PC compatibility with ST3143-3211.

PEP440 version support in https://github.com/wbond/package_control/tree/wip/pep440 is nearly complete - except some minor edge cases with regards to local versions. But I don't think we need to care about them.

Parsing and resolving requirements is somewhat open. It doesn't look too dificult, but well, I am already spending a lot of spare time for ST stuff and it's not my main domain.

I'd already be happy to get the current PC4.0 released and packagecontrol.io updated to support new 4.0.0 scheme version, so we could even step forward to using python 3.8 packages at all.

[kylebebak]

Thanks for responding @wbond =)

One of the most popular packages (SFTP, written by myself, and the impetus for Package Control in the fist place), used compiled 3.8.

If the compiled 3.8 code for SFTP is in a wheel somewhere, we could upgrade the plugin host to a newer Python, the installer could reinstall the wheel, and the plugin could keep working. If that compiled 3.8 code is vendored then you'd have to update that plugin code manually or PC is stuck running the 3.8 plugin host forever

Maintain a fork as they leave old versions behind

If we only support installable packages for a single "dynamic" plugin host (3.8 now, 3.X in the future), we can upgrade this plugin host frequently enough to always use a pip that's maintained. Python 3.8 is already old, but pip still works great for this version

Handle constraint solving yourself anyway since you’ll have N packages that will request specific versions

PC could collect all the plugin deps, put them in a single requirements.txt file, and run pip install -r requirements.txt on that. pip does constraint solving. E.g. the following, which I ran in a Python 3.8.13 venv using pip 22.0.4

requests==2.15.1
requests==2.16.1

pip install -r requirements.txt
Collecting requests==2.15.1
  Downloading requests-2.15.1-py2.py3-none-any.whl (558 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 558.7/558.7 KB 2.8 MB/s eta 0:00:00
ERROR: Cannot install requests==2.15.1 and requests==2.16.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested requests==2.15.1
    The user requested requests==2.16.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

If pip can't solve constraints, we can surface this error and tell the user they can't install package X with package Y because of Z. If we want to, it would be easy to build an opt-in override scheme for when plugins have conflicting package deps. That's the best a resolver can do

---

Thank you also @deathaxe for responding, and for working on PC

it is very likely to crash ST's plugin_host!
pip is about 6.5MB in size (PC4.0 only used 0.5MB!).
as Will said, it would use a totally different download mechanism, which might provide its own caveats or challanges to setup.
it is an unmaintainable path without deeper knowledge about pip
using current pip would break PC compatibility with ST3143-3211.

1, sounds gnarly, I'm guessing there's a fix
2, I'm guessing Sublime users aren't worried
3, true, but the download mechanism is understood by a lot more people than the current mechanism
4, I'm guessing pip, which has a huge community and is actively maintained, proves to be more maintainable, not less
5, we shouldn't support installable packages for ST3 / Python 3.3. Plugin devs have gotten on without them forever, and this would be another reason to upgrade from ST3 -> ST4

And again, if we use pip, requirements parsing and resolution are pretty much solved

That said, if you think there's a good way to do installable packages for 3.8 and future plugin host versions without depending on pip I (and I'm guessing everyone else) would be in favor of that too

---

Will, I agree with you that ideally Sublime HQ takes responsibility for the future of Package Control. IMO you and PC are a huge reason Sublime is successful, and if PC isn't maintained Sublime loses a big part of its value

[deathaxe]

If the compiled 3.8 code for SFTP is in a wheel somewhere,

To clarify: Don't mix up Sublime Text packages and Python packages! Those are totally different things which don't share anything!

SFTP is a Sublime Text package and will therefore never be published as WHEEL file. As all the other Sublime Text packages won't.

pip can't be used to handle Sublime Text packages, as it is not designed to understand their structure.

Even if it was theoretically possible to use the WHEEL format to publish packages/plugins for Sublime Text, it would significantly degrate developer experience as pip's package format is a terrible mess and creating wheels involves a build process, which is currently not required.

This also answers: "3, true, but the download mechanism is understood by a lot more people than the current mechanism".

The current download mechanism will not go away as it is crucial for handling Sublime Text packages. It may be replaced by something else in future, but not by pip.

To conclude: Current mechanisms to install/upgrade/manage Sublime Text packages is not going to change - just may evolve.

It's working fine for ST3 and 4 without any reason to drop support for any older build or ST3.

---

The whole discussion about pip or using parts of it, is just about support for what was called "dependencies" in ST3/PC3 world.

Instead of managing own sets of python packages in a ST compatible format, existing WHEEL files are to be supported to

1. lower burden for package authors to use and maintain external libraries/dependencies.
2. (theoretically) allow users to use pip directly via CLI to install libraries into ST. That's however a rather hopeless idea as it required python 3.8 to be installed on OS.

The infrastructure (Libs/python... folder(s)) to install python libraries is supported as of ST3143, which is therefore the least required ST build for PC4.0

Supporting ST3 just means to install existing dependencies in the new format, which already works.

It's just to keep Package Control compatible with ST3, no py38 libs and syntax can be used.

---

we shouldn't support installable packages for ST3 / Python 3.3. Plugin devs have gotten on without them forever, and this would be another reason to upgrade from ST3 -> ST4

unacceptable and useless, because

1. leaving ST3 users behind with PC3.4 failing on OpenSSL3 is unacceptable.
2. PC4.x changes the way how dependencies are installed and so it must take care of handling it in backward compatible ways. Breaking any python 3.3 package on ST4 is unacceptable as many of them still work fine and majority very likely won't ever get updated to py3.8 as being unmaintained. I also doubt we'll find enough volunteers to fork and convert them.
3. handling multiple plugin_hosts on ST4 is not the deal breaker. The effort is just to set it up in a ST3 compatible way (see 1.), which is not undoable, but just involves learning some python package management basics.

---

4, I'm guessing pip, which has a huge community and is actively maintained, proves to be more maintainable,

It's always a maintenance burden to use foreign projects due to requirement to keep up with their development cycles. They decide to introduce breaking changes, good luck. Something stops working for you? Good luck. Many unexpected things can happen, which need to be debugged and cost time as well. So careful decisions need to be made about what deps to use.

If it is so easy to maintain, just tell my why it's crashing ST's plugin_host. Is it so easy to find out?

[kylebebak]

Hey @deathaxe

To clarify: Don't mix up Sublime Text packages and Python packages! Those are totally different things which don't share anything!

pip can't be used to handle Sublime Text packages, as it is not designed to understand their structure.

Yeah, I know... I think there was a misunderstanding. Here's what I'm saying:

• We enable using pip to install Python package dependencies from wheels, not to install ST packages/plugins
  • This way ST plugins can depend on Python packages without having to vendor them
  • pip is good at installing Python packages, and is better for that purpose than anything the ST community can cook up
• There's no reason to support this new feature (using pip to install Python package deps from wheels) for ST plugins written against the 3.3 plugin host
  • It's an opt-in feature, for 3.8 plugins only
  • It doesn't affect how Package Control dependencies work
  • This doesn't break 3.3 plugins (or 3.8 plugins)
  • No one has to update existing plugins

Maybe it was confusing that I called ST packages "plugins" instead of packages. I was trying to make things less confusing by doing this, because "package" is the term used for Python packages, and because Sublime Text docs tend to mix these terms.

[twwildey]

Installing PIP modules and all of their dependencies needed for a Sublime Text plugin is consistently the highest friction part for developing plugins. Sublime Text has been my favorite editor for over a decade, but the difficulty of developing plugins for it will be its demise. I've easily spent 60-80% of my time developing plugins manually installing/resolving PIP modules that I could've installed with PIP idiomatically in any other situation where I was developing Python code.

I would vastly prefer to install Python dependencies through PIP when developing plugins with Package Control. There is significant overhead for maintaining an additional Git repo for each PIP module you want to install as a dependency through Package Control. Authors will stop maintaining these packages, after which plugin developers will end up abandoning their plugins as well.

It should be possible for existing dependency management mechanisms to continue working as-is in Package Control, while supporting an additional mechanism for installing Python dependencies with PIP:

1. Package Control should provide versions of PIP for every version of Python supported by Sublime Text's plugin hosts.
2. A schema extension to dependencies.json or a new file/schema altogether can define the PIP modules to be installed for a package installed by Package Control. Let's call this new file/schema pip-dependencies.txt for the sake of argument:
   1. pip-dependencies.txt follows the exact format/structure of requirements.txt that would be used in a pip install -r command.
   2. pip-dependencies.txt can be defined in the root of the package or under any sub-folder following the {stv}_{py}_{os}_{arch} convention, enabling PIP module requirements to lock to specific versions for each {stv}_{py}_{os}_{arch} combination.
3. Package Control can install these PIP modules to <st>\Data\Lib\{py} using PIP directly.

There is a concern about different Sublime Text plugins/packages having different/conflicting versions of PIP modules being installed, but this is already not solved for robustly. Plugins relying strictly on the Package Control Channel Repository will have a consistency guarantee; however, I'm certain there plugins that have to operate using PIP modules outside of this. I am one of them.

[deathaxe]

To clarify:

Normal python packages can be installed into $data/Lib/python33 or $data/Lib/python38 for private use via pip since ST3143 by specifying them as custom site-packages via pip's CLI. That's supported by ST regardless Package Control being present. You just need to somehow ensure to install packages for correct interpreter version. PC3 just wasn't the tool to deploy such setups globally.

Package Control 4 is designed to mimic pip behavior.

It...

1. converts legacy dependencies to wheels before installing them to those new $data/Lib/python3x folders.
2. directly supports installing wheels (python packages in *.whl format) via asset based releases.
3. can directly install wheels from pypi.org

There is no need to maintain dedicated git repos for libraries anymore, which are already deployed via pypi.org!

In fact most existing dependencies have already been migrated to be installed from pypi.org for at least python 3.8 environment.

---

While the legacy dependency format (see: https://github.com/packagecontrol/example-dependency) has been extended to support sub directories such as {stv}_{py}_{os}_{arch} to satisfy those who want to continue this way of deployment, the primary goal is to encourage library developers to ship ordinary wheels. This strategy however requires package authors to learn how to use python build to create wheels and how to provide (platform-specific) release assets via Github/Gitlab releases.

The only requirement for python packages to be available via Package Control currently is them being registered to https://github.com/packagecontrol/channel.

---

The only missing feature, currently is recursively resolving requirements.

There are various reasons why pip can't be used to manage libraries, directly:

1. Pip primarily resolves dependencies via pypi.org directly. That's not possible, if other release sources are to be supported (such as Github releases). It may work, but hasn't been investigated in detail so far, what it means within ST environment.
2. Pip crashes ST's plugin_host
3. Pip is designed to install packages without their using script is not running, while Package Control needs to be able to update them during normal execution.
4. Pip can't handle legacy dependencies, so a full 2nd code branch was required.
5. Pip ships a lot of content, not required in ST environment.

---

It was probably possible to levarage some basic packages, also used by pip, but...

All official python packages such as packaging, resolvelib, ... require python 3.8 these days. Using them would break compatibility with python 3.3 plugins and ST3. It would even break all python 3.3 plugins running on ST4, which import from Package Control. Even just those using the "events" module would be affected.

What's missing, technically is completing the functionality to interpret python requirements and building the dependency tree from it, to generate a list of packages to install.

Currently all required sub-requirements need to be specified manually by a package author, which is also state of the art in pyscript for instance.

In order to use pydantic, a package author would need to specify a dependencies.json with

{
    "*": {
        "*": [
            "annotated-types",
            "eval-type-backport",
            "pydantic",
            "pydantic-core",
            "typing-extensions"
        ]
    }
}

That's probably not ideal, but is also not impossible to manage.

[twwildey]

@deathaxe Appreciate the thoughtful and detailed message here. It seems like improvement has been made on this front, and the need to unfold transitive PIP dependencies into a dependencies.json file is not that bad. However, the need to represent all PIP modules that can be installed in the standard Package Control channel is still less than desirable.

Given your feedback, I would update my proposal to create another Sublime Text plugin - registered in the standard channel for Package Control - called sublime-text-plugin-pip (naming flexible still) that would:

1. Provide an API for Sublime Text plugins to register callbacks that will be invoked when their PIP modules have been installed.
   1. Sublime Text plugins using this registration API can delay import statements for PIP modules that are not guaranteed to be available until their PIP module dependencies have been installed.
   2. The callback methods registered with this API would be responsible for initializing the plugin with Sublime Text. Plugin authors will need to take care that the plugin_loaded and plugin_unloaded methods for Sublime Text may be invoked before their PIP modules have been installed.
2. Install an identical version of Python to one used in the plugin_host-{py} executable.
   1. The version of Python used by plugin_host-{py} can be resolved with sys.version executing within the plugin code.
3. Resolve/download a version of PIP that was directly compatible with the version of Python used by the plugin_host-{py} executable.
4. Scan all Sublime Text plugins for the pip-requirements.txt file discussed above (naming convention still flexible here) to produce a set of PIP module versions to install to <st>\Data\Lib\{py}.
   1. Some mechanism in this plugin can coalesce all PIP modules across all pip-requirements.txt files and resolve duplicates.
      1. Improvements to this plugin can spot incompatible PIP modules due to versioning requirements, which can be logged or explicitly notified to the user.
         1. Given the effectively constant version of Python used by Sublime Text, most plugin authors will identify PIP module versions that work with the version of Python used by Sublime Text. This will significantly reduce the likelihood of version incompatibility issues for PIP module dependencies, since duplicative PIP modules with different versions will still work with the version of Python included with Sublime Text fundamentally.
         2. How users are notified can be something they configure, since many version incompatibilities for PIP module dependency requirements often are moot. Nonetheless, users should have some indication that things might break for specific plugins due to dependency mismatch issues.
5. Execute the version of PIP resolved/downloaded by the plugin with the Python binary that it also installed (i.e. not plugin_host-{py}) to install the set of PIP module versions identified across all pip-requirements.txt files.
   1. This obviates issues with the plugin_host-{py} executable crashing when executing PIP install commands.
   2. This isolates PIP dependencies and content from the Python environment executing plugins for Sublime Text.
   3. Native PIP modules can be built correctly and used by plugins for Sublime Text.
   4. Installing PIP modules executes in a separate process and does not block the plugin_host-{py} process from executing independently.
6. Invoke the registered callbacks for all Sublime Text plugins that invoked the callback registration API discussed earlier in this list.
   1. All Sublime Text plugins expecting their PIP modules to be installed through sublime-text-plugin-pip will have programmatic guarantees that their PIP modules are available at the time they would import them in their code.

I understand that the LSP plugin has functionality which does performs some of this today, which could likely be re-used for this purpose:

1. https://github.com/sublimelsp/lsp_utils/blob/9de78d5bb86e394c5ace8aa3126f55b26dab820c/st3/lsp_utils/server_pip_resource.py
2. https://github.com/sublimelsp/lsp_utils/blob/9de78d5bb86e394c5ace8aa3126f55b26dab820c/st3/lsp_utils/pip_client_handler.py

I would propose that this plugin only support Python 3.8 starting out, as I suspect effectively all plugin authors will only use 3.8 at this point. Nonetheless, the plugin should be written to automatically support future versions of Python that Sublime Text will adopt.

I would be interested in your feedback to this proposal. Ultimately, this could be delivered without official "blessing" by Package Control, but I would prefer to learn from everyone's experience here. You've all spent far more time supporting plugin development and developers than I have.

[deathaxe]

The requrirement to install a dedicated python and pip next to ST just to be able to install dependencies on all different OSs is not desirable. ST2 relied on Macs python26 and does therefore no longer work on modern macs as python2 has been removed. Same is true on Linux for python33. None of the current versions supports it (by shipping packages). Same will happen soon for python 3.8.

Actually Package Control even shifted to UnitTesting package to handle CI tests, just to avoid external python dependencies which caused tests to start failing due to python 3.3 not being supported anymore.

PIP is a dead end.

Package Control ships everything required to install python packages.

What's missing is using pypi's index directly to look out for packages and some logic to recursively resolve dependencies.

[toddwildey]

Is my understanding correct that a Sublime Text plugin could provide its own repository JSON that referenced PIP modules under its libraries in order to install *.whl files from PyPI?

[deathaxe]

Theoretically yes. Practically not with official packagecontrol.io channal though, as it does not yet support required scheme v4.0.0. Such a repository couldn't be registered in default channel. That's actually the reason why we currently ship a separate channel_v4.json for libraries. That channel's infrastructure is however not designed to take packages.

[twwildey]

@deathaxe Would I be able to add a package to the channel_v4.json?

It seems that we should not be waiting on plugins to adopt the new mechanism. I am willing to use this mechanism instead of PIP, but I need to be able to use it now.

[deathaxe]

Adding python packages aka. libraries there is ok, but no packages. The crawler is not capable of handling github api rate limits and will break the whole channel json when that happens. I haven't found the time to make it work like the original one, which is backed by a database to only crawl a subset of registered packages at a time.

Furthermore those (normal) packages wouldn't be visible on packagecontrol.io or anywhere else, but in the quick panel.

[toddwildey]

What is the crawler being referred to here? Is it Sublime Text on a user's machine? Any code/issue links would be appreciated.

Are there existing initiatives to migrate the Package Control Channel/Repository schemas to v4?

I understand that @wbond is MIA apparently, but the community should not be structed to hold up the entire show on one person.