Temporarily restore --build-dir for PyCharm

[sfelixkim]

Environment

• pip version: 20.3
• Python version: 3.8
• OS: Windows 10

Description
Cannot install any packages when using PyCharm python interpreter.
The virtual environment is created using virtualenv.
No problem when executing pip install [package] on either PyCharm terminal or cmd prompt directly.
Problem resolved when pip is downgraded to 20.2 (had to be done on terminal by pip install pip==20.2.4)

Expected behavior
Install packages successfully.

How to Reproduce

1. create a new project on PyCharm (then the installed pip version is 20.3)
2. install any package (e.g. flask) by typing the name and click the "Install Package" button
3. An error occurs.

Output
executed command:

pip install Flask

Error occurred:

Non-zero exit code (2)

proposed solution:

Try to run this command from the system terminal. Make sure that you use the correct version of 'pip' installed for your Python interpreter located at 'C:\path\to\project\folder\venv\Scripts\python.exe'.

command output:

Usage:   
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] <requirement specifier> [package-index-options] ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] -r <requirements file> [package-index-options] ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] [-e] <vcs project url> ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] [-e] <local project path> ...
  C:\path\to\project\folder\venv\Scripts\python.exe -m pip install [options] <archive url/path> ...

no such option: --build-dir

[uranusjr]

See: https://mail.python.org/archives/list/distutils-sig@python.org/message/6Z5DQGRDGOU4OCO3JSS4FVYWV5JBUPVP/

@pradyunsg Would it be a good idea to restore the option before PyCharm can release a fix on their end?

[gaborbernat]

Do we know why PyCharm developers did not see the warning about that option being deprecated? Maybe we should improve on that.

[jerryvurbl]

so this is basically a pycharm issue? Is 6 months warning enough? Probably. Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably. Where do we go from here? We have to open a PyCharm issue?

[gaborbernat]

Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably.

All major IDE developers have access to the pip changelog. The pip developer team has no way to know what deprecated interfaces these IDEs use (against emitting warnings for 6+ months), especially sometimes when those sources are behind closed doors. Yes, you should be opening a PyCharm issue. The issue lies with them at the end of the day. Should they perhaps have a test suite against the master branch/RC versions of pip? Probably.

[sbidoul]

Would it help if the option was still present as a no-op ?

[gaborbernat]

The PyCharm issue https://youtrack.jetbrains.com/issue/PY-45712. Seems there was a commit to remove the deprecated flag yesterday on the PyCharm code base JetBrains/intellij-community@a618daa

[pfmoore]

@sbidoul I was going to say yes, leaving the option as a no-op for a short while might be helpful. But if JetBrains have already committed the change, it sounds like they are in a position to release a fix, so I think we can leave it to them.

[pradyunsg]

Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem?

Ok, this annoys me more than I should probably allow it to.

pip's maintainers have communicated about this change across multiple public channels, (as noted earlier) this option has been printing warnings on every invocation for the past few releases and we even had a beta release for 20.3 out for nearly 1 month, where this option was removed. I'm genuinely not sure what more we could've done to avoid this situation from our end. From my perspective, no reports/requests came in related to this deprecation from end users, so this was supposed to be "the easy one" -- I'd even dropped it from our 20.3 release announcement publicity push, because I thought: well, this removal seemed to affect no one so why call it out in the emails (LOL). To imply that pip's devs didn't do enough feels exceedingly dismissive of our efforts, and is the kind of thing that results in folks walking away from projects.

---

sigh Sorry, I got worked up there.

---

@pradyunsg Would it be a good idea to restore the option before PyCharm can release a fix on their end?

Yes.

It seems like PyCharm has been able to make the changes on their end to accomodate for the removal, but I'd imagine it might still be necessary to reduce breakage as they roll out the fix. If someone from the PyCharm team could clarify/provide an answer to the question:

Would it help if the option was still present as a no-op ?

I don't want to reintroduce the --build-dir functionality to be very honest -- that removal was blocking internal cleanups I'd really like to get to -- but maintaining a no-op option for a few releases is very "cheap" in terms of maintenance effort. Looking at the removal patch, it seems like this would be OK?

I'm happy to cut a pip 20.3.1 later today (assuming someone can file a PR adding this, with a comment pointing back to this issue), if that would help with this breakage for PyCharm's users.

[pfmoore]

To imply that pip's devs didn't do enough feels exceedingly dismissive of our efforts, and is the kind of thing that results in folks walking away from projects.

Agreed.

Can I also note that the comments I saw from the PyCharm developers were polite, reasonable, and not at all confrontational, so I'd like to publicly thank them for taking a completely reasonable approach here.

Drive-by criticism of either project over this from 3rd-party bystanders is neither helpful nor welcome, in my view.

[jerryvurbl]

Fair enough, will do. That said maybe this should have been a semver major release change on the pip side? it should be noted that a user has to adopt this pip change to break things and it can be rolled back in the same way. maybe its me but i dont read all the release notes for the hundreds of projects i use and track but a major release increment usually gets for attention. a breaking change appearing between x.2.4 to x.3.0 seems to violate this seemingly basic heads up but i didnt review this issue close enough to know that my assertion here is correct. if it is i dont know what semver is for anymore. apologies if this seems strident i dont mean it to be. i had some intensive rollbacks i had to do where i was used to trusting these upgrades without doubt. anyway, its mostly a pycharm issue i agree but a breaking change is a breaking change. my limited perspective says that the built in warning systems did not work.

…

On Dec 1, 2020, 2:14 AM -0800, Bernát Gábor ***@***.***>, wrote: > Should this critical breaking change be coordinated between the biggest IDE's developers and the developers of the most important executable in the python ecosystem? Probably. All major IDE developers have access to the pip changelog. The pip developer team has no way to know what deprecated interfaces these IDEs use (against emitting warnings for 6+ months), especially sometimes when those sources are behind closed doors. Yes, you should be opening a PyCharm issue. The issue lies with them at the end of the day. Should they perhaps have a test suite against the master branch of pip? Probably. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[gaborbernat]

That said maybe this should have been a semver major release change on the pip side?

If you take a look at pip version number (2020.4) you'll notice that's a fairly big number. It's because pip does not use semantic versioning but instead https://calver.org/overview.html. As such there's no concept of major release you might be familiar with semver.

it should be noted that a user has to adopt this pip change to break things and it can be rolled back in the same way.

The warning message printed on every invocation for the last few months (at least) should suffice as a notification of you're doing something wrong. Please familiarize yourself with it https://calver.org/overview.html#when-to-use-calver 👍

[pradyunsg]

That said maybe this should have been a semver major release change on the pip side?

If we followed Semver -- sure, it would've been -- but we don't, for a lot of reasons that I won't go into. I'll flag that pip has been one of the most prominent projects using CalVar for over 2 years now.

We also have a documented deprecation policy about making breaking changes: https://pip.pypa.io/en/stable/development/release-process/#deprecation-policy

my limited perspective says that the built in warning systems did not work.

This is definitely true. I do want us to identify why this happened, so that we can avoid this class of issues in the future.

---

I say that we let this thread be "at rest" now, until tomorrow or until one of the PyCharm developers responds here. :)

[east825]

Hi everyone! I'm from a PyCharm team. We didn't notice the deprecation because this option was used only in an internal script that we normally don't launch manually. We can definitely improve in this regard by reporting any deprecation messages from the tool to stderr, at least in the internal/EAP mode as an early diagnostic, so that it doesn't happen again.

As for why we needed --build-dir in the first place, it seems that now it's more of a historical artifact back from the days when pip didn't build binary packages in TMP by default, so we had to emulate this behavior (creating a temporary directory, building in it and then removing) manually. Unfortunately, the history is somewhat scarce, and the option was added in 2013. Can someone shed the light on what was the former behavior, so that we were entirely sure that by removing the option we don't break some corner cases? In the distutils-sig thread that I started Tzu-ping Chung has already mentioned that this behavior changed around pip 6.x. Is it accurate? Would you mind pointing me to the corresponding changelog items?

Making --build-dir a no-op for a little while, until we prepare updates to already released versions of PyCharm, especially if we really don't need it any longer, sounds just perfect.