Make pip gracefully detect git servers that don't support `--filter=blob:none`

[anichari]

Description

This behaviour is observed since pip version 21.3 which has the changes for (#9086)

The change introduces the addition of the --filter=blob:none option for git clone when installing a pip package from a git repo. As a result of the change, the repo package that I'm trying to install errors out with the following signature

(fast2) ➜  ~ pip install zcsdgapi@git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
Collecting zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
  Cloning https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git to /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  Running command git clone --filter=blob:none -q https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

The problem is reproduced on a mac terminal and ubuntu terminal , git versions are
OSX

(fast2) ➜  ~ git --version
git version 2.32.0 (Apple Git-132)
(fast2) ➜  ~ and 

UBUNTU

$ git --version
git version 2.25.1

The complete workflow I performed is attached below. When trying with pip version 21.2.4 the install works correctly, when attempting with 21.3 it fails and the git clone command can be seen with the filter=blob:none flags set.

(fast2) ➜  ~ pip install --upgrade pip==21.2.4
Collecting pip==21.2.4
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.3
    Uninstalling pip-21.2.3:
      Successfully uninstalled pip-21.2.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fast 22.4.15 requires zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git, which is not installed.
Successfully installed pip-21.2.4
(fast2) ➜  ~ pip install zcsdgapi@git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
Collecting zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
  Cloning https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git to /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-nir9f8fj/zcsdgapi_77ada8a7032f41c0a49a2ffb2cf02c19
  Running command git clone -q https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-nir9f8fj/zcsdgapi_77ada8a7032f41c0a49a2ffb2cf02c19
  Resolved https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git to commit 26390c3b0eb26c94a8702af9d2a734faba4bcd8d
Requirement already satisfied: urllib3>=1.15 in ./.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages (from zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git) (1.26.9)
Requirement already satisfied: six>=1.10 in ./.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages (from zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git) (1.16.0)
Requirement already satisfied: certifi in ./.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages (from zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git) (2021.10.8)
Requirement already satisfied: python-dateutil in ./.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages (from zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git) (2.8.2)
Using legacy 'setup.py install' for zcsdgapi, since package 'wheel' is not installed.
Installing collected packages: zcsdgapi
    Running setup.py install for zcsdgapi ... done
Successfully installed zcsdgapi-6.1.0
WARNING: You are using pip version 21.2.4; however, version 22.0.4 is available.
You should consider upgrading via the '/Users/achari/.pyenv/versions/3.10.0/envs/fast2/bin/python3.10 -m pip install --upgrade pip' command.
(fast2) ➜  ~ pip uninstall zcsdgapi
Found existing installation: zcsdgapi 6.1.0
Uninstalling zcsdgapi-6.1.0:
  Would remove:
    /Users/achari/.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages/zcsdgapi-6.1.0-py3.10.egg-info
    /Users/achari/.pyenv/versions/3.10.0/envs/fast2/lib/python3.10/site-packages/zcsdgapi/*
Proceed (Y/n)? Y
  Successfully uninstalled zcsdgapi-6.1.0
(fast2) ➜  ~ pip install --upgrade pip==21.3
Collecting pip==21.3
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fast 22.4.15 requires zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git, which is not installed.
Successfully installed pip-21.3
(fast2) ➜  ~ pip install zcsdgapi@git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
Collecting zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
  Cloning https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git to /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  Running command git clone --filter=blob:none -q https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

Expected behavior

pip install package from a git repo is expected to work as it did before pip version 21.3 and install the package successfully.

pip version

21.3

Python version

3.8, 3.9 and 3.10.0

OS

Mac OS and Ubuntu, docker python3.8:latest

How to Reproduce

1. Install python3.8 and higher and pip version 21.3 and higher
2. Install a package from a git repo (dont know if there are server side settings required to get this reproduced as we dont have access to the server settings we are installing from)
3. the fatal error seen as below

The complete workflow I performed is attached below. When trying with pip version 21.2.4 the install works correctly, when attempting with 21.3 it fails and the git clone command can be seen with the filter=blob:none flags set.

Output

(fast2) ➜  ~ pip install --upgrade pip==21.3
Collecting pip==21.3
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fast 22.4.15 requires zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git, which is not installed.
Successfully installed pip-21.3
(fast2) ➜  ~ pip install zcsdgapi@git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
Collecting zcsdgapi@ git+https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git
  Cloning https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git to /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  Running command git clone --filter=blob:none -q https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git /private/var/folders/z5/f81ktg8d5ng_pskb9wg2q4qr0000gn/T/pip-install-_jsv_zuz/zcsdgapi_e5a7fb3aae9d4eef9d8d91c42547d486
  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

  fatal: bad revision '43995bd42fa23bbc4c3a59d5fb9da99042fcf139'
  error: https://bitbucket.corp.zscaler.com/scm/eio/zcsdgapi.git did not send all necessary objects

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

@nipunn1313

[nipunn1313]

From some quick googling, bitbucket should be supporting this feature of git. It does look like you have a hosted bitbucket instance, which should default to supporting this unless explicitly disabled.

https://jira.atlassian.com/browse/BSERV-11639

That being said, perhaps there’s a way to make pip backwards compatible here

[anichari]

Thanks! I don't have much info/control on the bitbucket server instance. If there is a way to make pip compatible, that would be great!

…

On Tue, Apr 19, 2022 at 9:01 AM Nipunn Koorapati ***@***.***> wrote: From some quick googling, bitbucket should be supporting this feature of git. It does look like you have a hosted bitbucket instance, which should default to supporting this unless explicitly disabled. https://jira.atlassian.com/browse/BSERV-11639 That being said, perhaps there’s a way to make pip backwards compatible here — Reply to this email directly, view it on GitHub <#11043 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABILQ7ZIMLIG2BDYSOCN5YTVF3KEDANCNFSM5TWRSE3Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Thanks and Regards Ani Chari

[uranusjr]

If error: [url] did not send all necessary objects is an error we can rely on, we could detect that and re-do the clone without --filter for compatibility. But first someone probably needs to get a guarantee this can be relied on (or another way for the client to detect lack of support from the server) from the Git team.

[KalleDK]

I ran into the same problem, and would really like for the combability, as I have no say in the server's version / settings. The solution right now (for me) is downgrade git to get the other branch. Autodetect would be nice, but for my cases a simple switch would be more than enough.

[pradyunsg]

Since the folks affected by this are likely institutional users with an internal source control deployment, I'll note that you're welcome to file PRs against pip to improve things for this usecase. If there's some way for pip to gracefully detect this, based on whether the server supports this feature, we'd be happy to accept a PR for it -- assuming we don't degrade the experience for users who have a new-enough git and are using a new-enough server instance (eg: github.com).