Variable expansion doesnt work as "pipenv requirements"

[RintaroMiyamoto]

Issue description

I am using private repo to install packages and it is refered in Pipfile.lock as like below

"sources": [
  "url":"https://$USER:$ACCESS_TOKEN@gitlab.com/private/repo"
]

Here, pipenv requirements > requirements.txt returns file using environment variables in unsupported format for pip install -r requirements.txt

Expected result(MODIFIED IN COMMENT)

# requirements.txt
-i https://expanded_username:expanded_token@gitlab.com/private/repo   
random_lib == 0.0.0  
...

pipenv lock --requirements > requirements.txt returned expected result(this command no longer available)

Actual result

# requirements.txt
-i https://$USER:$ACCESS_TOKEN@gitlab.com/private/repo   
random_lib == 0.0.0  
...

Only POSIX format is supported for pip install -r.
https://pip.pypa.io/en/stable/reference/requirements-file-format/#using-environment-variables

[dqkqd]

The requirements file generated by the old command is definitely a bug, environment variables should not be expanded.
But according to pipenv doc, pipenv supports for both $VARIABLE and %VARIABLE%, while pip doesn't. That causes the issue. In order to work with pip, ${USER} should be used instead of $USER.

[RintaroMiyamoto]

@dqkqd
Thanks for the comment! I totally agree with you.
So the expected result would be:

Expected result

# requirements.txt
-i https://${USER}:${ACCESS_TOKEN}@gitlab.com/private/repo   
random_lib == 0.0.0  
...

[EugeniuZ]

What is the preferred way to generate a requirements.txt file with private repo (and credentials) after 2022.8.5?
pipenv lock -r does variable substitution (as of 2022.8.5) while pipenv requirements does not. This is needed when including private pypi repos with credentials that are substituted by env variables. The lock command was removed in 2022.8.13.

[matteius]

@EugeniuZ there is an enhancement that should be made to the requirements command for this case. Ideally someone that is setup to utilize this case could also work on the fix for it based on the discussion in this ticket, I think the variables are just mis-represented and should have brackets around them. I understand that this is a high priority item for some folks, but we need to fix the new command to be correct rather than restore the old lock -r code.

[dqkqd]

pip want requirements file could be platform independent and could be used inside URL, so it only supports POSIX format with UPPERCASE. There's a discussion about that pypa/pip#3514.
Should pipenv does this as well?

[matteius]

As a workaround, couldn't @RintaroMiyamoto specify the variables in the same way in the Pipfile?

"sources": [ "url":"https://${USER}:${ACCESS_TOKEN}@gitlab.com/private/repo" ]

[matteius]

What is the preferred way to generate a requirements.txt file with private repo (and credentials) after 2022.8.5? pipenv lock -r does variable substitution (as of 2022.8.5) while pipenv requirements does not. This is needed when including private pypi repos with credentials that are substituted by env variables.

@EugeniuZ The problem is that it is a security risk for us to automatically import credentials into artifacts that are intended to be checked into source control. The correct way to handle this is using pip compliant environment variables, meaning ${VARIABLE} and not $VARIABLE -- then pipenv will bring in the variables unexpanded, and they should work well with pip and just need to be defined on the users system -- a system administrator could set these for a CI, and local users can set these values as well in a .env file for the project which is intended to not be checked into source control. Pipenv should already support all of this as described, but please correct me if you find a bug using this approach.