Installing dependencies from a private repository for git pull deployments

[michael-follmann]

Is it somehow possible to install private dependencies for git-based deployments?

I did not have any issues to access a private Git repository or installing public Python packages (using the job variable pip_packages). However, I couldn't find any details about installing Python packages from a private repository (Sonatype Nexus Repository in my case). Is it somehow possible to do this? Maybe by providing additional configuration/credentials to configure pip? Or using the run_shell_script utility step?

I'm aware that I can achieve this with Docker images.

[bdalpe]

@michael-follmann I believe you can use the PIP_EXTRA_INDEX_URL environment variable to accomplish this.

Expanding on the code for Add dependencies, add the PIP_EXTRA_INDEX_URL to the env block.

Note: I'm using Prefect's EXTRA_PIP_PACKAGES env for clarity in this example, but you can just include the package in the pip_packages variable if you like.

from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="https://github.com/prefecthq/demo.git",
        entrypoint="flow.py:my_flow",
    ).deploy(
        name="test-managed-flow",
        work_pool_name="my-managed-pool",
        job_variables={
          "pip_packages": ["pandas", "prefect-aws"],
          "env": {
            "PIP_EXTRA_INDEX_URL": "http://user:pass@nexus/repository/my-private-package",
            "EXTRA_PIP_PACKAGES": "my-private-package"
          }
        }
    )

Give this a try, and please let us know!