Flyte v2: with_uv_project - how to install all optional dependencies

[hmf]

I am testing v2 of flyte (on our internal cluster) using an existing project from a Git repository. This project uses uv´s pyproject.toml file, which uses the project.optional-dependencies group. Testing shows that these optional dependencies are not installed. How does one use with_uv_project to pull in all dependencies?

TIA

[omni-front]

with_uv_project doesn't directly handle installing optional dependencies specified in pyproject.toml. Here's how you can tackle this:

1. Modify with_uv_project: You might consider extending or wrapping with_uv_project to include a step that installs those optional dependencies.

2. Use pip: A straightforward way is to install the optional dependencies manually using pip. You can do this by specifying the extras in your pip install command. For example:

   pip install .[all]

   Replace all with whatever key you have that groups all your optional dependencies in the pyproject.toml.

3. Automate with a script: If you're repeatedly doing this, a script might help. Here's a simple example in Python:

   import subprocess
   
   def install_dependencies():
       result = subprocess.run(['pip', 'install', '.[all]'], check=True)
       if result.returncode == 0:
           print("Dependencies installed successfully.")
       else:
           print("Failed to install dependencies.")
   
   if __name__ == "__main__":
       install_dependencies()

4. CI/CD Pipeline: If this is for a CI/CD pipeline, ensure your pipeline script includes the step to install optional dependencies before invoking with_uv_project.

5. Alternative Approach: If you're open to restructuring, consider whether all these optional dependencies are truly optional. Sometimes consolidating them can simplify dependency management.

For more details on dependency groups, check out the PEP 621 which outlines how pyproject.toml should be structured.

Let me know if there's a specific part of the process causing issues.

[omni-front]

not sure about this one tbh. maybe check if there's an issue already open in their repo or ask around in the flyte chat?

[hmf]

@omni-front Thanks for the feedback. After revisiting this I think it is possible to use one of these:

uv pip install -r pyproject.toml --all-extras
uv sync --all-extras

To automate this, I used:

image = flyte.Image.from_debian_base(
    name="demo-image",
    registry="docker-registry.url/union-ai/registry",
    registry_secret="my-secret"
  ).with_uv_project(
      pyproject_file="pyproject.toml",
      extra_args="--all-groups --all-extras"
      )

Think this is working ok.

HTHs anyone else with this issue.

[omni-front]

Nice, glad you found a solution for automating it!