Skip to content

Add pyproject.toml support for container dependency installs#84

Merged
JyotinderSingh merged 2 commits intomainfrom
toml-builds
Mar 9, 2026
Merged

Add pyproject.toml support for container dependency installs#84
JyotinderSingh merged 2 commits intomainfrom
toml-builds

Conversation

@JyotinderSingh
Copy link
Collaborator

Summary

Fixes #81

  • Support pyproject.toml as an alternative to requirements.txt for specifying container dependencies. [project.dependencies] are extracted and installed via pip
  • Discovery walks up the directory tree checking for requirements.txt first, then pyproject.toml at each level; requirements.txt always takes precedence
  • Extracted dependencies are fed through the existing pipeline (JAX filtering, content hashing, Dockerfile generation) with no changes to downstream logic

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances Keras Remote's dependency management by introducing support for pyproject.toml files. This provides users with more flexibility in defining their project dependencies, aligning with modern Python packaging standards. The changes ensure that dependencies declared in pyproject.toml are correctly identified, parsed, and integrated into the container build process, maintaining the existing robust dependency handling and caching mechanisms.

Highlights

  • Pyproject.toml Support: Added support for pyproject.toml as an alternative to requirements.txt for specifying container dependencies. The [project.dependencies] section is now extracted and used for installation.
  • Dependency Discovery Logic: The dependency discovery mechanism now walks up the directory tree, checking for requirements.txt first, then pyproject.toml at each level. requirements.txt takes precedence if both files exist in the same directory.
  • Integration with Existing Pipeline: Extracted dependencies from pyproject.toml are fed through the existing pipeline, including JAX filtering, content hashing, and Dockerfile generation, without changes to downstream logic.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • README.md
    • Updated documentation to reflect pyproject.toml support for dependencies, including an example and clarification on requirements.txt precedence.
  • keras_remote/backend/execution.py
    • Modified _find_requirements to search for pyproject.toml in addition to requirements.txt, with requirements.txt taking precedence.
    • Updated docstrings and logging messages to reflect the new dependency file discovery logic.
  • keras_remote/backend/execution_test.py
    • Added comprehensive unit tests for _find_requirements to verify correct detection of pyproject.toml.
    • Included tests for requirements.txt precedence over pyproject.toml when both are present.
    • Added tests for parent directory searching for pyproject.toml.
  • keras_remote/cli/output.py
    • Updated a user tip message to include pyproject.toml as an alternative for specifying dependencies.
  • keras_remote/infra/container_builder.py
    • Imported the tomllib module for parsing TOML files.
    • Implemented a new function _parse_pyproject_dependencies to extract [project.dependencies] from a pyproject.toml file.
    • Modified get_or_build_container to conditionally parse pyproject.toml or requirements.txt based on the detected file type.
  • keras_remote/infra/container_builder_test.py
    • Added imports for os and tempfile for test utility.
    • Introduced TestParsePyprojectDependencies class with tests for extracting dependencies, handling missing dependencies, and ignoring optional dependencies from pyproject.toml.
  • tests/e2e/pyproject_deps_test.py
    • Added a new end-to-end test file to verify that dependencies specified in pyproject.toml are correctly installed and available in the remote environment.
    • Included tests for JAX filtering of dependencies from pyproject.toml.
    • Verified handling of pyproject.toml files without declared dependencies.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for pyproject.toml as an alternative to requirements.txt for specifying container dependencies, which is a valuable enhancement. The implementation is well-structured, with clear logic and comprehensive unit and end-to-end tests. The documentation and user-facing messages have also been updated accordingly. I have one suggestion to improve type consistency in the new parsing function. Overall, this is a high-quality contribution.

Comment on lines +86 to +106
def _parse_pyproject_dependencies(pyproject_path: str) -> str | None:
"""Extract ``[project.dependencies]`` from a pyproject.toml file.

Reads only the core dependency list defined under the ``[project]`` table.
Optional dependency groups (``[project.optional-dependencies]``) are ignored;
users who need those should use a ``requirements.txt`` instead.

Args:
pyproject_path: Absolute path to a ``pyproject.toml`` file.

Returns:
Newline-separated dependency strings in PEP 508 format suitable for
``pip install``, or ``None`` if the file declares no dependencies.
"""
with open(pyproject_path, "rb") as f:
data = tomllib.load(f)

deps = data.get("project", {}).get("dependencies", [])
if not deps:
return None
return "\n".join(deps) + "\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This is a well-implemented function. For better type consistency and to align with how empty requirements.txt files are handled (which are read as an empty string), consider changing the return type from str | None to just str. The function can return an empty string "" when no dependencies are found.

This would involve changing the function signature, implementation, and docstring. The corresponding unit tests would also need to be updated to check for "" instead of None on failure cases.

def _parse_pyproject_dependencies(pyproject_path: str) -> str:
  """Extract ``[project.dependencies]`` from a pyproject.toml file.

  Reads only the core dependency list defined under the ``[project]`` table.
  Optional dependency groups (``[project.optional-dependencies]``) are ignored;
  users who need those should use a ``requirements.txt`` instead.

  Args:
      pyproject_path: Absolute path to a ``pyproject.toml`` file.

  Returns:
      Newline-separated dependency strings in PEP 508 format suitable for
      ``pip install``, or an empty string if the file declares no dependencies.
  """
  with open(pyproject_path, "rb") as f:
    data = tomllib.load(f)

  deps = data.get("project", {}).get("dependencies", [])
  return "\n".join(deps) + "\n" if deps else ""

@JyotinderSingh JyotinderSingh merged commit 83f83c6 into main Mar 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for pyproject.toml for container package installs

2 participants