Thoughts on monorepos (dependency isolation)

[astrojuanlu]

With the deprecation of micropackaging #3854, it would be good to explore monorepo approaches. What "monorepo" actually means depends on the use case.

[astrojuanlu]

One request that comes up from time to time is "1 Kedro project, different groups of dependencies". At the moment, 1 Kedro project = 1 Python package, therefore having completely disjoint dependencies is not possible. There are several possible solutions:

• 1 Python package
  • one could have a very minimal set of common dependencies (say, just kedro>=0.19.6) and then groups of dependencies per use case using PEP 621 [project.optional-dependencies].
  • There's also the draft PEP 735 about dependency groups (to be submitted in 2 weeks?)
• 1 "root" Python package, other connected packages
  • There's uv workspaces (akin to Cargo monorepos) https://docs.astral.sh/uv/concepts/workspaces/

[astrojuanlu]

Requested clarification on "non-workspaces" astral-sh/uv#5605 (comment)

[astrojuanlu]

Beyond the "different groups of dependencies" use case, today a user asked about having different packages under src:

my_project
├── src
|   ├── use_case_1
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_1_data_engineering
|       |   ├── use_case_1_modeling
|   ├── use_case_2
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_2_data_ingestion
|       |   ├── use_case_2_data_science

I tried this with both setuptools and PDM build backends and it worked like a charm (hatchling on the other hand chocked). Sample pyproject.toml:

[project]
name = "multi"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.11"
dependencies = []

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

However, I'm sure Kedro has some hardcoded logic that looks into a specific path inside src/. That logic also enables kedro pipeline create and other commands that manipulate the source code. So supporting this fully in Kedro might require some thought.

[astrojuanlu]

To note, PEP 735 (Dependency Groups) is now accepted. https://peps.python.org/pep-0735/

[astrojuanlu]

uv and tox already shipped support for it

...but pip hasn't, and might never do it pypa/pip#12963 (comment)

[astrojuanlu]

Useful ecosystem overview of how other languages do this https://gist.github.com/konstin/6d04f111563641beb10facb617fe0eb3

[astrojuanlu]

Yet another user concerned about the 1 Kedro pipeline = 1 set of requirements https://kedro.hall.community/evaluating-kedro-for-data-engineering-processes-qFYCwWk5VKQh

[astrojuanlu]

@marrrcin @Galileo-Galilei @sbrugman would be interested if you also agree that dependency isolation is something we need to also consider within this problem space?

I think it's a valid consideration. At the very least, there needs to be an answer as to how you handle pipelines that have multiple sets of dependencies (e.g. should data_engineering and data_science be separate pipelines, or should they be separate Kedro projects, if they have a separate set of, potentially-incompatible, dependencies?). In my past work on large Kedro projects (ages ago), I heavily leveraged monorepo approaches, as in #4147, because there is also a tradeoff to maintaining multiple Kedro projects for a single use case.

I can't find my previous comments to this extent (perhaps they were verbal), but my opinions are:

• Micro-packaging was always a bad idea (for Kedro framework to support); they were introduced to solve a QB-internal use case that never even leveraged them/built their own solution.
• However, being able to define separate requirements for top-level pipelines IS useful, especially for deployment. The story for running a pipeline locally with multiple sets of dependencies was never really there, but it seems useful.
• Supporting complex node grouping is unnecessary, at least to start with; that's akin to the micro-packaging over-engineering problem. The vast majority of users would benefit from being able to deploy each modular pipeline (with it's own namespace) separately. This also creates pretty clear boundaries where node persistence, etc. may be required.

TL;DR I agree with the idea of namespaces and supporting deployment based on namespaces, but I further posit that namespaces just being per modular pipeline is sufficient, even for deployment, and that will also make them easier to adopt.

Originally posted by @deepyaman in #4319 (comment)

[astrojuanlu]

I mostly agree with all of deepyaman's comments. I actually crave for pipelines requirements isolation, but I've never thought about requirements isolation at namespace level. This would make sense from an orchestrator point of view with "big nodes" filtered out from namespaces pipelines, but I am not sure this is what we should focus on in the short term because it's likely a hard problem.

Originally posted by @Galileo-Galilei in #4319 (comment)

[astrojuanlu]

I agree with the need for dependency isolation for large projects 👍🏻

Originally posted by @marrrcin in #4319 (comment)

[astrojuanlu]

@datajoely says:

Also on the monorepo point I saw this recently

https://chrismati.cz/posts/uv-pex-monorepo/

(#4618 (comment))

[astrojuanlu]

And pex wouldn't even be necessary for us. Yes, I think it's clear uv got a lot of things right, also for monorepos :)

[astrojuanlu]

crazy idea for the "conflicting dependencies" problem: a modified find_pipelines so that you can do

# pipeline_registry.py

# from kedro.framework.project import find_pipelines
from kedro_monorepo.util import find_pipelines  # <-------------

...
def register_pipelines() -> dict[str, Pipeline]:
    # Unchanged
    pipelines = find_pipelines()
    pipelines["__default__"] = sum(pipelines.values())
    return pipelines

but this find_pipelines scans for importlib entry points, so that you can define your pipelines in external packages, that might have their own set of dependencies.

The only missing bit would be having some tooling to make this easier, like kedro-monorepo pipeline create ....

[datajoely]

this is clever, I was more thinking about a uv dependency group for each isolated pipeline

[astrojuanlu]

That's simpler at the cost of keeping everything in 1 package. Worth evaluating pros and cons, or just ask users what they'd prefer

[astrojuanlu]

Still, I think it's clear that, one way or another, this is easily solvable with current tech

[astrojuanlu]

Been thinking more about the dependency group idea. I think it's an intermediate step towards something else - in other words, it doesn't solve the problem that micropackaging used to solve:

Micro-packaging allows users to share Kedro micro-packages across codebases, organisations and beyond. A micro-package can be any part of Python code in a Kedro project including pipelines and utility functions.

[datajoely]

This is a fair, but in terms of history micropackaging was primarily introduced for an internal pain point which was still bypassed by internal teams with their own bespoke solution.

[astrojuanlu]

uv solves the dependency isolation issue for Bruin by just running everything in isolated, ephemeral environments https://www.linkedin.com/posts/burakkarakan_theres-no-other-tool-in-the-market-that-activity-7325440443893092353-rbFu?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAXcavUB6j26_YjUtYinmzmdjTchguHHuG4

[gitgud5000]

Hi @astrojuanlu, thanks for the insights above.

I have a Kedro project where different pipelines (or parts of them) run on heterogeneous infrastructure — some on AzureML with GPU/CUDA support, others on CPU-only CI/VMs.

Naturally, some parts of the project depend on GPU libraries like torch, which aren’t available or necessary in all environments. I want to avoid import-time errors/warnings when these libraries aren’t installed or CUDA devices aren’t available.

To work around this, I’m using a pattern like:

def gpu_check():
    if torch.cuda.is_available():
        import torch

Would you consider this an acceptable practice within Kedro's design?

Also, would you recommend managing these environment-specific dependencies via [project.optional-dependencies] in pyproject.toml (e.g., gpu, cpu, etc.), or is it better to go further and split into separate Kedro projects/packages if things get more complex?

I want to keep a clean, maintainable structure that avoids these import-time issues across environments.

Thanks!

[astrojuanlu]

[project.optional-dependencies] in pyproject.toml (e.g., gpu, cpu, etc.), or is it better to go further and split into separate Kedro projects/packages if things get more complex?

For the time being, I think the 3 approaches are basically fine:

1. Conditional imports (ugly, but it works)
2. Optional dependencies (might need conditional imports anyway, otherwise it might interfere with Kedro's discovery mechanism or give warnings)
3. Split the Kedro project (more below)

@datajoely is an advocate of (2), I think it solves the dependency issue but I think it might still require (1), which is ugly (haven't checked). It will definitely work though - notice that in recent versions, Kedro is fully compatible with all the Python Packaging standards.

In this long monologue thread I'm trying to figure out a way to do (3) such that instead of different Kedro projects, we still have 1 Kedro project + a leaner / more lightweight way of splitting the pipelines.

[astrojuanlu]

Another problem worth looking at while solving this is how to help teams separate the business logic from the Kedro logic even more. Some users put Kedro logic inside their node functions because they want to do dynamic or unconventional stuff. However, this makes their code more Kedro-dependent and less reusable.

[datajoely]

this is a good point, because I'd argue us autogenerating the nodes.py nudges people into this pattern of high coupling. In my opinion your declaration of 'flow', pipeline.py is correct, but your functional logic should live outside of Kedro in independently well tested packages.