We've been having some trouble with Github CI recently on this repository, with frequent broken builds, flakiness and slowdowns, with kedro-datasets being the main culprit. It would be interesting to investigate if we can improve our CI setup to mitigate those issues.
Some points to be taken into consideration:
Dependency management
kedro-datasets alone has over 50 test requirements. A lot of them have no upper bound constraints, which means they can push a release that might break our CI at any moment. And with new datasets being added regularly, this number tends to grow.
For some specific examples:
transformers[torch], ibis-framework[duckdb,examples] and huggingface_hub` have no constraints at all.polars>=1.0 and langchain-*>=0.1.x are only limited on the lower bound and have frequent releases.dask[complete]>=2021.10 has a 5-year-old lower bound and no upper bound.SQLAlchemy>=1.2 spans the 1.x to 2.0 boundary which dropped the legacy API
Any of these releasing a breaking change silently fails CI with no change on our side.
Consider also that CI is using the latest version of uv to run tests, and running them using the main branch of the Kedro repo and not the latest release, so these are two other points of unpredictability.
Limited computational resources
We're working with the Standard GitHub-hosted runners, that have a relatively small amount of RAM available - 16gb for ubuntu-latest and windows-latest, and only 7gb for macos-latest.
With unit tests running on ubuntu-latest and windows-latest each on python versions from 3.10 to 3.13, we have 8 parallel unit-test jobs spawned per PR push, plus lint, check-docs, and detect-secrets on top. Each ubuntu job runs --numprocesses 4 with Spark included. That's up to 16 Spark JVMs from a single PR push.
This causes slowdowns that become particularly noticeable when multiple people are pushing PRs at the same time and CI slows down to a crawl. It's not that uncommon to see the CI lasting over 30 minutes to finish in situations like this, with some extreme cases taking multiple hours to finish (the recent Pyladies OS event is an example).
Factor in that there's no concurrency: cancel-in-progress on any workflow. If someone pushes 3 commits in quick succession (e.g. addressing review comments), 3 full workflow runs will be happening simultaneously and none will cancel the previous one. And on top of all of that, there's no explicit timeout for any of the jobs. If a session hangs (deadlock, ran out of memory, zombie process, etc.), the job runs until it reaches GitHub's 6 hour workflow limit.
We've been having some trouble with Github CI recently on this repository, with frequent broken builds, flakiness and slowdowns, with
kedro-datasetsbeing the main culprit. It would be interesting to investigate if we can improve our CI setup to mitigate those issues.Some points to be taken into consideration:
Dependency management
kedro-datasetsalone has over 50 test requirements. A lot of them have no upper bound constraints, which means they can push a release that might break our CI at any moment. And with new datasets being added regularly, this number tends to grow.For some specific examples:
transformers[torch], ibis-framework[duckdb,examples] andhuggingface_hub` have no constraints at all.polars>=1.0andlangchain-*>=0.1.xare only limited on the lower bound and have frequent releases.dask[complete]>=2021.10has a 5-year-old lower bound and no upper bound.SQLAlchemy>=1.2spans the 1.x to 2.0 boundary which dropped the legacy APIAny of these releasing a breaking change silently fails CI with no change on our side.
Consider also that CI is using the latest version of
uvto run tests, and running them using themainbranch of the Kedro repo and not the latest release, so these are two other points of unpredictability.Limited computational resources
We're working with the Standard GitHub-hosted runners, that have a relatively small amount of RAM available - 16gb for
ubuntu-latestandwindows-latest, and only 7gb formacos-latest.With unit tests running on
ubuntu-latestandwindows-latesteach on python versions from 3.10 to 3.13, we have 8 parallel unit-test jobs spawned per PR push, plus lint, check-docs, and detect-secrets on top. Each ubuntu job runs--numprocesses 4with Spark included. That's up to 16 Spark JVMs from a single PR push.This causes slowdowns that become particularly noticeable when multiple people are pushing PRs at the same time and CI slows down to a crawl. It's not that uncommon to see the CI lasting over 30 minutes to finish in situations like this, with some extreme cases taking multiple hours to finish (the recent Pyladies OS event is an example).
Factor in that there's no
concurrency: cancel-in-progresson any workflow. If someone pushes 3 commits in quick succession (e.g. addressing review comments), 3 full workflow runs will be happening simultaneously and none will cancel the previous one. And on top of all of that, there's no explicit timeout for any of the jobs. If a session hangs (deadlock, ran out of memory, zombie process, etc.), the job runs until it reaches GitHub's 6 hour workflow limit.