Skip to content

Conversation

@amyssnippet
Copy link
Contributor

Closes #127

This PR refactors the CI pipeline to improve speed and developer experience.

Changes:

  • Parallel Execution: Split the monolithic build-and-test job into separate lint, test, matrix-tests, and docs jobs.
    • lint and test run in parallel for faster feedback.
    • matrix-tests and docs run only after lint passes to save resources.
  • Dependency Management: Updated pyproject.toml with optional-dependencies (lint, test, docs) to allow granular installation.
  • Fix: Ensured lint and test jobs install necessary dependencies (including examples) to prevent import errors during checks.

@amyssnippet
Copy link
Contributor Author

image

@amyssnippet
Copy link
Contributor Author

@ryan112358 i think this is an important update related to developer experience.

earlier the check system was monolith and everything seems to be sequential, now it is modular and directed acyclic graph (dag) architecture for the ci workflows

here is a comparision table

CI Workflow Comparison: Before vs. After

Feature Old CI (Monolith) New CI (Modular)
Structure Single Giant Job (build-and-test) Multiple Specialized Jobs (lint, test, matrix, docs)
Execution Flow Linear / Sequential (Waterfall)
Step 1 $\rightarrow$ Step 2 $\rightarrow$ Step 3
Parallel / DAG (Directed Graph)
lint & test run together; others wait.
Feedback Loop "Whack-a-Mole"
If Lint fails, the pipeline stops immediately. You never know if Tests pass until you fix the Lint error.
Immediate & Complete
You see Lint errors AND Unit Test errors at the same time.
Total Time Slower
Sum of all steps: $T_{lint} + T_{test} + T_{heavy}$
Faster
Determined by the longest path: $max(T_{lint}, T_{test}) + T_{heavy}$
Dependencies Bloated
Installed everything (TensorFlow, Keras, etc.) just to check spelling.
Optimized
lint installs light tools; matrix-tests installs heavy tools.
Cost Efficiency Medium
Stops early on failure, but wastes time installing heavy deps for simple checks.
High
Blocks expensive matrix-tests if code is "ugly" (Lint fails), but runs cheap Unit Tests immediately.
Maintainability Hard
One 50-line script block that is hard to debug.
Easy
Separate distinct blocks. If docs break, you know exactly where to look.

@amyssnippet
Copy link
Contributor Author

@ryan112358 i think this step should be taken into action, and if any other internal google checks are there, you can tell it here i will implement so everything will be very clear to the developer, regarding their code changes.

let me know if anything need improvements, Thanks!

@debanganghosh08
Copy link

Closes #127

This PR refactors the CI pipeline to improve speed and developer experience.

Changes:

  • Parallel Execution: Split the monolithic build-and-test job into separate lint, test, matrix-tests, and docs jobs.

    • lint and test run in parallel for faster feedback.
    • matrix-tests and docs run only after lint passes to save resources.
  • Dependency Management: Updated pyproject.toml with optional-dependencies (lint, test, docs) to allow granular installation.

  • Fix: Ensured lint and test jobs install necessary dependencies (including examples) to prevent import errors during checks.

@amyssnippet Anmol, this is a great refactor. Moving to a modular DAG architecture will definitely speed up the feedback loop. I ran into the dependency issue with Flax in PR #126, and having a dedicated requirements/optional deps setup for examples would have made debugging much more smoother. I support this improvement as it would add a better developer experience! 👍

@amyssnippet
Copy link
Contributor Author

@ryan112358 , i guess each and every internal checks seems to be good. so shall we proceed towards merging this soon, as this will actively improve the DX

@ryan112358
Copy link
Collaborator

Separate distinct blocks. If docs break, you know exactly where t

Great improvements, thanks for the initiative! Should be merged today, just waiting on a second internal review

@copybara-service copybara-service bot merged commit dbf94ba into google-deepmind:main Jan 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor CI pipeline: Parallelize jobs and decouple dependencies

3 participants