Skip to content

Support multiple Git DAG bundles - #6

Merged
eugene-panin merged 1 commit into
mainfrom
agent/multi-git-dag-bundles
Jul 22, 2026
Merged

Support multiple Git DAG bundles#6
eugene-panin merged 1 commit into
mainfrom
agent/multi-git-dag-bundles

Conversation

@eugene-panin

@eugene-panin eugene-panin commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add --bundle-manifest to load multiple Git-backed DAG sources into one Airflow instance.
  • Create an independent Airflow Git Connection and native GitDagBundle for every source.
  • Validate required manifest fields, duplicate source names, and derived Connection ID collisions.
  • Document the manifest format, shared-image contract, and trust boundary.

Rationale

The workbench remains shared infrastructure for trusted products. DAG entrypoints may come from separate repositories while product business logic stays with each product. Python dependencies are never installed while parsing a DAG: a multi-source deployment uses an explicit, tested shared image with pinned product package revisions.

Validation

  • python3 -m py_compile bin/etl-workbench
  • python3 -m unittest discover -s tests -v — 3 tests passed

Boundary

This separates operational ownership through product-specific Connections, Pools, and object prefixes. It does not make the local workbench an isolation boundary for untrusted DAG code.

@eugene-panin eugene-panin self-assigned this Jul 22, 2026
@eugene-panin
eugene-panin force-pushed the agent/multi-git-dag-bundles branch from a240c68 to e529416 Compare July 22, 2026 17:42
@eugene-panin eugene-panin changed the title Поддержать несколько Git DAG bundles Support multiple Git DAG bundles Jul 22, 2026
@eugene-panin
eugene-panin marked this pull request as ready for review July 22, 2026 17:49
@eugene-panin
eugene-panin merged commit e9cd06a into main Jul 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant