Releases: astronomer/dag-factory
Releases · astronomer/dag-factory
v1.0.1
Fixed
- Remove sla_miss_callback from dag_kwargs for Airflow >= 3.1.0 by @jroachgolf84 in #586
- Add logs for the removal of sla_miss_callback by @pankajastro in #588
Docs
Other Changes
- Restrict the hatch version to fix the CI by @pankajastro in #585
- Bump GitHub Actions dependencies by @dependabot in #578, #579, #580 and #581
Full Changelog: v1.0.0...v1.0.1
v1.0.0
Breaking Changes
- Airflow providers are now optional dependencies by @pankajastro in #486
- Previously,
dag-factoryenforced the installation ofapache-airflow-providers-httpandapache-airflow-providers-cncf-kubernetes. These Airflow providers dependencies are now optional. If your DAGs depend on these providers, you must install them manually. Alternatively, you can installdag-factorywith extras likedag-factory[all],dag-factory[kubernetes], etc.
- Previously,
- Removed
clean_dagsfunction by @pankajastro in #498- You no longer need to call
example_dag_factory.clean_dags(globals())in your DAG files. DAG cleanup is now controlled via the Airflow config settingAIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL.
- You no longer need to call
- Remove
schedule_intervalparameter from DAG configuration YAML by @viiccwen in #503- Use
scheduleparameter instead ofschedule_interval.
- Use
- Change
DagFactoryclass access to private by @pankajastro in #509- The import path
from dagfactory import DagFactoryhas been removed. - The class
DagFactoryhas been renamed to_DagFactory. - The
generate_dagsmethod ofDagFactoryhas been renamed to_generate_dags.
- The import path
- Remove Inconsistent Parameters for Airflow Consistent by @pankajastro in #512
- Removed
dagrun_timeout_secfrom dag param. - Removed
retry_delay_sec,sla_secsfrom default_args. - Removed accepting
execution_timeoutas integer. - Removed
execution_timeout_secs,sla_secsandexecution_delta_secsfrom task param.
- Removed
- Remove custom parsing for Kubernetes object and refactor KPO to use
__type__syntax by @pankajastro in #523- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
__type__syntax to supply Kubernetes object in your YAML DAG. For an example KPO configuration, visit: KubernetesPodOperator Documentation.
- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
- Consolidate
!and,!or,!join,andandorkey in YAML DAG configuration by @pankajastro in #525- Use
__and__,__or__andjoin__instead
- Use
- Remove custom parsing for DAG parameter
timetableby @pankajastro in #533- Use the
__type__annotation for thetimetableparameter.
- Use the
- Rename parameter of
load_yaml_dagsand_DagFactoryto reflect behaviour by @pankajastro in #546- Rename
configtoconfig_dict - Rename
default_args_config_pathtodefaults_config_path - Rename
default_args_config_dicttodefaults_config_dict
- Rename
Added
- Support dag-level arguments in global defaults by @gyli in #480
- Support
*argsin custom Python object by @pankajastro in #484 - Support tasks and task_groups as lists by @pankajkoti in #487
- Support overriding
defaults.ymlbased on the directory hierarchy by @tatiana in #500 - Introduced DAG Factory CLI by @tatiana in #510
- Added
lintcommand to CLI by @tatiana in #513 - Add convert CLI command to migrate from af2 to af3 by @tatiana in 539
- Allow yaml/yml suffix for shared defaults by @pankajastro in #567
Fixed
- Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by @pankajastro in #485
- Ensure
dag_paramscontainschedulebefore operating on it by @pankajkoti in #488 - Fix
start_date,end_dateat the DAG level by @pankajastro in #495 - Allow
execution_timeoutindefault_argsby @pankajastro in #501 - Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in #544
- Fix require response_check param in http sensor by @pankajastro in #576
- Add none check before evaluating lambda func in HttpSensor @pankajastro in #577
Docs
- Restore basic DAG example by @pankajastro in #483
- Replace the usages in example dags, tests and docs for tasks and taskgroups to be list by @pankajkoti in #492
- Update default documentation based on #500 by @tatiana in #504
- Add more examples for Custom Python object by @pankajastro in #506
- Add documentation for DAG Factory CLI by @tatiana in #511
- Add documentation and example YAMLs for task and task_group configuration formats by @pankajkoti in #530
- Add migration guide docs by @pankajastro in #532
- Docs: Fix rendering of note block by @pankajastro in #537
- Document the Asset example DAG by @pankajastro in #538
- Add docs for the CLI convert command by @tatiana in #541
- Add remaining breaking changes in migration guide by @pankajastro in #549
- Fix typos in scheduling and datasets docs by @viiccwen in #565
- docs: Make markdownlint happy (fix MD007 ul-indent in dev/README.md) by @viiccwen in #566
- Update Index.md by @pankajastro in #570
- Document dag-factory reserved keys by @pankajastro in #571
- Add an introduction to the migration guide by @pankajastro in #572
Other Changes
- Improve unit tests to disregard
$AIRFLOW_HOMEby @tatiana in #490 - Resolve unpinned action reference error alerts raised by Zizmor by @pankajkoti in #493
- Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #494
- Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #496
- CI: Add GitHub CodeQL analysis workflow (
codeql.yml) by @pankajkoti in #497 - Fix deploy pages job missing credentials by @pankajkoti in #499
- Add the breaking changes to changelog by @pankajastro in #502
- Add pre-commit to update
uv.lockby @pankajastro in #514 - Remove
clean_dagsusage from object storage DAG by @pankajastro in #515 - Remove broad exceptions and catch more specific exceptions by @pankajastro in #519
- Add missing env in contributing doc by @pankajastro in #522
- Enhance PyPI Stats API error handling by @viiccwen in #535
- Update example to be Airflow 3 compatible by @tatiana in 540
- Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in #543
- Bump actions/download-artifact from 4 to 5 by @dependabot in #548, #558, #559, #560, #562, #653 and #575
- Update pyproject.toml to Sync Test Versions with CI/CD Pipeline by @viiccwen in #553
- Fix file URI format in ObjectStoragePath example to prevent duplicate slashes by @viiccwen in #556
- CI: Only build docs on PR push event and deploy for merge and release by @pankajastro in #568
v1.0.0a4
What's Changed
- CI: Only build docs on PR push event and deploy for merge and release by @pankajastro in #568
- Add an introduction to the migration guide by @pankajastro in #572
- Update Index.md by @pankajastro in #570
- Allow yaml/yml suffix for shared defaults by @pankajastro in #567
- Bump astral-sh/setup-uv from 6.6.0 to 6.6.1 by @dependabot[bot] in #575
- Fix require response_check param in http sensor by @pankajastro in #576
- Document dag-factory reserved keys by @pankajastro in #571
Full Changelog: v1.0.0a3...v1.0.0a4
v1.0.0a3
CI: Only build docs on PR push event and deploy for merge and release PR: #568
Full Changelog: v1.0.0a2...v1.0.0a3
v1.0.0a2
[1.0.0a2] - 2025-08-11
Breaking Changes
- Airflow providers are now optional dependencies by @pankajastro in #486
- Previously,
dag-factoryenforced the installation ofapache-airflow-providers-httpandapache-airflow-providers-cncf-kubernetes. These Airflow providers dependencies are now optional. If your DAGs depend on these providers, you must install them manually. Alternatively, you can installdag-factorywith extras likedag-factory[all],dag-factory[kubernetes], etc.
- Previously,
- Removed
clean_dagsfunction by @pankajastro in #498- You no longer need to call
example_dag_factory.clean_dags(globals())in your DAG files. DAG cleanup is now controlled via the Airflow config settingAIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL.
- You no longer need to call
- Remove
schedule_intervalparameter from DAG configuration YAML by @viiccwen in #503- Use
scheduleparameter instead ofschedule_interval.
- Use
- Change
DagFactoryclass access to private by @pankajastro in #509- The import path
from dagfactory import DagFactoryhas been removed. - The class
DagFactoryhas been renamed to_DagFactory. - The
generate_dagsmethod ofDagFactoryhas been renamed to_generate_dags.
- The import path
- Remove Inconsistent Parameters for Airflow Consistent by @pankajastro in #512
- Removed
dagrun_timeout_secfrom dag param. - Removed
retry_delay_sec,sla_secsfrom default_args. - Removed accepting
execution_timeoutas integer. - Removed
execution_timeout_secs,sla_secsandexecution_delta_secsfrom task param.
- Removed
- Remove custom parsing for Kubernetes object and refactor KPO to use
__type__syntax by @pankajastro in #523- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
__type__syntax to supply Kubernetes object in your YAML DAG. For an example KPO configuration, visit: KubernetesPodOperator Documentation.
- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
- Consolidate
!and,!or,!join,andandorkey in YAML DAG configuration by @pankajastro in #525- Use
__and__,__or__andjoin__instead
- Use
- Remove custom parsing for DAG parameter
timetableby @pankajastro in #533- Use the
__type__annotation for thetimetableparameter.
- Use the
- Rename parameter of
load_yaml_dagsand_DagFactoryto reflect behaviour by @pankajastro in #546- Rename
configtoconfig_dict - Rename
default_args_config_pathtodefaults_config_path - Rename
default_args_config_dicttodefaults_config_dict
- Rename
Added
- Support dag-level arguments in global defaults by @gyli in #480
- Support
*argsin custom Python object by @pankajastro in #484 - Support tasks and task_groups as lists by @pankajkoti in #487
- Support overriding
defaults.ymlbased on the directory hierarchy by @tatiana in #500 - Introduced DAG Factory CLI by @tatiana in #510
- Added
lintcommand to CLI by @tatiana in #513 - Add convert CLI command to migrate from af2 to af3 by @tatiana in 539
Fixed
- Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by @pankajastro in #485
- Ensure
dag_paramscontainschedulebefore operating on it by @pankajkoti in #488 - Fix
start_date,end_dateat the DAG level by @pankajastro in #495 - Allow
execution_timeoutindefault_argsby @pankajastro in #501 - Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in #544
Docs
- Restore basic DAG example by @pankajastro in #483
- Replace the usages in example dags, tests and docs for tasks and taskgroups to be list by @pankajkoti in #492
- Update default documentation based on #500 by @tatiana in #504
- Add more examples for Custom Python object by @pankajastro in #506
- Add documentation for DAG Factory CLI by @tatiana in #511
- Add documentation and example YAMLs for task and task_group configuration formats by @pankajkoti in #530
- Add migration guide docs by @pankajastro in #532
- Docs: Fix rendering of note block by @pankajastro in #537
- Document the Asset example DAG by @pankajastro in #538
- Add docs for the CLI convert command by @tatiana in #541
- Add remaining breaking changes in migration guide by @pankajastro in #549
Other Changes
- Improve unit tests to disregard
$AIRFLOW_HOMEby @tatiana in #490 - Resolve unpinned action reference error alerts raised by Zizmor by @pankajkoti in #493
- Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #494
- Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #496
- CI: Add GitHub CodeQL analysis workflow (
codeql.yml) by @pankajkoti in #497 - Fix deploy pages job missing credentials by @pankajkoti in #499
- Add the breaking changes to changelog by @pankajastro in #502
- Add pre-commit to update
uv.lockby @pankajastro in #514 - Remove
clean_dagsusage from object storage DAG by @pankajastro in #515 - Remove broad exceptions and catch more specific exceptions by @pankajastro in #519
- Add missing env in contributing doc by @pankajastro in #522
- Enhance PyPI Stats API error handling by @viiccwen in #535
- Update example to be Airflow 3 compatible by @tatiana in 540
- Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in #543
- Bump actions/download-artifact from 4 to 5 by @dependabot in #548
v1.0.0a1
[1.0.0a1] - 2025-08-01
Breaking Changes
- Airflow providers are now optional dependencies by @pankajastro in #486
- Previously,
dag-factoryenforced the installation ofapache-airflow-providers-httpandapache-airflow-providers-cncf-kubernetes. These Airflow providers dependencies are now optional. If your DAGs depend on these providers, you must install them manually. Alternatively, you can installdag-factorywith extras likedag-factory[all],dag-factory[kubernetes], etc.
- Previously,
- Removed
clean_dagsfunction by @pankajastro in #498- You no longer need to call
example_dag_factory.clean_dags(globals())in your DAG files. DAG cleanup is now controlled via the Airflow config settingAIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL.
- You no longer need to call
- Remove Inconsistent Parameters for Airflow Consistent by @pankajastro in #512
- Removed
dagrun_timeout_secfrom dag param. - Removed
retry_delay_sec,sla_secsfrom default_args. - Removed accepting
execution_timeoutas integer. - Removed
execution_timeout_secs,sla_secsandexecution_delta_secsfrom task param.
- Removed
- Remove custom parsing for Kubernetes object and refactor KPO to use
__type__syntax by @pankajastro in #523- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
__type__syntax to supply Kubernetes object in your YAML DAG. For an example KPO configuration, visit: KubernetesPodOperator Documentation.
- The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use
Added
- Support dag-level arguments in global defaults by @gyli in #480
- Support
*argsin custom Python object by @pankajastro in #484 - Support tasks and task_groups as lists by @pankajkoti in #487
- Support overriding
defaults.ymlbased on the directory hierarchy by @tatiana in #500 - Introduced DAG Factory CLI by @tatiana in #510
- Added
lintcommand to CLI by @tatiana in #513
Fixed
- Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by @pankajastro in #485
- Ensure
dag_paramscontainschedulebefore operating on it by @pankajkoti in #488 - Fix
start_date,end_dateat the DAG level by @pankajastro in #495 - Allow
execution_timeoutindefault_argsby @pankajastro in #501
Docs
- Restore basic DAG example by @pankajastro in #483
- Replace the usages in example dags, tests and docs for tasks and taskgroups to be list by @pankajkoti in #492
- Update default documentation based on #500 by @tatiana in #504
- Add more examples for Custom Python object by @pankajastro in #506
- Add documentation for DAG Factory CLI by @tatiana in #511
- Add documentation and example YAMLs for task and task_group configuration formats by @pankajkoti in #530
Other Changes
- Improve unit tests to disregard
$AIRFLOW_HOMEby @tatiana in #490 - Resolve unpinned action reference error alerts raised by Zizmor by @pankajkoti in #493
- Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #494
- Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #496
- CI: Add GitHub CodeQL analysis workflow (
codeql.yml) by @pankajkoti in #497 - Fix deploy pages job missing credentials by @pankajkoti in #499
- Add the breaking changes to changelog by @pankajastro in #502
- Remove
clean_dagsusage from object storage DAG by @pankajastro in #515 - Add pre-commit to update
uv.lockby @pankajastro in #514 - Remove broad exceptions and catch more specific exceptions by @pankajastro in #519
- Add missing env in contributing doc by @pankajastro in #522
v0.23.0
[0.23.0] - 2025-07-14
Breaking Change
- Drop Airflow 2.2 Support by @pankajastro in #388
- Drop Python 3.8 support by @pankajastro in #435
- Drop Airflow 2.3 Support by @pankajastro in #456
Airflow 3 Support
- Add tools for Dag-factory Airflow3 testing by @pankajastro in #395
- Fix schedule args for Airflow 3 by @pankajastro in #413
- Fix import path for BranchPythonOperator, PythonOperator and PythonSensor by @pankajastro in #414
- Add scheduling docs for Airflow 3 by @pankajastro in #424
- Enable Airflow 3 tests in CI by @pankajastro in #436
- Add env AUTO_CONVERT_TO_AF3 in Dockerfile by @pankajastro in #455
- Validate DAG's on Airflow 3 by @pankajastro in #457
- Refactor schedule to use the Python object @pankajastro in #458
- Fix CI and import issues for Airflow 3 compatibility @pankajastro in #463
Added
- Add support for defining inlets by @IvanSviridov in #380
- Add HttpOperator JSON serialization support with tests by @a-chumagin in #382
- Add support for custom Python object by @pankajastro in #444
- Support env var in default by @gyli in #452
- Pass default arguments via dictionary in .py file by @jroachgolf84 in #465
Fixed
- Remediated
defaultbehavior, added documentation by @jroachgolf84 in #378 - Upgrade
apache-airflow-providers-cncf-kubernetesprovider by @pankajastro in #407 - Include error message trace in exception by @pankajastro in #408
Docs
- Remove Unreleased heading section from the CHANGELOG.md by @pankajkoti in #365
- Add Documentation for Conditional Dataset Scheduling with dag-factory by @ErickSeo in #367
- Add copy right in docs footer by @pankajastro in #371
- Updating docs for callbacks by @jroachgolf84 in #375
- Add stable/latest version in docs by @pankajastro in #391
- Migrate old content to new documentation structure by @pankajastro in #393
- Update Airflow supported version 2.3+ in docs by @pankajastro in #412
- Doc: Add step to fork repo in contributing guide by @pankajastro in #427
- Add setting CONFIG_ROOT_DIR in the contribution doc by @gyli in #432
- Add DAG example showcasing runtime params usage by @pankajastro in #449
- Add jinja2 template example by @pankajastro in #450
Others
- Add Scraf Pixels for telemetry by @pankajastro in #373
- feat: bumped http provider versions to 2.0+ by @a-chumagin in #389
- Add --verbosity debug in astro-cli cmd by @pankajastro in #390
- Add missing Python file for dynamic task example by @pankajastro in #392
- Pin apache-airflow-providers-cncf-kubernetes<10.4.2 by @pankajastro in #400
- Add script to check version and tag by @pankajastro in #395
- Assert DagRunState in integration test by @pankajastro in #415
- Move to uv for package management by @jlaneve in #419
- Install uv in CI by @jlaneve in #421
- Bump astral-sh/setup-uv from 5 to 6 by @dependabot in #423
- Add Airflow 2.11 in test matrix by @pankajastro in #425
- fix example_dag_factory.yml typo causing catchup: false to not be respected by @RNHTTR in #431
- Remove expandvars in utils.get_python_callable by @gyli in #440
- Delete unused img folder by @pankajastro in #446
- Clean print statement by @pankajastro #447
- Add hatch to uv dev dependencies by @gyli in #453
- Add Authorize Job in CI by @pankajastro in #460
- Remove PyPI token for releasing packages by @tatiana in #461
- Change CI on trigger event to pull_request from pull_request_target by @pankajkoti in #464
- Update cicd.yaml: Use pull_request for authorize as we don't have pull_request_target event configured by @pankajkoti in #466
- Add environment for pypi publish job by @pankajastro in #478
v0.23.0a9
v0.23.0a10
[0.23.0a10] - 2025-07-11
Breaking Change
- Drop Airflow 2.2 Support by @pankajastro in #388
- Drop Python 3.8 support by @pankajastro in #435
- Drop Airflow 2.3 Support by @pankajastro in #456
Airflow 3 Support
- Add tools for Dag-factory Airflow3 testing by @pankajastro in #395
- Fix schedule args for Airflow 3 by @pankajastro in #413
- Fix import path for BranchPythonOperator, PythonOperator and PythonSensor by @pankajastro in #414
- Add scheduling docs for Airflow 3 by @pankajastro in #424
- Enable Airflow 3 tests in CI by @pankajastro in #436
- Add env AUTO_CONVERT_TO_AF3 in Dockerfile by @pankajastro in #455
- Validate DAG's on Airflow 3 by @pankajastro in #457
- Refactor schedule to use the Python object @pankajastro in #458
- Fix CI and import issues for Airflow 3 compatibility @pankajastro in #463
Added
- Add support for defining inlets by @IvanSviridov in #380
- Add HttpOperator JSON serialization support with tests by @a-chumagin in #382
- Add support for custom Python object by @pankajastro in #444
- Support env var in default by @gyli in #452
- Pass default arguments via dictionary in .py file by @jroachgolf84 in #465
Fixed
- Remediated
defaultbehavior, added documentation by @jroachgolf84 in #378 - Upgrade
apache-airflow-providers-cncf-kubernetesprovider by @pankajastro in #407 - Include error message trace in exception by @pankajastro in #408
Docs
- Remove Unreleased heading section from the CHANGELOG.md by @pankajkoti in #365
- Add Documentation for Conditional Dataset Scheduling with dag-factory by @ErickSeo in #367
- Add copy right in docs footer by @pankajastro in #371
- Updating docs for callbacks by @jroachgolf84 in #375
- Add stable/latest version in docs by @pankajastro in #391
- Migrate old content to new documentation structure by @pankajastro in #393
- Update Airflow supported version 2.3+ in docs by @pankajastro in #412
- Doc: Add step to fork repo in contributing guide by @pankajastro in #427
- Add setting CONFIG_ROOT_DIR in the contribution doc by @gyli in #432
- Add DAG example showcasing runtime params usage by @pankajastro in #449
- Add jinja2 template example by @pankajastro in #450
Others
- Add Scraf Pixels for telemetry by @pankajastro in #373
- feat: bumped http provider versions to 2.0+ by @a-chumagin in #389
- Add --verbosity debug in astro-cli cmd by @pankajastro in #390
- Add missing Python file for dynamic task example by @pankajastro in #392
- Pin apache-airflow-providers-cncf-kubernetes<10.4.2 by @pankajastro in #400
- Add script to check version and tag by @pankajastro in #395
- Assert DagRunState in integration test by @pankajastro in #415
- Move to uv for package management by @jlaneve in #419
- Install uv in CI by @jlaneve in #421
- Bump astral-sh/setup-uv from 5 to 6 by @dependabot in #423
- Add Airflow 2.11 in test matrix by @pankajastro in #425
- fix example_dag_factory.yml typo causing catchup: false to not be respected by @RNHTTR in #431
- Remove expandvars in utils.get_python_callable by @gyli in #440
- Delete unused img folder by @pankajastro in #446
- Clean print statement by @pankajastro #447
- Add hatch to uv dev dependencies by @gyli in #453
- Add Authorize Job in CI by @pankajastro in #460
- Remove PyPI token for releasing packages by @tatiana in #461
- Change CI on trigger event to pull_request from pull_request_target by @pankajkoti in #464
- Update cicd.yaml: Use pull_request for authorize as we don't have pull_request_target event configured by @pankajkoti in #466
- Add environment for pypi publish job by @pankajastro in #478
v0.23.0a8
[0.23.0a8] - 2025-01-07
Add support for custom Python object by @pankajastro in #444