Task-level filtering for backfill in Airflow 3

[ginjan-dev]

Summary

In Airflow 2, the airflow dags backfill command supported the -t TASK_REGEX flag, which allowed filtering specific task_ids to backfill. When combined with options like -i (--ignore-dependencies), this made it possible to run only selected tasks while skipping upstream dependencies.

In Airflow 3, this task-level backfill granularity appears to be missing.

While Airflow 3 still provides airflow tasks clear --task-regex for re-running existing task instances, this only works when task instances already exist. It does not help in cases where a new task is added to an existing DAG and needs to be backfilled for historical execution dates.

Use case

• A new task is introduced into an existing Airflow 3 DAG.
• Historical runs need to be backfilled only for that specific task.
• Currently, there is no straightforward way to achieve this without triggering full DAG backfills.

Suggestion:

Reintroduce task-level backfill functionality to allow selectively backfilling specific tasks within a DAG.

[cookesan]

This looks like a real Airflow 3 feature gap, not a hidden flag.

The Airflow 3 backfill path is centered on creating DagRuns for a DAG and date range. I do not see an equivalent of the old task regex flag on airflow backfill create. If the task instances already exist, airflow tasks clear can target selected tasks over a date range, but that is not the same as creating a new task-filtered backfill.

For now the practical options are to run a full backfill with the unwanted work guarded/short-circuited, use a temporary DAG version for the migration/backfill, or clear selected existing task instances when the DagRuns already exist. A PR for task-scoped backfill would need to define the scheduler semantics carefully, because creating only part of a DAG run interacts with dependencies, mapped tasks, assets, and downstream state.