Skip to content

Conversation

@nailo2c
Copy link
Contributor

@nailo2c nailo2c commented Jan 28, 2026

closes: #44228

Why

The older ADLSListOperator uses AzureDataLakeHook, which uses Gen 1 SDK is already retired.

self._conn = core.AzureDLFileSystem(credential, store_name=self.account_name)

How

Replace it with AzureDataLakeStorageV2Hook, which uses Gen 2 SDK.

Given Gen1 is retired, the impact should be limited, but this is a breaking change.

What

I created an object (blob) in an Azure Storage account.

issue-44228-azure

And I used this DAG to test whether I could fetch it.

from datetime import datetime

from airflow import DAG
from airflow.providers.microsoft.azure.operators.adls import ADLSListOperator
from airflow.providers.standard.operators.python import PythonOperator

with DAG(
    dag_id="test_adls_issue_44228_fixed",
    start_date=datetime(2026, 1, 1),
    schedule=None,
    catchup=False,
) as dag:

    list_files = ADLSListOperator(
        task_id="list_adls_files",
        file_system_name="testcontainer",
        path="",
        azure_data_lake_conn_id="adls_gen2_default",
    )

    def print_files(ti):
        files = ti.xcom_pull(task_ids="list_adls_files")
        print("=" * 50)
        print(f"Files found: {files}")
        print(f"Total count: {len(files) if files else 0}")
        print("=" * 50)

    print_result = PythonOperator(
        task_id="print_result",
        python_callable=print_files,
    )

    list_files >> print_result

It works pretty well.
issue-44228-airflow-dag

Discussion

It seems AzureDataLakeHook uses the Gen 1 SDK. Perhaps we need to add @deprecated(...) to it?


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    Claude Opus 4.5

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@nailo2c
Copy link
Contributor Author

nailo2c commented Jan 29, 2026

Hmmm… it looks like ADLSToGCSOperator inherits from ADLSListOperator. Let me take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure Data Lake connection will not work for blob.core.windows.net domain

1 participant