Skip to content

Comments

HADOOP-19802. [ABFS] Removing azure storage dependency from hadoop-azure code.#8229

Open
anmolanmol1234 wants to merge 3 commits intoapache:trunkfrom
ABFSDriver:HADOOP-19802
Open

HADOOP-19802. [ABFS] Removing azure storage dependency from hadoop-azure code.#8229
anmolanmol1234 wants to merge 3 commits intoapache:trunkfrom
ABFSDriver:HADOOP-19802

Conversation

@anmolanmol1234
Copy link
Contributor

@anmolanmol1234 anmolanmol1234 commented Feb 5, 2026

This PR introduces ABFS REST implementations for ListContainers and DeleteContainer, eliminating the dependency on the Azure Storage SDK in the hadoop-azure module for these operations.

A separate PR addresses the removal of WASB support — #8207
. Once that change is merged, the azure-storage dependency can be safely removed from the hadoop-azure pom.xml, completing the SDK deprecation effort.

@anmolanmol1234 anmolanmol1234 marked this pull request as ready for review February 5, 2026 11:45
@hadoop-yetus
Copy link

hadoop-yetus commented Feb 5, 2026

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 20m 14s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 49m 14s trunk passed
+1 💚 compile 1m 4s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 1m 8s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 1m 7s trunk passed
+1 💚 mvnsite 1m 11s trunk passed
+1 💚 javadoc 1m 6s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 1m 3s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 1m 46s trunk passed
+1 💚 shadedclient 31m 1s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 31m 31s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 38s the patch passed
+1 💚 compile 0m 35s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 0m 35s the patch passed
+1 💚 compile 0m 38s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 0m 38s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 30s the patch passed
+1 💚 mvnsite 0m 47s the patch passed
+1 💚 javadoc 0m 31s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 0m 30s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 1m 22s the patch passed
+1 💚 shadedclient 28m 28s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 3m 5s hadoop-azure in the patch passed.
-1 ❌ asflicense 0m 36s /results-asflicense.txt The patch generated 1 ASF License warnings.
148m 51s
Subsystem Report/Notes
Docker ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/1/artifact/out/Dockerfile
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 1827ffb1da0a 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8222ac0
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/1/testReport/
Max. process+thread count 633 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/1/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Feb 5, 2026

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 22s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 47m 34s trunk passed
+1 💚 compile 1m 1s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 0m 59s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 0m 59s trunk passed
+1 💚 mvnsite 1m 10s trunk passed
+1 💚 javadoc 1m 0s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 0m 58s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 1m 38s trunk passed
+1 💚 shadedclient 28m 52s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 29m 23s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 37s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 0m 36s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 0m 36s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 29s the patch passed
+1 💚 mvnsite 0m 41s the patch passed
+1 💚 javadoc 0m 31s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 0m 32s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 1m 20s the patch passed
+1 💚 shadedclient 27m 28s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 3m 7s hadoop-azure in the patch passed.
-1 ❌ asflicense 0m 36s /results-asflicense.txt The patch generated 1 ASF License warnings.
124m 44s
Subsystem Report/Notes
Docker ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/2/artifact/out/Dockerfile
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 5b19f5367e49 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8222ac0
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/2/testReport/
Max. process+thread count 608 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8229/2/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@anmolanmol1234
Copy link
Contributor Author

anmolanmol1234 commented Feb 5, 2026

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 908, Failures: 0, Errors: 0, Skipped: 222
[WARNING] Tests run: 159, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 911, Failures: 0, Errors: 0, Skipped: 168
[WARNING] Tests run: 159, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 908, Failures: 0, Errors: 0, Skipped: 233
[WARNING] Tests run: 136, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus.testEmptyListingInSubsequentCall -- Time elapsed: 1.379 s <<< ERROR!

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 758, Failures: 0, Errors: 0, Skipped: 144
[ERROR] Tests run: 159, Failures: 0, Errors: 1, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 755, Failures: 0, Errors: 0, Skipped: 156
[WARNING] Tests run: 159, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 750, Failures: 0, Errors: 0, Skipped: 202
[WARNING] Tests run: 136, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 782, Failures: 0, Errors: 0, Skipped: 231
[WARNING] Tests run: 159, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

@anmolanmol1234
Copy link
Contributor Author

anmolanmol1234 commented Feb 19, 2026

Hi @steveloughran requesting your review on this PR as this can help to remove azure sdk dependency from 3.5.0 release

steveloughran
steveloughran approved these changes Feb 19, 2026
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
if you are happy with it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees

No one assigned

Labels

ABFS TOOLS trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@anmolanmol1234 @hadoop-yetus @steveloughran