Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-19471. ABFS: Support Fixed SAS Token at Container Level #7461

Draft
wants to merge 2 commits into
base: trunk
Choose a base branch
from

Conversation

manika137
Copy link
Contributor

Description of PR

The ABFS driver currently lacks support for multiple SAS tokens for the same storage account across different containers.
Introducing this support through this PR.

To use fixed SAS token at container level the configuration to be used is:
fs.azure.sas.fixed.token.<container-name>.<storage-account-name>

How was this patch tested?

Adding the test results in the comments below.

@manika137 manika137 marked this pull request as draft March 4, 2025 06:01
@manika137 manika137 changed the title HADOOP-19471. ABFS: Support Multiple SAS Tokens for Same Account HADOOP-19471. ABFS: Support Fixed SAS Token at Container Level Mar 4, 2025
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 46m 8s trunk passed
+1 💚 compile 0m 43s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 35s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 30s trunk passed
+1 💚 mvnsite 0m 40s trunk passed
+1 💚 javadoc 0m 40s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 32s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 9s trunk passed
+1 💚 shadedclient 38m 49s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 29s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 29s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 6 new + 4 unchanged - 0 fixed = 10 total (was 4)
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 30s /results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10)
-1 ❌ javadoc 0m 26s /results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10)
+1 💚 spotbugs 1m 9s the patch passed
+1 💚 shadedclient 40m 48s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 47s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
140m 41s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/1/artifact/out/Dockerfile
GITHUB PR #7461
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 1292e9229efb 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8c4413e
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/1/testReport/
Max. process+thread count 527 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@manika137 manika137 force-pushed the HADOOP-19471_SASTokenSupport branch from c5b14bb to 4a182ac Compare March 4, 2025 09:51
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 23s trunk passed
+1 💚 compile 0m 26s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 22s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 23s trunk passed
+1 💚 mvnsite 0m 25s trunk passed
+1 💚 javadoc 0m 28s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 47s trunk passed
+1 💚 shadedclient 19m 54s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 17s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 17s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 13s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4)
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 javadoc 0m 17s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 43s the patch passed
+1 💚 shadedclient 19m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 19s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 23s The patch does not generate ASF License warnings.
71m 48s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/3/artifact/out/Dockerfile
GITHUB PR #7461
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux a33d76a9617b 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4a182ac
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/3/testReport/
Max. process+thread count 671 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

passchars = rawConfig.getPassword(accountConf(key));
if(passchars == null){
passchars = rawConfig.getPassword(key);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplified as :- char[] passchars = rawConfig.getPassword(containerConf(key)) != null ?
rawConfig.getPassword(containerConf(key)) :
rawConfig.getPassword(accountConf(key)) != null ?
rawConfig.getPassword(accountConf(key)) :
rawConfig.getPassword(key);

@@ -493,7 +497,7 @@ public AbfsConfiguration(final Configuration rawConfig,
*/
public AbfsConfiguration(final Configuration rawConfig, String accountName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we should add a separate constructor with this new param

* Helper method to get the Fixed SAS token value
*/
private String getFixedSASToken(AbfsConfiguration config) throws Exception {
return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), "read");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use constants for string

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 41m 14s trunk passed
+1 💚 compile 0m 42s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 36s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 32s trunk passed
+1 💚 mvnsite 0m 41s trunk passed
+1 💚 javadoc 0m 43s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 8s trunk passed
+1 💚 shadedclient 40m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 30s the patch passed
+1 💚 compile 0m 35s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 35s the patch passed
+1 💚 compile 0m 28s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4)
+1 💚 mvnsite 0m 30s the patch passed
+1 💚 javadoc 0m 29s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 27s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 7s the patch passed
+1 💚 shadedclient 41m 36s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 47s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
137m 59s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/2/artifact/out/Dockerfile
GITHUB PR #7461
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 4be05fc1184a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c5b14bb
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/2/testReport/
Max. process+thread count 529 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7461/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@manika137
Copy link
Contributor Author

Test Results

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 771, Failures: 0, Errors: 0, Skipped: 147
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 774, Failures: 0, Errors: 0, Skipped: 115
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 613, Failures: 0, Errors: 0, Skipped: 199
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 771, Failures: 0, Errors: 0, Skipped: 152
[WARNING] Tests run: 124, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 616, Failures: 0, Errors: 0, Skipped: 142
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 610, Failures: 0, Errors: 0, Skipped: 201
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 613, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 611, Failures: 0, Errors: 0, Skipped: 161
[WARNING] Tests run: 124, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 645, Failures: 0, Errors: 0, Skipped: 150
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 610, Failures: 0, Errors: 0, Skipped: 199
[WARNING] Tests run: 147, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 264, Failures: 0, Errors: 0, Skipped: 24

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants