HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) #6950

LiuGuH · 2024-07-19T09:26:04Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-2139

This is used for compile to verify this is correct. It is for test case . And in the futuer this will split to subtask submission.

It support now:
(1) hdfs dfs -fastcp
(2) Distcp with fastcp

How was this patch tested?

Add test case

hadoop-yetus · 2024-07-19T18:05:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 27s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 9s		trunk passed
+1 💚	compile	16m 45s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	15m 21s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 25s		trunk passed
+1 💚	mvnsite	5m 4s		trunk passed
+1 💚	javadoc	3m 54s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 19s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	9m 29s		trunk passed
+1 💚	shadedclient	33m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 21s		the patch passed
+1 💚	compile	16m 23s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	cc	16m 23s		the patch passed
+1 💚	javac	16m 23s		the patch passed
+1 💚	compile	15m 37s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	cc	15m 37s		the patch passed
+1 💚	javac	15m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 14s	/results-checkstyle-root.txt	root: The patch generated 47 new + 1030 unchanged - 6 fixed = 1077 total (was 1036)
+1 💚	mvnsite	4m 59s		the patch passed
+1 💚	javadoc	3m 56s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 17s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	10m 23s		the patch passed
+1 💚	shadedclient	33m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	19m 21s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	unit	2m 46s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	222m 25s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	29m 47s		hadoop-distcp in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		517m 44s

Reason	Tests
Failed junit tests	hadoop.fs.TestHarFileSystem
	hadoop.fs.TestFilterFileSystem
	hadoop.hdfs.server.namenode.TestFSNamesystemLockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/1/artifact/out/Dockerfile
GITHUB PR	#6950
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat xmllint
uname	Linux 65eed2f25d58 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6cbf7c9`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/1/testReport/
Max. process+thread count	3470 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zeekling

We already has hadoop distcp, Why do we need hdfs dfs -distcp?

zeekling · 2024-07-20T16:20:59Z

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

@@ -2544,6 +2544,10 @@ public void moveFromLocalFile(Path src, Path dst)
    copyFromLocalFile(true, src, dst);
  }

+  public void fastCopy(Path src, Path dst, boolean overwrite) throws IOException {
+


Why empty method ?

hadoop-yetus · 2024-07-22T16:59:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	33m 8s		trunk passed
+1 💚	compile	16m 55s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	15m 34s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 22s		trunk passed
+1 💚	mvnsite	5m 9s		trunk passed
+1 💚	javadoc	4m 0s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 18s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	9m 36s		trunk passed
+1 💚	shadedclient	33m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 13s		the patch passed
+1 💚	compile	16m 26s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	cc	16m 26s		the patch passed
+1 💚	javac	16m 26s		the patch passed
+1 💚	compile	15m 41s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	cc	15m 41s		the patch passed
+1 💚	javac	15m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 18s	/results-checkstyle-root.txt	root: The patch generated 49 new + 1205 unchanged - 6 fixed = 1254 total (was 1211)
+1 💚	mvnsite	5m 6s		the patch passed
+1 💚	javadoc	3m 59s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 20s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	10m 23s		the patch passed
+1 💚	shadedclient	34m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 23s		hadoop-common in the patch passed.
+1 💚	unit	2m 43s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	220m 27s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	29m 43s		hadoop-distcp in the patch passed.
+1 💚	asflicense	1m 7s		The patch does not generate ASF License warnings.
		529m 15s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestFSNamesystemLockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/2/artifact/out/Dockerfile
GITHUB PR	#6950
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat xmllint
uname	Linux a24adcfdc185 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3bad2c1`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/2/testReport/
Max. process+thread count	3264 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-07-23T11:21:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 53s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 32s		trunk passed
+1 💚	compile	17m 35s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	16m 5s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 34s		trunk passed
+1 💚	mvnsite	5m 10s		trunk passed
+1 💚	javadoc	3m 54s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 18s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	9m 44s		trunk passed
+1 💚	shadedclient	36m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 22s		the patch passed
+1 💚	compile	17m 4s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	cc	17m 4s		the patch passed
+1 💚	javac	17m 4s		the patch passed
+1 💚	compile	15m 56s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	cc	15m 56s		the patch passed
+1 💚	javac	15m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 39s	/results-checkstyle-root.txt	root: The patch generated 49 new + 1205 unchanged - 6 fixed = 1254 total (was 1211)
+1 💚	mvnsite	5m 1s		the patch passed
+1 💚	javadoc	3m 51s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 14s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	10m 21s		the patch passed
+1 💚	shadedclient	35m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 20s		hadoop-common in the patch passed.
+1 💚	unit	2m 42s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	217m 35s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	29m 58s		hadoop-distcp in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		521m 21s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestFSNamesystemLockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/3/artifact/out/Dockerfile
GITHUB PR	#6950
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat xmllint
uname	Linux 6918e2d2e0f0 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `db19346`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/3/testReport/
Max. process+thread count	3889 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…out federation.

hadoop-yetus · 2024-07-26T18:07:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 28s		Maven dependency ordering for branch
-1 ❌	mvninstall	32m 54s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	17m 13s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	18m 11s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 46s		trunk passed
+1 💚	mvnsite	5m 3s		trunk passed
+1 💚	javadoc	3m 35s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	3m 50s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	8m 53s		trunk passed
+1 💚	shadedclient	36m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 30s		the patch passed
+1 💚	compile	17m 15s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	cc	17m 15s		the patch passed
+1 💚	javac	17m 15s		the patch passed
+1 💚	compile	15m 31s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	cc	15m 31s		the patch passed
+1 💚	javac	15m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 27s	/results-checkstyle-root.txt	root: The patch generated 49 new + 1205 unchanged - 6 fixed = 1254 total (was 1211)
+1 💚	mvnsite	5m 10s		the patch passed
+1 💚	javadoc	3m 46s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	4m 17s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	11m 3s		the patch passed
+1 💚	shadedclient	40m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 53s		hadoop-common in the patch passed.
+1 💚	unit	2m 47s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	222m 28s		hadoop-hdfs in the patch passed.
+1 💚	unit	30m 38s		hadoop-distcp in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		533m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/4/artifact/out/Dockerfile
GITHUB PR	#6950
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat xmllint
uname	Linux eb29cb108240 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `205d342`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/4/testReport/
Max. process+thread count	3245 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-08-09T19:36:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 3s		trunk passed
+1 💚	compile	16m 46s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	15m 38s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 18s		trunk passed
+1 💚	mvnsite	5m 8s		trunk passed
+1 💚	javadoc	4m 3s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 23s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	9m 35s		trunk passed
+1 💚	shadedclient	34m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 18s		the patch passed
+1 💚	compile	16m 20s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	cc	16m 20s		the patch passed
+1 💚	javac	16m 20s		the patch passed
+1 💚	compile	15m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	cc	15m 42s		the patch passed
+1 💚	javac	15m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 15s	/results-checkstyle-root.txt	root: The patch generated 53 new + 1205 unchanged - 6 fixed = 1258 total (was 1211)
+1 💚	mvnsite	5m 7s		the patch passed
+1 💚	javadoc	3m 57s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	10m 18s		the patch passed
+1 💚	shadedclient	34m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 23s		hadoop-common in the patch passed.
+1 💚	unit	2m 41s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	218m 2s		hadoop-hdfs in the patch passed.
+1 💚	unit	29m 16s		hadoop-distcp in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		525m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/5/artifact/out/Dockerfile
GITHUB PR	#6950
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat xmllint
uname	Linux fe34b1067a1f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e5022fe`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/5/testReport/
Max. process+thread count	3814 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6950/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomscut · 2024-12-26T05:59:33Z

...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java

-        Token<BlockTokenIdentifier> accessToken = getBlockAccessToken(b,
-            EnumSet.of(BlockTokenIdentifier.AccessMode.WRITE),
-            targetStorageTypes, targetStorageIds);
+        Token<BlockTokenIdentifier> accessToken =


There's a lot of code formatting here.

github-actions bot added HDFS DistCp trunk Common TOOLS labels Jul 19, 2024

LiuGuH changed the title ~~FastCopy~~ HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) Jul 19, 2024

zeekling reviewed Jul 21, 2024

View reviewed changes

LiuGuH force-pushed the trunk-fastcopy branch from 6cbf7c9 to 3bad2c1 Compare July 22, 2024 08:09

LiuGuH added 3 commits July 26, 2024 17:08

HDFS-16757. Add a new method copyBlockCrossNamespace to DataNode

3e2e2c3

HDFS-17581. Add FastCopy tool and support dfs -fastcp command

3af0f21

HDFS-17582. Distcp support fastcopy

205d342

LiuGuH force-pushed the trunk-fastcopy branch from db19346 to 205d342 Compare July 26, 2024 09:12

HDFS-17592. FastCopy support data copy in different nameservices with…

e5022fe

…out federation.

tomscut reviewed Dec 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) #6950

HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) #6950

LiuGuH commented Jul 19, 2024

hadoop-yetus commented Jul 19, 2024

zeekling left a comment

zeekling Jul 20, 2024

hadoop-yetus commented Jul 22, 2024

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 26, 2024

hadoop-yetus commented Aug 9, 2024

tomscut Dec 26, 2024

HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) #6950

Are you sure you want to change the base?

HDFS-2139. Fast copy for HDFS.FastCopy (This is used for test case . And in the futuer this will split to subtask submission. ) #6950

Conversation

LiuGuH commented Jul 19, 2024

Description of PR

How was this patch tested?

hadoop-yetus commented Jul 19, 2024

zeekling left a comment

Choose a reason for hiding this comment

zeekling Jul 20, 2024

Choose a reason for hiding this comment

hadoop-yetus commented Jul 22, 2024

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 26, 2024

hadoop-yetus commented Aug 9, 2024

tomscut Dec 26, 2024

Choose a reason for hiding this comment