HBASE-29272 When Spark reads an HBase snapshot, it always read empty … #6947

terrytlu · 2025-04-25T10:49:46Z

Fix the issue that after Spark 3.2.0, when Spark reads an HBase snapshot, it always read empty, even if the hbase snapshot actually has data.

...-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java

guluo2016 · 2025-05-07T14:08:37Z

...-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java


    // constructor for mapreduce framework / Writable
    public InputSplit() {
    }

    public InputSplit(TableDescriptor htd, RegionInfo regionInfo, List<String> locations, Scan scan,
      Path restoreDir) {
+      this(htd, regionInfo, locations, scan, restoreDir, 1);


This doesn't seem quite right in here, because SnapShotStats.getStoreFilesSize() would return 0 if the table has no any data.
What do you think?

Thanks for reviewing 😃 , it shouldn't always be 1 here, let me try to fix it..

Do we still want to keep this constructor? The parent class is IA.Private, which means we are free to change anything.

let me try to remove it..

Apache9 · 2025-05-22T13:54:42Z

hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java

+    SnapshotStats(final Configuration conf, final FileSystem fs, final SnapshotManifest mainfest)
+      throws CorruptedSnapshotException {
+      this.snapshot = SnapshotDescriptionUtils.readSnapshotInfo(fs, mainfest.getSnapshotDir());
+      ;


Remove this?

hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotRegionSizeCalculator.java

…data.

…data. HBASE-29272 When Spark reads an HBase snapshot, it always read empty data.

Apache-HBase · 2025-05-28T11:32:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 23s		master passed
+1 💚	compile	3m 45s		master passed
+1 💚	checkstyle	0m 49s		master passed
+1 💚	spotbugs	2m 5s		master passed
+1 💚	spotless	0m 51s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	3m 44s		the patch passed
+1 💚	javac	3m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 47s		the patch passed
+1 💚	spotbugs	2m 19s		the patch passed
+1 💚	hadoopcheck	12m 14s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 18s		The patch does not generate ASF License warnings.
		42m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6947/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6947
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 1b6c803bc06b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9c3d569`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6947/9/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-28T15:08:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 14s		master passed
+1 💚	compile	1m 17s		master passed
+1 💚	javadoc	0m 41s		master passed
+1 💚	shadedjars	6m 4s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	1m 17s		the patch passed
+1 💚	javac	1m 17s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed
+1 💚	shadedjars	6m 3s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	210m 29s		hbase-server in the patch passed.
+1 💚	unit	20m 27s		hbase-mapreduce in the patch passed.
		258m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6947/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6947
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 2b2a5b6ed9c5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9c3d569`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6947/9/testReport/
Max. process+thread count	4831 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6947/9/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

terrytlu · 2025-06-04T07:53:26Z

Hi @guluo2016 and @Apache9 , could you help review this pr again? 🙏 thanks

This comment has been minimized.

Sign in to view

guluo2016 reviewed Apr 27, 2025

View reviewed changes

...-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java Show resolved Hide resolved

terrytlu force-pushed the master-HBASE-29272 branch 3 times, most recently from a96d09e to 5e35c8d Compare April 29, 2025 13:17

This comment has been minimized.

Sign in to view

terrytlu force-pushed the master-HBASE-29272 branch from 5e35c8d to 79c6087 Compare May 7, 2025 08:13

This comment has been minimized.

Sign in to view

guluo2016 reviewed May 7, 2025

View reviewed changes

Apache9 reviewed May 22, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

terrytlu added 2 commits May 27, 2025 14:40

HBASE-29272 When Spark reads an HBase snapshot, it always read empty …

93133af

…data.

HBASE-29272 When Spark reads an HBase snapshot, it always read empty …

c9825db

…data.

terrytlu force-pushed the master-HBASE-29272 branch from 0b34b27 to e45f5fc Compare May 27, 2025 06:41

HBASE-29272 When Spark reads an HBase snapshot, it always read empty …

5f5ee39

…data. HBASE-29272 When Spark reads an HBase snapshot, it always read empty data.

terrytlu force-pushed the master-HBASE-29272 branch from e45f5fc to 5f5ee39 Compare May 27, 2025 06:50

This comment has been minimized.

Sign in to view

HBASE-29272 When Spark reads an HBase snapshot, it always read empty …

9c3d569

…data. HBASE-29272 When Spark reads an HBase snapshot, it always read empty data.

terrytlu force-pushed the master-HBASE-29272 branch from a4bf605 to 9c3d569 Compare May 28, 2025 08:22

HBASE-29272 When Spark reads an HBase snapshot, it always read empty … #6947

Are you sure you want to change the base?

HBASE-29272 When Spark reads an HBase snapshot, it always read empty … #6947

Uh oh!

Conversation

terrytlu commented Apr 25, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

guluo2016 May 7, 2025

Choose a reason for hiding this comment

Uh oh!

terrytlu May 8, 2025

Choose a reason for hiding this comment

Uh oh!

Apache9 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

terrytlu May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Apache9 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

terrytlu May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented May 28, 2025

Uh oh!

Apache-HBase commented May 28, 2025

Uh oh!

terrytlu commented Jun 4, 2025

Uh oh!

Uh oh!