-
Notifications
You must be signed in to change notification settings - Fork 3.3k
HBASE-29272 When Spark reads an HBase snapshot, it always read empty … #6947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
...-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
Show resolved
Hide resolved
a96d09e
to
5e35c8d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
5e35c8d
to
79c6087
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
||
// constructor for mapreduce framework / Writable | ||
public InputSplit() { | ||
} | ||
|
||
public InputSplit(TableDescriptor htd, RegionInfo regionInfo, List<String> locations, Scan scan, | ||
Path restoreDir) { | ||
this(htd, regionInfo, locations, scan, restoreDir, 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't seem quite right in here, because SnapShotStats.getStoreFilesSize() would return 0 if the table has no any data.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reviewing 😃 , it shouldn't always be 1 here, let me try to fix it..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we still want to keep this constructor? The parent class is IA.Private, which means we are free to change anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let me try to remove it..
SnapshotStats(final Configuration conf, final FileSystem fs, final SnapshotManifest mainfest) | ||
throws CorruptedSnapshotException { | ||
this.snapshot = SnapshotDescriptionUtils.readSnapshotInfo(fs, mainfest.getSnapshotDir()); | ||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
/** | ||
* Utility class to calculate the size of each region in a snapshot. | ||
*/ | ||
public class SnapshotRegionSizeCalculator { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add IA annotation for this class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added it.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
0b34b27
to
e45f5fc
Compare
…data. HBASE-29272 When Spark reads an HBase snapshot, it always read empty data.
e45f5fc
to
5f5ee39
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…data. HBASE-29272 When Spark reads an HBase snapshot, it always read empty data.
a4bf605
to
9c3d569
Compare
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
Hi @guluo2016 and @Apache9 , could you help review this pr again? 🙏 thanks |
Fix the issue that after Spark 3.2.0, when Spark reads an HBase snapshot, it always read empty, even if the hbase snapshot actually has data.