Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark, API: Enhance hashing efficiency by operating on raw UTF-8 bytes #12657

Merged
merged 3 commits into from
Mar 31, 2025
+25 −4
Merged

Spark, API: Enhance hashing efficiency by operating on raw UTF-8 bytes #12657

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d978fbc
Spark, API: Enhance hashing efficiency by operating on raw UTF-8 bytes
xiaoxuandev Mar 26, 2025
e3dfb93
Add tests to verify that hash string and hash bytes produce the same …
xiaoxuandev Mar 27, 2025
dc2775e
keep original assertion same as before
xiaoxuandev Mar 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions api/src/main/java/org/apache/iceberg/util/BucketUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ public static int hash(CharSequence value) {
return MURMUR3.hashString(value, StandardCharsets.UTF_8).asInt();
}

public static int hash(byte[] value) {
return MURMUR3.hashBytes(value).asInt();
}

public static int hash(ByteBuffer value) {
if (value.hasArray()) {
return MURMUR3
Expand Down
8 changes: 6 additions & 2 deletions spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/functions/BucketFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,12 @@ public static Integer invoke(int numBuckets, UTF8String value) {
return null;
}

// TODO - We can probably hash the bytes directly given they're already UTF-8 input.
return apply(numBuckets, hash(value.toString()));
return apply(numBuckets, hash(value.getBytes()));
}

// Visible for testing
public static int hash(byte[] value) {
return BucketUtil.hash(value);
}

// Visible for testing
Expand Down
7 changes: 6 additions & 1 deletion spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkBucketFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,12 @@ public void testSpecValues() {
Assert.assertEquals(
"Spec example: hash(\"iceberg\") = 1210000089",
1210000089,
new BucketFunction.BucketString().hash("iceberg"));
new BucketFunction.BucketString().hash("iceberg".getBytes(StandardCharsets.UTF_8)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We could just leave this the same as before just to keep the diff smaller.


Assert.assertEquals(
"Verify that the hash string and hash raw bytes produce the same result",
new BucketFunction.BucketString().hash("iceberg"),
new BucketFunction.BucketString().hash("iceberg".getBytes(StandardCharsets.UTF_8)));

ByteBuffer bytes = ByteBuffer.wrap(new byte[] {0, 1, 2, 3});
Assert.assertEquals(
Expand Down
8 changes: 6 additions & 2 deletions spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/functions/BucketFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,12 @@ public static Integer invoke(int numBuckets, UTF8String value) {
return null;
}

// TODO - We can probably hash the bytes directly given they're already UTF-8 input.
return apply(numBuckets, hash(value.toString()));
return apply(numBuckets, hash(value.getBytes()));
}

// Visible for testing
public static int hash(byte[] value) {
return BucketUtil.hash(value);
}

// Visible for testing
Expand Down
6 changes: 5 additions & 1 deletion spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkBucketFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,14 @@ public void testSpecValues() {
.as("Spec example: hash(2017-11-16T22:31:08) = -2047944441")
.isEqualTo(-2047944441);

assertThat(new BucketFunction.BucketString().hash("iceberg"))
assertThat(new BucketFunction.BucketString().hash("iceberg".getBytes(StandardCharsets.UTF_8)))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Same as above, we could just leave this assertion the same as earlier just to keep the diff smaller.

.as("Spec example: hash(\"iceberg\") = 1210000089")
.isEqualTo(1210000089);

assertThat(new BucketFunction.BucketString().hash("iceberg".getBytes(StandardCharsets.UTF_8)))
.as("Verify that the hash string and hash raw bytes produce the same result")
.isEqualTo(new BucketFunction.BucketString().hash("iceberg"));

ByteBuffer bytes = ByteBuffer.wrap(new byte[] {0, 1, 2, 3});
assertThat(new BucketFunction.BucketBinary().hash(bytes))
.as("Spec example: hash([00 01 02 03]) = -188683207")
Expand Down