Spec: Allow the use of `source-id` in V3 #12644

Fokko · 2025-03-25T18:58:24Z

Changes around the multi-argument transforms, mainly two things:

Up for debate. The spec does not point out an actual implementations of transforms that accept multiple arguments. From the existing transforms, the only contender is the bucket transform. Should we include this in the V3 spec? It will only allow to prune metadata if you do an equality expression on all the fields that are part of the transform.
Along the way, we've removed something that we did not intend. First we allowed to write source-id and source-ids based on the number of arguments. This has been changed to only allow source-ids for V3 in a PR that introduces backward compatibility. I think this makes the JSON parsers/producers more complex than needed (specifically PyIceberg). Also, in Java we would need to plumb down the table version to the PartitionSpecParser.java. I think it would be great to simplify this.

RussellSpitzer · 2025-03-25T19:39:29Z

format/spec.md

@@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p

 | V1       | V2       | V3       | Field            | JSON representation | Example      |
 |----------|----------|----------|------------------|---------------------|--------------|
-| required | required | omitted  | **`source-id`**  | `JSON int`          | 1            |


Can't we still have a reader which reads both source-id and source-ids? Basically in our motto of reading potentially out of spec but writing in spec, we can always check if source-id is set as well as source-ids. The only time we need to break is if the two don't match.

What should the value be if source-ids is present?

These are implementation details but I would suggest if source-id and source-ids are both there
source-ids = list(source-id) - Fine on read, on write omit source-id
source-ids != list(source-id) - Error on read
source-id = x && source-ids == null - Fine on read, promote to list(source-id) on write
source-id = null && source-ids = list - Spec compliant config

Yes, I agree that we should break if they doesn't match. We have a similar situation with schema and schemas with current-schema-id that might point to two different schemas.

When we write both, we can resolve that, but I think we don't need them both?

Yeah my suggestion is that we still say in V3 that source-id is omitted but in the implementation we can still handle an out of spec metadata if we come across one. That way the spec reader code doesn't have to be version dependent.

If both source-ids and source-id are written for single-arg transform, a v3-supported reader can take a shortcut on source-ids and only fall back to source-id when source-ids is missing. Isn't this a desirable feature? Though someone may argue that we have to check the consistency between them like the above discussion.

That way the spec reader code doesn't have to be version dependent.

My goal was to be completely independent of the table version, also on the write side. Why not entirely remove this constraint by writing source-id or source-ids based on the number of arguments? :) I think they should be written mutually exclusive, to avoid any confusion.

Oh, sorry for misclarification, my question is for multi arg transform like bucket(a, b, c), which source id's value should we write since it's now changed to required.

I think we can be strict on the write side ... unless we want to keep both fields indefinitely which I'm not sure is super beneficial?

sfc-gh-bhannel · 2025-03-26T20:14:42Z

format/spec.md

@@ -540,7 +540,7 @@ Notes:
 2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
 3. Strings are truncated to a valid UTF-8 string with no more than `L` code points.
 4. In contrast to strings, binary values do not have an assumed encoding and are truncated to `L` bytes.
-
+5. For multi-argument bucketing, the hashes are `xor`'ed: `hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`.


I think it would be good to be explicit about how to handle nulls and dropped columns. Should it be calculated as if hash(null) == 0?

It also might be worth adding parentheses for those who don't recall the precedence order between XOR and modulo.

Also - good to meet you, and thanks for working on this! I'm a developer at Snowflake working on Iceberg functionality, and we really appreciate how detailed the spec is!

Based on the fact that bucket(null) returns null, if any value is null then the entire hash value should be null? I agree that this should be explicit.

Great to meet you as well @sfc-gh-bhannel, thanks for jumping in here.

It also might be worth adding parentheses for those who don't recall the precedence order between XOR and modulo.

Thanks, there was a bracket missing actually ;)

The spec states (just below the table):

All transforms must return null for a null input value.

I think we can amend this for the multi-arg transforms that we leave out null values, but we can still hash the other not-null fields.

Suggested change

5. For multi-argument bucketing, the hashes are `xor`'ed: `hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`.

5. For multi-argument bucketing, the hashes for the not-null input values are `xor`'ed: `(hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`. The transform will return `null` when all input values are `null`.

sfc-gh-bhannel · 2025-03-26T20:15:56Z

format/spec.md

 |          | required | required | **`field-id`**   | `JSON int`          | 1000         |
 | required | required | required | **`name`**       | `JSON string`       | `id_bucket`  |
 | required | required | required | **`transform`**  | `JSON string`       | `bucket[16]` |

+Notes:
+
+1. For partition fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted.


Would it be more consistent to always use source-ids for v3 writers? Or do you want to use source-id to allow v3 writers to be compatible with v2 readers if you don't use v3 features?

This makes the serialization logic much easier, since we don't need to know about the table version. Otherwise, we would need to plumb down the table version into the serializer.

sfc-gh-bhannel · 2025-03-26T20:16:42Z

format/spec.md

-In v3 metadata, writers must use only `source-ids` because v3 requires reader support for multi-arg transforms.
+Notes:
+
+1. For sort fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted.


Did you mean to make the transform row refer to this footnote? I would expect it to be the source-id and source-ids rows.

This was the idea:

But I guess it is not obvious? I think @wgtmac raised the same question in #12644 (comment)?

szehon-ho · 2025-03-27T00:31:25Z

Hi @Fokko i havent take a look yet at spec change, but for multi bucket we had some discussions last year. For reference the pr is here : #8259 with more discussion from: #8579

wgtmac · 2025-03-27T07:57:08Z

format/spec.md

@@ -540,7 +540,7 @@ Notes:
 2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
 3. Strings are truncated to a valid UTF-8 string with no more than `L` code points.
 4. In contrast to strings, binary values do not have an assumed encoding and are truncated to `L` bytes.
-
+5. For multi-argument bucketing, the hashes are `xor`'ed: `hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`.


Is there a specific reason to choose xor but not other arithmetic operations?

I took inspiration from Apache Hive. I think it plays well with the hashing.

wgtmac · 2025-03-27T07:58:43Z

format/spec.md

@@ -540,7 +540,7 @@ Notes:
 2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
 3. Strings are truncated to a valid UTF-8 string with no more than `L` code points.
 4. In contrast to strings, binary values do not have an assumed encoding and are truncated to `L` bytes.
-
+5. For multi-argument bucketing, the hashes are `xor`'ed: `hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`.


Based on the fact that bucket(null) returns null, if any value is null then the entire hash value should be null? I agree that this should be explicit.

wgtmac · 2025-03-27T08:17:35Z

format/spec.md

@@ -1453,13 +1457,15 @@ Each sort field in the fields list is stored as an object with the following pro

 | V1       | V2       | V3       | Field            | JSON representation | Example     |
 |----------|----------|----------|------------------|---------------------|-------------|
-| required | required | required | **`transform`**  | `JSON string`       | `bucket[4]` |
-| required | required | omitted  | **`source-id`**  | `JSON int`          | 1           |
+| required | required | required¹ | **`transform`**  | `JSON string`       | `bucket[4]` |


I guess the note is in the wrong place?

format/spec.md

wgtmac · 2025-03-27T08:22:43Z

format/spec.md

@@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p

 | V1       | V2       | V3       | Field            | JSON representation | Example      |
 |----------|----------|----------|------------------|---------------------|--------------|
-| required | required | omitted  | **`source-id`**  | `JSON int`          | 1            |


If both source-ids and source-id are written for single-arg transform, a v3-supported reader can take a shortcut on source-ids and only fall back to source-id when source-ids is missing. Isn't this a desirable feature? Though someone may argue that we have to check the consistency between them like the above discussion.

Co-authored-by: Gang Wu <[email protected]>

Spec: Allow the use of source-id in V3

f17f457

github-actions bot added the Specification Issues that may introduce spec changes. label Mar 25, 2025

RussellSpitzer reviewed Mar 25, 2025

View reviewed changes

sfc-gh-bhannel reviewed Mar 26, 2025

View reviewed changes

wgtmac reviewed Mar 27, 2025

View reviewed changes

Add the

d60f5e9

Co-authored-by: Gang Wu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spec: Allow the use of `source-id` in V3 #12644

Spec: Allow the use of `source-id` in V3 #12644

Fokko commented Mar 25, 2025

RussellSpitzer Mar 25, 2025

liurenjie1024 Mar 26, 2025

RussellSpitzer Mar 26, 2025

Fokko Mar 26, 2025

RussellSpitzer Mar 26, 2025

wgtmac Mar 27, 2025

Fokko Mar 27, 2025 •

edited

Loading

liurenjie1024 Mar 27, 2025

RussellSpitzer Mar 28, 2025

sfc-gh-bhannel Mar 26, 2025

wgtmac Mar 27, 2025

Fokko Mar 27, 2025 •

edited

Loading

sfc-gh-bhannel Mar 26, 2025

Fokko Mar 27, 2025

sfc-gh-bhannel Mar 26, 2025

Fokko Mar 27, 2025

szehon-ho commented Mar 27, 2025

wgtmac Mar 27, 2025

Fokko Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

	5. For multi-argument bucketing, the hashes are `xor`'ed: `hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`.
	5. For multi-argument bucketing, the hashes for the not-null input values are `xor`'ed: `(hash(col1) ⊕ hash(col2) ⊕ ... ⊕ hash(colN)) % W`. The transform will return `null` when all input values are `null`.

Spec: Allow the use of source-id in V3 #12644

Are you sure you want to change the base?

Spec: Allow the use of source-id in V3 #12644

Conversation

Fokko commented Mar 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fokko Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fokko Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szehon-ho commented Mar 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Spec: Allow the use of `source-id` in V3 #12644

Spec: Allow the use of `source-id` in V3 #12644

Fokko Mar 27, 2025 •

edited

Loading

Fokko Mar 27, 2025 •

edited

Loading