Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spec: Variant lower/upper bounds #12658

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

spec: Variant lower/upper bounds #12658

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a93bd21
Variant bound spec
aihuaxu Mar 26, 2025
31dbfa2
Add array path and change to only use normalized path.
aihuaxu Mar 27, 2025
af19925
Clarify the condition to collect bounds
aihuaxu Mar 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions format/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,9 @@ Notes:
5. The `content_offset` and `content_size_in_bytes` fields are used to reference a specific blob for direct access to a deletion vector. For deletion vectors, these values are required and must exactly match the `offset` and `length` stored in the Puffin footer for the deletion vector blob.
6. The following field ids are reserved on `data_file`: 141.

For `variant` type, the `lower_bounds` and `upper_bounds` store the minimum and maximum values for all shredded subcolumns within a file. These bounds are represented as a Variant object, where each subcolumn path serves as a key and the corresponding bound value as the value. The object is then serialized into binary format (see [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For variant type, the lower_bounds and upper_bounds store the minimum and maximum values for all shredded subcolumns within a file.

I'm not quite sure if we need to add the collection behavior of lower_bound and upper_bound under different conditions as mentioned in the proposal. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XBaith What kind of collection behavior you are referring to? Can you clarify?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lower and upper bound statistics for subcolumns are collected for each data file based on the following conditions:
Uniform Value Types:
If all subcolumn values match the shredded type, lower/upper bounds are collected.
Example: For event.location.longitude, if all values are of the double type, the lower/upper bounds are written to the manifest file.
Mixed Value Types:
If the subcolumn contains multiple types (e.g., double and string), lower/upper bound statistics are not collected.
Example: For event.location.longitude, if the values include both double and string, lower/upper bounds are excluded.
Some subcolumn values are nulls or missing:
If some subcolumn values are null or missing in a file, but the available values match the shredded type, lower/upper bounds are still collected.
If all the subcolumn values are nulls, then lower/upper bounds are not collected. null_value_counts stat can be collected in later implementation to be used with value_counts to know they are all nulls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense. Let me include that: basically we only collect bounds when all the subcolumn values are of the same shredded type or null.

Subcolumn paths follow the JSON path format, such as `$.event.event_type` for standard keys or `$.event.["user.name"]` for keys containing dots.
Copy link
Contributor

@rdblue rdblue Mar 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that $.event.["user.name"] should not have a . after event in the format. I would also say that if we are asking for JSON path notation, then we should use the canonical "Normalized Path" form (Section 2.7), which is to always use ['name'] syntax (with single quotes, not double). I think that's probably a good idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense. That would reduce the variation. I also update to add the array representation to use always index 0 like $['tags'][0]. Let me know your thoughts.


For `geometry` and `geography` types, `lower_bounds` and `upper_bounds` are both points of the following coordinates X, Y, Z, and M (see [Appendix G](#appendix-g-geospatial-notes)) which are the lower / upper bound of all objects in the file. For the X values only, xmin may be greater than xmax, in which case an object in this bounding box may match if it contains an X such that `x >= xmin` OR`x <= xmax`. In geographic terminology, the concepts of `xmin`, `xmax`, `ymin`, and `ymax` are also known as `westernmost`, `easternmost`, `southernmost` and `northernmost`, respectively. For `geography` types, these points are further restricted to the canonical ranges of [-180 180] for X and [-90 90] for Y.

The `partition` struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct's field ids must match the ids from the partition spec.
Expand Down Expand Up @@ -1558,6 +1561,7 @@ The binary single-value serialization can be used to store the lower and upper b
|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **`geometry`** | A single point, encoded as a x:y:z:m concatenation of its 8-byte little-endian IEEE 754 coordinate values. x and y are mandatory. This becomes x:y if z and m are both unset, x:y:z if only m is unset, and x:y:NaN:m if only z is unset. |
| **`geography`** | A single point, encoded as a x:y:z:m concatenation of its 8-byte little-endian IEEE 754 coordinate values. x and y are mandatory. This becomes x:y if z and m are both unset, x:y:z if only m is unset, and x:y:NaN:m if only z is unset. |
| **`variant`** | A `Variant` object, where each subcolumn path serves as a key and the corresponding bound value as the value. Subcolumn paths follow the JSON path format. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variant is not something that we can reference here because that is a class in the implementation. I think what you want to say is that the serialized value is a variant metadata (v1) concatenated with a variant object. The object's fields are field paths in the normalized JSON path format and the values are the upper or lower bound corresponding to the shredded type in the data file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense. Here we want to describe how exactly the data is laid out


### JSON single-value serialization

Expand Down