Skip to content

Parquet reader does not support unknown physical/logical type combinations per GH-607 #10941

Description

@etseidl

Describe the bug

parquet-format GH-607 adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an INT32 column annotated with UUID). This crate currently checks for unsupported combinations and errors when encountering them.

To Reproduce

Try reading the file added in apache/parquet-testing#122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).

Expected behavior

Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.

It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.

Additional context

We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions