Skip to content

Custom metadata in data files #106

@mccheah

Description

@mccheah

It would be useful for consumers of Iceberg tables to be able to specify additional metadata in data files that enable them to know how to read the files. Some examples of custom metadata include:

  • Encryption keys required to read the file,
  • Compression codecs specified on the file without needing to have a specific file extension,
  • Metadata that's specific to a custom file format. Suppose we supported CSV tables in Iceberg down the road. It would be nice to attach the column delimiter on a per-file basis so that a table can be comprised of multiple files that may not necessarily be uniform in terms of the exact layout, but have compatible schemas.

The custom metadata field should be of type Map<String, String> and can be an optional column.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions