Skip to content

Support reading parquet maps from older writers #95

@parsnips

Description

@parsnips

Describe the bug
AWS Kinesis produces maps in an old way, most notably using map instead of key_value

example:

optional group new (MAP) {
    repeated group map (MAP_KEY_VALUE) {
      required binary key (UTF8);
      optional group value {
        optional binary b (UTF8);
        optional binary n (UTF8);
      }
    }
  }

When trying to populate a struct from a parquet file with this, it'll throw an error.

Unit test to reproduce

https://github.com/parsnips/parquet-go-1/tree/parsnips/support-filling-map has unit test and fix.

parquet-go specific details
reproduced on the latest version on main.

Misc Details
This is parquet generated by AWS Kinesis Firehose ParquetSerDe V2.

I have a test file here: https://gist.github.com/parsnips/928c14d850331dd21c8d917227f77c5c#file-kinesisserde-v2-parquet

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions