Skip to content

Map google.protobuf.Struct to JSON in decoder#28295

Open
ccampo133 wants to merge 3 commits intotrinodb:masterfrom
ccampo133:feat/decode_pbf_struct
Open

Map google.protobuf.Struct to JSON in decoder#28295
ccampo133 wants to merge 3 commits intotrinodb:masterfrom
ccampo133:feat/decode_pbf_struct

Conversation

@ccampo133
Copy link

Description

Map google.protobuf.Struct, google.protobuf.Value, and google.protobuf.ListValue well-known types to Trino's JSON type in the Kafka protobuf schema parser and decoder. This follows the same pattern used for google.protobuf.Any (which is already mapped to JSON, per 5ebbd2e).

I don't believe any configuration flag is needed like the one used for google.protobuf.Any. Unlike Any (which
requires a DescriptorProvider to resolve the type_url field), Struct is entirely self-describing and always safe to decode as JSON.

See the description in #28294 for why this approach is appropriate.

Additional context and related issues

The changes span two modules:

  • Schema parsing in ProtobufSchemaParser: Intercepts Struct/Value/ListValue types before the recursive type detection and maps them to JSON.

  • Runtime decoding in ProtobufColumnDecoder and ProtobufValueProvider: Converts Struct DynamicMessage instances to sorted JSON strings using JsonFormat.printer(). Handles Struct both as top-level fields and when nested inside arrays, maps, or row types.

The sorted() helper in ProtobufColumnDecoder was also generalized to handle non-object JSON values (e.g., "foo" from a Value field or ["foo","bar"] from a ListValue field) by using Object.class instead of Map.class in the Jackson mapper. This is backward-compatible since existing callers (Any, Oneof) always produce JSON objects.

Closes #28294

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

* Add support for `google.protobuf.Struct`, `Value`, and `ListValue` types in the Kafka protobuf decoder, mapping them to Trino's `JSON` type.

Map google.protobuf.Struct, Value, and ListValue
well-known types to Trino's JSON type in the Kafka
protobuf schema parser and decoder, following the
same pattern as google.protobuf.Any and Timestamp.
@cla-bot
Copy link

cla-bot bot commented Feb 13, 2026

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@github-actions github-actions bot added the kafka Kafka connector label Feb 13, 2026
@Praveen2112 Praveen2112 self-requested a review February 16, 2026 12:20
@cla-bot
Copy link

cla-bot bot commented Feb 17, 2026

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@cla-bot
Copy link

cla-bot bot commented Feb 17, 2026

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kafka Kafka connector

Development

Successfully merging this pull request may close these issues.

Kafka protobuf decoder rejects google.protobuf.Struct as recursive type

1 participant

Comments