Complete protobuf schemas for ingestion - #1375
Draft
jwils wants to merge 1 commit into
Draft
Conversation
Preserve singular field presence in proto2 and proto3, represent nested lists with wrapper messages, and include types used only as indexing sources. Compile both syntaxes and verify their binary round trips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The protobuf generator rejects nested lists and omits source-only types used by multi-source indexing. Its proto3 scalar fields also lose the distinction between omitted values and explicit defaults.
What
Generate wrapper messages for nested lists, include source-only types and their dependencies, and emit optional singular fields in both protobuf syntaxes.
How
Each inner list gets a message with a repeated
valuesfield. Source-only types come from the existingsourced_fromupdate targets. Compiler tests exercise both syntaxes and verify field presence and nested values through binary round trips.Risk
Generated proto3 schemas now require protoc 3.15 or newer. Publishers must regenerate code to preserve explicit zero, false, and empty string values. Repeated fields still cannot represent null lists or null elements.
Testing
No manual testing.
Bigger picture
This completes schema generation needed for native protobuf ingestion. It is independent of the ingestion adapter stack; runtime decoding and indexing will follow separately.