[2/x] (alternative to 1/x) feat: add proto plugin for typed serialization and UC Volume I/O#215
Closed
[2/x] (alternative to 1/x) feat: add proto plugin for typed serialization and UC Volume I/O#215
Conversation
Lean alternative to the full gRPC stack. Provides protobuf message
definitions as cross-language data contracts with build-integrated
codegen and typed Volume I/O via FilesConnector.
What's included:
- Proto definitions (proto/appkit/v1/): common.proto, pipeline.proto
- Build-integrated codegen: buf generate → shared/src/proto/ (Turbo-cached)
- ProtoSerializer: binary/JSON ser/de + UC Volume read/write
- Uses FilesConnector (not raw SDK) for upload bug fix + telemetry
- ProtoPlugin: thin wrapper with interceptor chain (retry/timeout)
- 1 runtime dep: @bufbuild/protobuf (~15KB)
What's NOT included (add later if needed):
- No gRPC server/client (no @connectrpc/connect)
- No HTTP/2, no Connect protocol, no service definitions
- No standalone server mode
Usage:
const app = await createApp({ plugins: [proto()] });
app.proto.writeToVolume(JobResultSchema, result, "/Volumes/.../r.pb");
const r = await app.proto.readFromVolume(JobResultSchema, "/Volumes/.../r.pb");
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Remove Volume I/O from proto plugin — file operations belong in the Files plugin. Proto plugin now provides only typed data contracts: create, serialize, deserialize, toJSON, fromJSON. Changes: - Remove writeToVolume, readFromVolume, exists from plugin and serializer - Remove FilesConnector dependency from serializer - Remove defaultVolume/timeout config (no longer relevant) - Add create() method for typed message construction - Update tests to verify serialization-only API surface - Add docs/docs/plugins/proto.md with usage examples - Add proto() to plugin index example Co-authored-by: Isaac Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Contributor
Author
|
Superseded by #220 — clean single-commit PR with simplified proto plugin (pure serialization, no Volume I/O). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lean alternative to the full gRPC stack. Provides protobuf message definitions as cross-language data contracts with build-integrated codegen and typed Volume I/O via FilesConnector.
What's included:
What's NOT included (add later if needed):
Usage:
const app = await createApp({ plugins: [proto()] }); app.proto.writeToVolume(JobResultSchema, result, "/Volumes/.../r.pb"); const r = await app.proto.readFromVolume(JobResultSchema, "/Volumes/.../r.pb");