Skip to content

Add MsgPack format support - #1068

Open
rahulbsw wants to merge 7 commits into
ArroyoSystems:masterfrom
rahulbsw:msgpack-format
Open

Add MsgPack format support#1068
rahulbsw wants to merge 7 commits into
ArroyoSystems:masterfrom
rahulbsw:msgpack-format

Conversation

@rahulbsw

Copy link
Copy Markdown

Summary

  • Adds msgpack as a first-class Arroyo Format, including SQL connector options and OpenAPI/client type exposure.
  • Implements MsgPack source deserialization for structured map records and into_unstructured_json payloads.
  • Implements MsgPack sink serialization, including native MsgPack binary values for Arrow binary fields.
  • Wires Kafka schema validation and format/schema validation coverage for the new format.

Notes

  • Structured MsgPack records are decoded through Arroyo's existing Arrow JSON path, so top-level records must be MsgPack maps with string keys.
  • MsgPack extension values are currently rejected instead of silently lossy-converted.

Testing

  • cargo fmt --check
  • cargo test -p arroyo-formats
  • cargo test -p arroyo-rpc msgpack
  • cargo check -p arroyo-connectors -p arroyo-planner -p arroyo-formats -p arroyo-rpc
  • cargo check -p arroyo-api with a temporary local Postgres database for compile-time SQL validation
  • pnpm run openapi
  • pnpm exec tsc --noEmit
  • pnpm run build

Fixes #904

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support MsgPack format/serdes

1 participant