Add a minimal msgpack serialiser for ccf - #7866
Open
cjen1-msft wants to merge 29 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a minimal, header-only MessagePack encoder under src/msgpack/ (including fluentd in_forward EventTime ext type support) to enable more efficient log export as part of the broader observability/log-export work (#7858).
Changes:
- Added a C++20 header-only msgpack encoder (
write_*primitives, container headers, andFluentdEventTime). - Added unit tests (boundary tables + property tests) and differential tests using
nlohmann::json::from_msgpackas an oracle. - Added a libFuzzer harness and integrated the new unit/fuzz tests into the build.
Custom instructions used:
.github/copilot-instructions.md.github/instructions/reviewing.instructions.md
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/msgpack/encode.h |
New header-only msgpack encoder API, error model, and Fluentd EventTime support |
src/msgpack/endian.h |
Big-endian writing helper for msgpack wire format |
src/msgpack/test/encode_test.cpp |
Property/boundary tests for smallest-format-wins encoding and EventTime layout |
src/msgpack/test/differential_test.cpp |
Differential encode-vs-nlohmann decode tests + fluentd message-mode pinned vector |
src/msgpack/test/gen.h |
Shared generator + script driver used by tests and fuzz harness |
src/msgpack/test/fuzz_script_test.cpp |
Deterministic/canned script tests pinning specific composite byte layouts |
src/msgpack/test/format_introspect.h |
Test helper to classify msgpack first-byte format families |
src/msgpack/test/msgpack_fuzz.cpp |
libFuzzer harness for encoder round-trip validation via nlohmann oracle |
CMakeLists.txt |
Registers msgpack_test unit test and msgpack_fuzz_test fuzz target |
cjen1-msft
marked this pull request as draft
May 6, 2026 10:33
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
cjen1-msft
marked this pull request as ready for review
August 18, 2026 08:49
Copilot started reviewing on behalf of
Amaury Chamayou (achamayou)
August 26, 2026 06:50
View session
Batch tags and big-endian payloads into a single vector growth, document reusable buffer storage, and restore benchmarks for fresh, reserved, and reused outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cjen1-msft
marked this pull request as draft
August 26, 2026 18:16
Resolve the Fluentd EventTime write conflict and adapt the new map32 path to the batched endian writer. Fix the updated base branch's unit and fuzz build breaks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use std::byteswap for MessagePack encoding and cite the format constants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Batch tags and big-endian payloads into a single vector growth, document reusable buffer storage, and restore benchmarks for fresh, reserved, and reused outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reviewed changes were rebased and applied as 3ae45a4. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cjen1-msft
marked this pull request as ready for review
August 28, 2026 16:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For #7858 we want to export more logs more efficiently.
I proposed that we should support msgpack export, and this is a minimal msgpack serialiser.
Here is a benchmark of the serialiser against nlohmann (json dump) and rapidjson (as an 'optimal baseline')