Skip to content

Add a minimal msgpack serialiser for ccf - #7866

Open
cjen1-msft wants to merge 29 commits into
microsoft:mainfrom
cjen1-msft:msgpack
Open

Add a minimal msgpack serialiser for ccf#7866
cjen1-msft wants to merge 29 commits into
microsoft:mainfrom
cjen1-msft:msgpack

Conversation

@cjen1-msft

Copy link
Copy Markdown
Contributor

For #7858 we want to export more logs more efficiently.
I proposed that we should support msgpack export, and this is a minimal msgpack serialiser.

Here is a benchmark of the serialiser against nlohmann (json dump) and rapidjson (as an 'optimal baseline')

 Name (* = baseline)      |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
--------------------------|--------:|----------:|--------:|-------:|----------:
 json_dump *              |     100 |     1.631 |   16306 |      - |    61323.7
 rapidjson_write          |     100 |     0.142 |    1423 |  0.087 |   702622.2
 msgpack_encode           |     100 |     0.189 |    1894 |  0.116 |   527779.7
 json_dump *              |    1000 |    10.267 |   10266 |      - |    97401.4
 rapidjson_write          |    1000 |     1.886 |    1886 |  0.184 |   530199.1
 msgpack_encode           |    1000 |     2.280 |    2280 |  0.222 |   438575.1
 json_dump *              |   10000 |    97.549 |    9754 |      - |   102512.6
 rapidjson_write          |   10000 |    12.481 |    1248 |  0.128 |   801223.8
 msgpack_encode           |   10000 |    16.415 |    1641 |  0.168 |   609205.9

@cjen1-msft
cjen1-msft requested a review from a team as a code owner May 6, 2026 08:54
Copilot AI review requested due to automatic review settings May 6, 2026 08:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a minimal, header-only MessagePack encoder under src/msgpack/ (including fluentd in_forward EventTime ext type support) to enable more efficient log export as part of the broader observability/log-export work (#7858).

Changes:

  • Added a C++20 header-only msgpack encoder (write_* primitives, container headers, and FluentdEventTime).
  • Added unit tests (boundary tables + property tests) and differential tests using nlohmann::json::from_msgpack as an oracle.
  • Added a libFuzzer harness and integrated the new unit/fuzz tests into the build.

Custom instructions used:

  • .github/copilot-instructions.md
  • .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/msgpack/encode.h New header-only msgpack encoder API, error model, and Fluentd EventTime support
src/msgpack/endian.h Big-endian writing helper for msgpack wire format
src/msgpack/test/encode_test.cpp Property/boundary tests for smallest-format-wins encoding and EventTime layout
src/msgpack/test/differential_test.cpp Differential encode-vs-nlohmann decode tests + fluentd message-mode pinned vector
src/msgpack/test/gen.h Shared generator + script driver used by tests and fuzz harness
src/msgpack/test/fuzz_script_test.cpp Deterministic/canned script tests pinning specific composite byte layouts
src/msgpack/test/format_introspect.h Test helper to classify msgpack first-byte format families
src/msgpack/test/msgpack_fuzz.cpp libFuzzer harness for encoder round-trip validation via nlohmann oracle
CMakeLists.txt Registers msgpack_test unit test and msgpack_fuzz_test fuzz target

Comment thread src/msgpack/test/gen.h Outdated
Comment thread src/msgpack/encode.h
Comment thread src/msgpack/test/gen.h Outdated
Comment thread src/msgpack/test/msgpack_unit.cpp
Comment thread src/msgpack/test/encode_test.cpp Outdated
Comment thread src/msgpack/encode.h Outdated
Comment thread src/msgpack/encode.h Outdated
@cjen1-msft
cjen1-msft marked this pull request as draft May 6, 2026 10:33
Comment thread src/msgpack/endian.h Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 38e26ecc-d8a3-452a-8f63-de338955dffb
@cjen1-msft
cjen1-msft marked this pull request as ready for review August 18, 2026 08:49
Comment thread src/msgpack/encode.h Outdated
Comment thread client_out.txt Outdated
Comment thread src/msgpack/encode.h Outdated
Comment thread src/msgpack/encode.h
Comment thread src/msgpack/test/gen.h Outdated
@achamayou
Amaury Chamayou (achamayou) requested a balanced review from Copilot August 26, 2026 06:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Comment thread src/msgpack/encode.h
Comment thread src/msgpack/encode.h Outdated
Comment thread src/msgpack/fluentd_event_time.h Outdated
Comment thread src/msgpack/test/encode_test.cpp Outdated
Batch tags and big-endian payloads into a single vector growth, document reusable buffer storage, and restore benchmarks for fresh, reserved, and reused outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cjen1-msft
cjen1-msft marked this pull request as draft August 26, 2026 18:16
Chris Jensen (Cjen1) and others added 8 commits August 27, 2026 08:27
Resolve the Fluentd EventTime write conflict and adapt the new map32 path to the batched endian writer. Fix the updated base branch's unit and fuzz build breaks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use std::byteswap for MessagePack encoding and cite the format constants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Batch tags and big-endian payloads into a single vector growth, document reusable buffer storage, and restore benchmarks for fresh, reserved, and reused outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reviewed changes were rebased and applied as 3ae45a4.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cjen1-msft
cjen1-msft marked this pull request as ready for review August 28, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants