Bump msgpackr to 1.12.1: fix + recover two-byte over-cap record corruption (v5.0) - #1179
Merged
Conversation
…ecovery) msgpackr's two-byte record path (used when maxOwnStructures + maxSharedStructures > 64; harper's primary stores set maxOwnStructures=256) mis-read self-contained over-cap record definitions: recordDefinition went through a second-byte reader that consumed the first value byte as a phantom high byte, yielding a bogus, never-defined record id -> "Record id is not defined for N". Any high-cardinality table (e.g. undeclared dynamic attributes) writing records past the shared cap produced undecodable records — on both the typed default and the randomAccessFields=false opt-out. msgpackr 1.12.1 fixes this on the read path only (the encoder is unchanged), so the bump both stops new corruption AND recovers records already written by v5.0.29 (their bytes were always correct, only the read was wrong). Keeps maxOwnStructures=256 (the read fix makes two-byte correct again; 256 shares moderate-cardinality shapes ~30% more compactly than a one-byte cap of 32). Adds a regression test that writes 3000 distinct shapes through a shared store and asserts every record decodes, on both the typed and readOnlyStructures paths (fails against 1.12.0, passes against 1.12.1). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
pbrumblay
pushed a commit
to pbrumblay/harper
that referenced
this pull request
Jun 26, 2026
v2 counterpart of the v5.0 msgpackr 1.12.1 bump. msgpackr's two-byte record path mis-read self-contained over-cap record definitions (recordDefinition went through a second-byte reader that consumed the first value byte as a phantom high byte → "Record id is not defined for N"), making high-cardinality records undecodable. Fixed upstream in msgpackr 2.0.4 (kriszyp/msgpackr#190), read-path only, so the bump fixes the decode and recovers any already-written data without changing the on-disk format. On main the typed path goes through structon; the classic two-byte path is reached via the randomAccessFields=false opt-out / toJSON fallback, so this is a latent-correctness fix for 5.1. Encoder regression coverage lives upstream (msgpackr#190) and on the v5.0 line (HarperFast#1179). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps msgpackr 1.12.0 → 1.12.1, which fixes the two-byte record-id decode bug that made high-cardinality records undecodable (
Record id is not defined for N). This is the real fix for the corruption reported on CDI/dev — and because msgpackr 1.12.1 is a read-path-only change (the encoder is unchanged), it recovers records already written by v5.0.29, not just future writes.Background
RecordEncodersetsmaxOwnStructures=256, which puts msgpackr on its two-byte record path. msgpackr ≤1.12.0 mis-read self-contained over-cap record definitions (recordDefinitionused a second-byte reader that consumed the first value byte as a phantom high byte → bogus, never-defined id). So any table exceeding the ~32-shape shared cap (e.g. undeclared dynamic attributes like CDI'sRaceEntry) wrote undecodable records — on both the typed default and therandomAccessFields=falseopt-out (#1169). Root-caused + fixed upstream in kriszyp/msgpackr#189 (v1.12.x).Why keep
maxOwnStructures=256(not the earlier cap→32 stop-gap, #1176)With the read fix, the two-byte path is correct again, so the original 256 is safe. Measured: at 256 a moderate-cardinality table (≈120 stable shapes) is ~32% smaller than at 32 (it shares shapes instead of re-inlining a definition per record); for extreme cardinality the two are within ~2%. So 256 is both correct and more compact. #1176 (cap→32) is superseded and will be closed.
Patch target
v5.0.30. This recovers CDI's currently-undecodable data on upgrade.
Where to look
package.json/package-lock.json— msgpackr1.12.1(exact pin).unitTests/resources/recordEncoder.test.js— new "two-byte over-cap" block: writes 3000 distinct shapes through a shared store, asserts every record decodes (typed +readOnlyStructures). Fails on 1.12.0, passes on 1.12.1.Notes
recordEncoder.test.jsconfig/byte-range tests flake under full-file ordering, and the resources suite has the environmental "Data read" setup crash. The new tests are robust to ambient config (they assert decodability).🤖 Generated by Claude (Opus 4.7).