Skip to content

Commit ca1bbc9

Browse files
kriszypclaude
andcommitted
fix: bump msgpackr to 1.12.1 (two-byte over-cap record corruption + recovery)
msgpackr's two-byte record path (used when maxOwnStructures + maxSharedStructures > 64; harper's primary stores set maxOwnStructures=256) mis-read self-contained over-cap record definitions: recordDefinition went through a second-byte reader that consumed the first value byte as a phantom high byte, yielding a bogus, never-defined record id -> "Record id is not defined for N". Any high-cardinality table (e.g. undeclared dynamic attributes) writing records past the shared cap produced undecodable records — on both the typed default and the randomAccessFields=false opt-out. msgpackr 1.12.1 fixes this on the read path only (the encoder is unchanged), so the bump both stops new corruption AND recovers records already written by v5.0.29 (their bytes were always correct, only the read was wrong). Keeps maxOwnStructures=256 (the read fix makes two-byte correct again; 256 shares moderate-cardinality shapes ~30% more compactly than a one-byte cap of 32). Adds a regression test that writes 3000 distinct shapes through a shared store and asserts every record decodes, on both the typed and readOnlyStructures paths (fails against 1.12.0, passes against 1.12.1). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent cd7962a commit ca1bbc9

3 files changed

Lines changed: 47 additions & 5 deletions

File tree

package-lock.json

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,7 @@
205205
"minimist": "1.2.8",
206206
"moment": "2.30.1",
207207
"mqtt-packet": "~9.0.1",
208-
"msgpackr": "1.12.0",
208+
"msgpackr": "1.12.1",
209209
"needle": "3.5.0",
210210
"node-forge": "^1.3.1",
211211
"node-stream-zip": "1.15.0",

unitTests/resources/recordEncoder.test.js

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,3 +177,45 @@ describe('RecordEncoder random-access fields opt-out (readOnlyStructures)', () =
177177
});
178178
});
179179
});
180+
181+
describe('RecordEncoder high-cardinality round-trip (msgpackr two-byte over-cap)', () => {
182+
// With the default maxOwnStructures (256) and a shared store, msgpackr uses two-byte record ids.
183+
// A table whose distinct record shapes exceed the shared cap emits self-contained two-byte record
184+
// definitions; msgpackr < 1.12.1 mis-read those ("Record id is not defined for N"), leaving
185+
// high-cardinality records permanently undecodable — on both the typed default and the
186+
// randomAccessFields=false opt-out. msgpackr 1.12.1's read fix makes them round-trip (and decodes
187+
// already-written data). These write far more distinct shapes than the cap through a shared store
188+
// and assert every record decodes; they fail against msgpackr 1.12.0 and pass against 1.12.1.
189+
function diverseRecords(count) {
190+
const records = [];
191+
for (let i = 0; i < count; i++) {
192+
const rec = { id: i };
193+
const attrCount = 1 + (i % 6);
194+
for (let a = 0; a < attrCount; a++) rec['attr_' + ((i * 7 + a) % 400)] = i % 2 ? 's' + i : i;
195+
records.push(rec);
196+
}
197+
return records;
198+
}
199+
200+
function assertAllDecode(extra) {
201+
const store = sharedStore();
202+
const writer = makeEncoder(store, extra);
203+
const reader = makeEncoder(store, extra);
204+
const records = diverseRecords(3000); // far exceeds the structure cap, forcing the two-byte over-cap path
205+
const buffers = records.map((r) => Buffer.from(writer.encode(r)));
206+
let failures = 0;
207+
for (let i = 0; i < buffers.length; i++) {
208+
const decoded = reader.decode(buffers[i]);
209+
if (!decoded || decoded.id !== records[i].id) failures++;
210+
}
211+
assert.strictEqual(failures, 0, `all ${records.length} records should decode past the structure cap`);
212+
}
213+
214+
it('decodes every record past the structure cap on the typed path (default)', () => {
215+
assertAllDecode(undefined);
216+
});
217+
218+
it('decodes every record past the structure cap with randomAccessFields off (readOnlyStructures)', () => {
219+
assertAllDecode({ readOnlyStructures: true });
220+
});
221+
});

0 commit comments

Comments
 (0)