Skip to content

fix(collector): treat a blank published value as absent, not as the epoch - #6275

Open
marmar9615-cloud wants to merge 1 commit into
Mintplex-Labs:masterfrom
marmar9615-cloud:fix/raw-text-null-published
Open

fix(collector): treat a blank published value as absent, not as the epoch#6275
marmar9615-cloud wants to merge 1 commit into
Mintplex-Labs:masterfrom
marmar9615-cloud:fix/raw-text-null-published

Conversation

@marmar9615-cloud

@marmar9615-cloud marmar9615-cloud commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Resolves #6274.

GET /v1/document/metadata-schema documents published as "epoch timestamp in ms | nullable", but sending it as null stamps the document with the Unix epoch.

collector/processRawText/index.js:42

if (isNaN(Number(published))) return new Date().toLocaleString();
return new Date(Number(published)).toLocaleString();

Number(null), Number("") and Number(" ") are all 0, so the isNaN guard never fires and the value becomes 1/1/1970 (12/31/1969 west of UTC). Omitting the key works, because Number(undefined) is NaN.

Why this is worth more than a wrong date column

published is plucked into chunkHeaderMeta (server/utils/TextSplitter/index.js:73-76) and stringifyHeader() prepends it as literal <document_metadata> text on every chunk (:135-147), called unconditionally by all eight vector providers. So the epoch date is embedded into the text the model reads. That path is traced through the source rather than exercised here: the tests in this PR assert the collector's published value, not the rendered chunk header. In the UI it only surfaces in the file row's hover tooltip (Directory/FileRow/index.jsx:12-17), which is the smaller half of the problem.

Change

Five inserted lines in one file, three of them comment. Guard on a trimmed string so null, "", whitespace and [] are all absent.

published: 0 is a genuine epoch timestamp and still resolves to 1970. No clause is needed for undefined: Number(undefined) is already NaN, so it falls to the existing isNaN branch. I checked that rather than adding a clause that does nothing.

Tests

Nine cases in collector/__tests__/processRawText/index.test.js, run against three variants in git archive exports so the working tree was never mutated:

variant result
master 5 failed on null, "", " ", "\t", []
the shorter if (!published) 4 failed: swallows the real published: 0, and still misses " ", "\t", []
this PR 9 passed

That middle row is why the guard trims rather than testing truthiness. !published looks equivalent and is not: it treats a legitimate 0 epoch as absent while letting whitespace through.

The four cases that pass on master are controls that must keep passing: an omitted key, a numeric epoch, a string epoch, and 0.

npx jest collector on this branch: 157 passed, 10 suites, against a 148 passed, 9 suites baseline on master. Prettier clean.

The nine new tests discriminate rather than merely pass:

implementation result
master (unfixed) 5 of 9 fail
if (!published) instead of the trimmed-string guard 4 of 9 fail, since " ", "\t" and [] are all truthy and fall through to Number()
this branch 9 of 9 pass

Repository-wide, yarn test is red on master before this change: 21 suites fail on Node 26 where jsonwebtoken reaches for the removed SlowBuffer. That is unrelated to this PR and this PR does not touch it, which is why the numbers above are scoped to npx jest collector.

One case deliberately left

published: false still resolves to 1970, because Number(false) is 0 and `${false}`.trim() is "false". A boolean is outside the documented type for this field, and separating it from 0 needs a typeof check for an input nothing sends. Happy to add it if you would rather the guard be total.

…poch

GET /v1/document/metadata-schema documents this key as
"epoch timestamp in ms | nullable", but Number(null), Number("") and
Number(" ") are all 0, so an explicitly null published stamps the document
1/1/1970 (12/31/1969 west of UTC).

That date is not only cosmetic. The value is plucked into chunkHeaderMeta and
stringifyHeader prepends it as literal <document_metadata> text on every chunk,
which all eight vector providers write, so the model reads the wrong date as
part of the document body.

Guard on a trimmed string so null, "", whitespace and [] are all treated as
absent. published: 0 is a real epoch value and still resolves to 1970.
undefined needs no clause: Number(undefined) is NaN, so it already fell to the
isNaN branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: A null published on /v1/document/raw-text dates the document to the epoch

1 participant