Bug Description
CogneeDataRecord.token_count is typed as number (non-nullable), implying
a valid count is always present. However, calling add() or
addAndCognify() and reading token_count from the immediate return value
consistently shows -1 — even with genuinely fresh, non-duplicate content.
Reproduction
const result = await cognee2.add(
{ type: "text", text: "Some genuinely unique text, never seen before." },
"fresh-dataset-name"
);
console.log(result.added[0].token_count); // -1
Confirmed with is_duplicate=false in the ingestion logs, ruling out
deduplication as the cause:
ingestion::pipeline: input processed data_id=... is_duplicate=false
Same result with addAndCognify():
const { add } = await cognee2.addAndCognify(
{ type: "text", text: "Another unique piece of text." },
"fresh-dataset-name-2"
);
console.log(add.added[0].token_count); // -1
This happens even though addAndCognify() has already run the full
cognify pipeline internally by the time it returns — the returned add
result still reflects -1.
What resolves it
Calling cognify() as a separate step (not combined via
addAndCognify()), then querying datasets.listData() afterward, shows
the correct value:
await cognee2.add({ type: "text", text: "..." }, "dataset-name");
await cognee2.cognify("dataset-name");
const datasets = await cognee2.datasets.list();
const ds = datasets.find(d => d.name === "dataset-name");
const items = await cognee2.datasets.listData(ds.id);
console.log(items[0].token_count); // correct positive value, e.g. 20
Why this matters
The type signature (token_count: number) gives no indication that this
field might hold a placeholder/uncalculated value. A caller reading
token_count right after add() or addAndCognify() — which is a very
natural thing to do, since both return the data record directly — will
silently get -1 instead of the real count, with nothing in the types or
docs to explain why.
Suggested fix
Either:
- Compute
token_count synchronously before returning from
add()/addAndCognify(), so the immediate result is always accurate, or
- Type the field as
number | null (or document that -1 specifically
means "not yet calculated") so callers aren't misled by a seemingly
always-valid, non-nullable type
Environment
- Package: @cognee/cognee-ts@0.1.3
- OS: Windows 11
Happy to help with a README note documenting this behavior (that
token_count may show -1 immediately after add()/addAndCognify(), and
needs a separate listData() call after cognify() completes to get the
real value) if that would be useful — let me know if a PR for that would
be welcome, or if you'd rather fix the underlying computation instead.
Bug Description
CogneeDataRecord.token_countis typed asnumber(non-nullable), implyinga valid count is always present. However, calling
add()oraddAndCognify()and readingtoken_countfrom the immediate return valueconsistently shows
-1— even with genuinely fresh, non-duplicate content.Reproduction
Confirmed with
is_duplicate=falsein the ingestion logs, ruling outdeduplication as the cause:
Same result with
addAndCognify():This happens even though
addAndCognify()has already run the fullcognify pipeline internally by the time it returns — the returned
addresult still reflects
-1.What resolves it
Calling
cognify()as a separate step (not combined viaaddAndCognify()), then queryingdatasets.listData()afterward, showsthe correct value:
Why this matters
The type signature (
token_count: number) gives no indication that thisfield might hold a placeholder/uncalculated value. A caller reading
token_countright afteradd()oraddAndCognify()— which is a verynatural thing to do, since both return the data record directly — will
silently get
-1instead of the real count, with nothing in the types ordocs to explain why.
Suggested fix
Either:
token_countsynchronously before returning fromadd()/addAndCognify(), so the immediate result is always accurate, ornumber | null(or document that-1specificallymeans "not yet calculated") so callers aren't misled by a seemingly
always-valid, non-nullable type
Environment
Happy to help with a README note documenting this behavior (that
token_count may show -1 immediately after add()/addAndCognify(), and
needs a separate listData() call after cognify() completes to get the
real value) if that would be useful — let me know if a PR for that would
be welcome, or if you'd rather fix the underlying computation instead.