Skip to content

token_count returns -1 immediately after add()/addAndCognify() — appears uncalculated at that point, not reflected in the type #99

Description

@akshayWork-19

Bug Description

CogneeDataRecord.token_count is typed as number (non-nullable), implying
a valid count is always present. However, calling add() or
addAndCognify() and reading token_count from the immediate return value
consistently shows -1 — even with genuinely fresh, non-duplicate content.

Reproduction

const result = await cognee2.add(
    { type: "text", text: "Some genuinely unique text, never seen before." },
    "fresh-dataset-name"
);
console.log(result.added[0].token_count); // -1

Confirmed with is_duplicate=false in the ingestion logs, ruling out
deduplication as the cause:

ingestion::pipeline: input processed data_id=... is_duplicate=false

Same result with addAndCognify():

const { add } = await cognee2.addAndCognify(
    { type: "text", text: "Another unique piece of text." },
    "fresh-dataset-name-2"
);
console.log(add.added[0].token_count); // -1

This happens even though addAndCognify() has already run the full
cognify pipeline internally by the time it returns — the returned add
result still reflects -1.

What resolves it

Calling cognify() as a separate step (not combined via
addAndCognify()), then querying datasets.listData() afterward, shows
the correct value:

await cognee2.add({ type: "text", text: "..." }, "dataset-name");
await cognee2.cognify("dataset-name");

const datasets = await cognee2.datasets.list();
const ds = datasets.find(d => d.name === "dataset-name");
const items = await cognee2.datasets.listData(ds.id);

console.log(items[0].token_count); // correct positive value, e.g. 20

Why this matters

The type signature (token_count: number) gives no indication that this
field might hold a placeholder/uncalculated value. A caller reading
token_count right after add() or addAndCognify() — which is a very
natural thing to do, since both return the data record directly — will
silently get -1 instead of the real count, with nothing in the types or
docs to explain why.

Suggested fix

Either:

  • Compute token_count synchronously before returning from
    add()/addAndCognify(), so the immediate result is always accurate, or
  • Type the field as number | null (or document that -1 specifically
    means "not yet calculated") so callers aren't misled by a seemingly
    always-valid, non-nullable type

Environment

  • Package: @cognee/cognee-ts@0.1.3
  • OS: Windows 11

Happy to help with a README note documenting this behavior (that
token_count may show -1 immediately after add()/addAndCognify(), and
needs a separate listData() call after cognify() completes to get the
real value) if that would be useful — let me know if a PR for that would
be welcome, or if you'd rather fix the underlying computation instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions