Do you need to file an issue?
Describe the bug
_merge_nodes_then_upsert merges an existing node's stored description with the newly extracted ones via already_description + sorted_descriptions (lightrag/operate.py:~2167), deduping only WITHIN the new batch, not between stored and new. The same is true for relations in _merge_edges_then_upsert (operate.py:~2525). Both feed _handle_entity_relation_summary, which — below the force-summary threshold — simply separator.join(...) without cross-dedup (operate.py:~335).
So re-running merge for a doc whose entities/relations already exist (any reprocess or resume) re-appends the same descriptions. Each reprocess reads the full stored description back into already_description and adds one more copy, so the fragment count grows by one per reprocess (N -> N+1): a single failed-then-retried entity goes 1 -> 2 (looks like a doubling), and repeated retries accumulate linearly.
Both reprocess paths do purge first (pipeline resume via _purge_stale_extraction_if_resuming; ainsert_custom_chunks via _purge_doc_chunks_and_kg), but purge discovers targets via the full_entities/full_relations indexes, which merge writes only in its final phase (operate.py:~3253). A merge that wrote partial graph state then failed leaves no index row -> purge finds nothing -> the retry accumulates.
Scope: source_id/file_path are deduped and safe; only descriptions accumulate. It is bounded (later re-summarization collapses it) but silently inflates entity/relation descriptions by one duplicate copy per failed-then-retried reprocess across all ingestion paths, and each reprocess re-reads the whole stored description so later merges also do more token work. Affects the file pipeline, ainsert, and ainsert_custom_chunks alike (all share merge_nodes_and_edges).
Steps to reproduce
Minimal, no DB/LLM — drives the real merge round-trip (get_node -> merge -> upsert_node) against an in-memory graph. Verified on the latest main.
import asyncio
import lightrag.operate as operate
from lightrag.operate import _merge_nodes_then_upsert, _handle_entity_relation_summary
from lightrag.constants import GRAPH_FIELD_SEP
class FakeTokenizer:
def encode(self, s): # token count == char count; thresholds kept slack
return list(range(len(s)))
class MemGraph: # minimal in-memory graph: real read-back + write round-trip
def __init__(self):
self.nodes = {}
async def get_node(self, name):
return self.nodes.get(name)
async def upsert_node(self, name, node_data):
self.nodes[name] = dict(node_data)
cfg = {
"tokenizer": FakeTokenizer(),
"summary_context_size": 1_000_000,
"summary_max_tokens": 1_000_000,
"force_llm_summary_on_merge": 6,
"source_ids_limit_method": operate.SOURCE_IDS_LIMIT_METHOD_KEEP,
"max_source_ids_per_entity": 10_000,
"max_file_paths": 100,
"file_path_more_placeholder": "...",
}
D = "Alice is a software engineer at Acme."
batch = {"entity_name": "ALICE", "entity_type": "person",
"description": D, "source_id": "chunk-1", "file_path": "doc1.txt", "timestamp": 1}
async def main():
g = MemGraph()
for _ in range(3): # 3 reprocesses of the SAME single description, no purge between
await _merge_nodes_then_upsert("ALICE", [dict(batch)], g, None, cfg)
frags = g.nodes["ALICE"]["description"].split(GRAPH_FIELD_SEP)
print(f"stored fragments: {len(frags)}")
joined, _ = await _handle_entity_relation_summary(
"Relation", "ALICE~ACME", [D, D], GRAPH_FIELD_SEP, cfg, None)
print("summary([D, D]) fragments:", len(joined.split(GRAPH_FIELD_SEP)))
asyncio.run(main())Expected Behavior
Re-merging a doc whose entities/relations already exist should be idempotent w.r.t. descriptions: a description already stored on a node/edge should not be appended again. Reprocessing the same content any number of times should leave the fragment count at 1 (for one unique description), not grow by one per reprocess.
LightRAG Config Used
The accumulation is independent of user config; it reproduces with default merge settings. The knobs that matter (kept generous above so the no-LLM join branch is exercised):
force_llm_summary_on_merge: 6
summary_max_tokens: 1000000
summary_context_size: 1000000
source_ids_limit_method: KEEP
max_source_ids_per_entity: 10000
Logs and screenshots
stored fragments: 1
stored fragments: 2
stored fragments: 3
summary([D, D]) fragments: 2
Node description after 3 reprocesses (real upsert_node value):
'Alice is a software engineer at Acme.<SEP>Alice is a software engineer at Acme.<SEP>Alice is a software engineer at Acme.'
Additional Information
Do you need to file an issue?
Describe the bug
_merge_nodes_then_upsertmerges an existing node's stored description with the newly extracted ones viaalready_description + sorted_descriptions(lightrag/operate.py:~2167), deduping only WITHIN the new batch, not between stored and new. The same is true for relations in_merge_edges_then_upsert(operate.py:~2525). Both feed_handle_entity_relation_summary, which — below the force-summary threshold — simplyseparator.join(...)without cross-dedup (operate.py:~335).So re-running merge for a doc whose entities/relations already exist (any reprocess or resume) re-appends the same descriptions. Each reprocess reads the full stored description back into
already_descriptionand adds one more copy, so the fragment count grows by one per reprocess (N -> N+1): a single failed-then-retried entity goes 1 -> 2 (looks like a doubling), and repeated retries accumulate linearly.Both reprocess paths do purge first (pipeline resume via
_purge_stale_extraction_if_resuming;ainsert_custom_chunksvia_purge_doc_chunks_and_kg), but purge discovers targets via thefull_entities/full_relationsindexes, which merge writes only in its final phase (operate.py:~3253). A merge that wrote partial graph state then failed leaves no index row -> purge finds nothing -> the retry accumulates.Scope:
source_id/file_pathare deduped and safe; only descriptions accumulate. It is bounded (later re-summarization collapses it) but silently inflates entity/relation descriptions by one duplicate copy per failed-then-retried reprocess across all ingestion paths, and each reprocess re-reads the whole stored description so later merges also do more token work. Affects the file pipeline,ainsert, andainsert_custom_chunksalike (all sharemerge_nodes_and_edges).Steps to reproduce
Minimal, no DB/LLM — drives the real merge round-trip (
get_node-> merge ->upsert_node) against an in-memory graph. Verified on the latestmain.Expected Behavior
Re-merging a doc whose entities/relations already exist should be idempotent w.r.t. descriptions: a description already stored on a node/edge should not be appended again. Reprocessing the same content any number of times should leave the fragment count at 1 (for one unique description), not grow by one per reprocess.
LightRAG Config Used
The accumulation is independent of user config; it reproduces with default merge settings. The knobs that matter (kept generous above so the no-LLM join branch is exercised):
Logs and screenshots
Node description after 3 reprocesses (real
upsert_nodevalue):Additional Information
main)already_descriptionagainst the new descriptions in_merge_nodes_then_upsert/_merge_edges_then_upsert, making re-merge idempotent regardless of whether purge found the prior index rows.