fix(core): prevent redundant re-embedding in incremental indexing with batch_size

[Alon Naor (AlonNaor22)]

Summary

• Fixes Chunks after batch_size treated as non-existant #32612
• When using index()/aindex() with cleanup="incremental" and batch_size smaller than the total number of documents sharing the same source_id, the per-batch incremental cleanup prematurely deleted records from later batches that hadn't been processed yet
• This caused all subsequent batches to re-embed and re-add documents on every single run, wasting significant compute (embedding API calls / GPU time)
• The fix moves the incremental cleanup from inside the batch loop to after all batches complete, collecting source IDs during iteration — ensuring all documents are updated in the record manager before stale ones are identified and deleted

Files changed

• libs/core/langchain_core/indexing/api.py — moved incremental cleanup block from inside the batch loop to after the loop, for both index() and aindex()
• libs/core/tests/unit_tests/indexing/test_indexing.py — updated existing test to expect correct behavior; added 3 new regression tests (sync, async, stale-docs-still-cleaned-up)

Test plan

• Updated test_incremental_indexing_with_batch_size to expect {num_added: 0, num_deleted: 0, num_skipped: 4} instead of the previous {num_added: 2, num_deleted: 2, num_skipped: 2}
• Added test_incremental_indexing_with_batch_size_no_redundant_work — 10 docs, batch_size=3, same source_id, verify 3 consecutive runs all skip without redundant work
• Added test_aincremental_indexing_with_batch_size_no_redundant_work — async equivalent
• Added test_incremental_indexing_stale_docs_still_cleaned_up — verifies that when document content actually changes, old versions are still properly deleted
• All 44 indexing tests pass (2 deselected are pre-existing failures in test_index_into_document_index unrelated to this change)
• ruff check and ruff format clean

🤖 Generated with Claude Code

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 13 untouched benchmarks
⏩ 23 skipped benchmarks¹

---

_{Comparing AlonNaor22:fix/incremental-indexing-batch-size-bug (356d491) with master (29134dc)}

Footnotes

1. 23 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩