feat(elasticsearch-plugin): avoid redundant reindex on stock movements

[LeftoversTodayAppAdmin]

Closes #50.

Adds two opt-in options to reduce search index churn. Both default to the existing behaviour, so current installs see no change until they enable them.

Options

• reindexOnStockMovement ('always' | 'onStockStatusChange', default 'always'): Controls whether a stock change enqueues an index update job. It covers order-driven StockMovementEvents (allocations, sales, cancellations, releases, adjustments) and admin variant updates whose input touches only stock fields (stockOnHand, stockLevels, trackInventory, outOfStockThreshold, useGlobalOutOfStockThreshold). With 'onStockStatusChange' the plugin checks before creating a job whether any moved variant's inStock would flip, and skips the job if not. A 50 to 49 sale that leaves the item in stock no longer costs a queue write, a poll and a worker cycle.
• incrementalIndexUpdates (boolean, default false): When enabled, a product update reads the product's indexed documents, compares them with the freshly built ones, and writes only the difference. Changed or new documents are written with a full-document index replace, documents that no longer exist are deleted, and nothing is written when nothing changed. A still-current document is never deleted first, so the product does not drop out of search during an update. A product with more documents than the result window, or a failed index read, falls back to the existing delete-then-recreate path. A full reindex is unchanged.

How the stock check works

• Scope: A variant's inStock can only flip if its own stock moved, and productInStock can only flip if some variant's inStock flipped. So the check only looks at the moved variants.
• Cost: One query loads the moved variants, one search (a terms query on productVariantId with a narrow _source) reads their indexed inStock values, and the saleable stock level is computed once per moved variant per indexed channel. For a single order line in one channel that is three small SQL queries and one search, regardless of product size.
• Fallbacks: The check enqueues whenever it cannot decide: a missing or soft-deleted variant, a variant with no indexed document, a document for a channel the variant is no longer in, a result that may be truncated, or any error.
• Custom mappings: If customProductMappings or customProductVariantMappings is configured, a custom field could derive from stock without an inStock flip, so the check turns itself off and every stock change enqueues as before.
• Trade-off: Sibling variants are not recomputed, so the check does not notice a productInStock that is already stale in the index for an unrelated reason. Fixing a stale index is left to a reindex or the next non-stock update.

Backwards compatibility

• Defaults: reindexOnStockMovement defaults to 'always' and incrementalIndexUpdates to false, so with no config change the write path and the job behaviour are the same as today.
• Writes: The reindex and default paths keep the original update plus doc_as_upsert operations. Only the incremental path uses the index replace.
• API: No GraphQL or public method signatures change.

Implementation notes

• Diffing: The document diff lives in src/indexing/index-diff.ts. It compares the built document as the index stores it (JSON round trip, so undefined fields, Dates and key order are handled) using the existing fast-deep-equal dependency.
• Stock-only updates: src/indexing/stock-only-update.ts decides whether a ProductVariantEvent 'updated' input touched only stock fields.
• Streaming: The reindex and default paths build documents through a visitor, so a large product is streamed in chunks instead of buffered.
• Shared logic: The guard and the document builder share one computeVariantInStock helper, so they cannot compute inStock differently.

Tests

• Unit: Guard tests cover flip and no flip, multi-channel, a missing document, truncation, errors, the 'always' and custom mapping short circuits, and a regression check that only the moved variant is loaded and evaluated once per channel. There are also tests for the diff helper and the stock-only update detection.
• e2e: A stock-guard spec places real orders to show that a 50 to 49 allocation creates no job and that a sell-out reindexes and flips inStock and productInStock in search. It also covers admin stock edits and non-stock edits. An incremental spec covers in-place writes, no-op updates and field removal. The default suite still covers delete-then-recreate.
• Backends: The suite passes against Elasticsearch 9.1.0 and OpenSearch 3.0.0.

[biggamesmallworld]

Nice approach on skipUnchangedIndexUpdates. Diffing against the real builder output is the right call.

Three blockers:

1. The refactor dropped bulk-operation chunking. Hits everyone, including full reindex, with both options off. See inline.
2. Please drop reindexOnStockMovement. Your description says skipUnchangedIndexUpdates is correct for any config and covers any trigger, which makes the other one a subset that's only correct sometimes. Add a stock-derived custom mapping later and you get a silently stale index. It also does an ES search plus a Product query plus per-variant getSaleableStockLevel in the subscriber, so order placement pays for it. Dropping it takes productStockStatusDiffersFromIndex and half the PR with it.
3. stableStringify reimplements fast-deep-equal, which we already depend on, and it's buggy. See inline.

Also missing @since on both options, and elasticsearch-options.mdx needs regenerating.

[biggamesmallworld]

Both mid-build flushes are gone, so this buffers every operation for a product before returning. The description says writes are still chunked, which is true, but the build isn't anymore.

Count is channels × languages × variants × 2, each holding a full VariantIndexItem. 5000 variants, 3 channels, 4 languages = 120k buffered docs. reindex() at line 388 goes through here, so full reindex on a big catalog can now OOM. No opt-in needed.

The diff path needs them in memory, that's fine, it's opt-in. But updateProductsOperationsOnly has to keep streaming. A flush callback would do it.

[biggamesmallworld]

fast-deep-equal is already in package.json and imported in elasticsearch.service.ts:19.

Also: stableStringify(new Date()) returns {}. A custom mapping returning a Date never matches its own _source (an ISO string), so the skip never fires and the option quietly does nothing. Tests only cover primitives and plain objects.

equal(JSON.parse(JSON.stringify(doc)), hit._source) handles undefined, Dates, and key order, and compares exactly what gets stored.

[biggamesmallworld]

buildProductVariantOperations has the id and the doc at construction time. Return { operations, documentsById } and this function plus its four tests go away. Array<{ operation: any }> also throws away the index field.

[biggamesmallworld]

This recomputes stock status differently from createVariantIndexItem and they already disagree. ProductVariant.deletedAt is a plain @Column, not @DeleteDateColumn, so line 249 pulls in soft-deleted variants. The builder filters them and also force-disables everything when product.enabled is false, which changes productInStock.

Both differences fail safe today. The issue is that isProductIndexUnchanged gets this right by calling the real builder, and this one will go stale next time createVariantIndexItem changes with nothing to catch it.

[biggamesmallworld]

10000 is the default index.max_result_window, so this truncates at the ceiling with no error. Same literal at 795.

Safe there (fewer hits = mismatch = write), not here: dropped hits mean dropped channel buckets, if (!channelDocs) continue skips them, guard says unchanged. Named constant plus handle the boundary. Both call sites are the same term: { productId } search too.

[biggamesmallworld]

These pass with the flag off: they assert search is correct, which it already was. Nothing checks a write was skipped.

Sentinel makes it deterministic: write an extra field into the index doc, run the no-op update, assert it survived. Proves the delete-then-recreate didn't happen.

it('finds an in-stock variant to exercise') is a beforeAll in disguise, and the next two break with undefined if it fails. Fixture is fixed, so assert the exact variant. take: 200 will silently truncate once the fixture grows.

Flipping the whole suite also means these 96 tests stop covering the default path. The uuid spec still does, but that's 7 tests. Prefer a targeted describe.

[vendure-developer-hub]

Community Plugins — View preview

_125b099

[LeftoversTodayAppAdmin]

Thanks for the thorough review. I reworked the PR around what turned out to be the real issue, the delete-then-recreate, and made every new behaviour opt-in, so with the defaults unchanged the plugin behaves exactly as before. Nothing changes for current installs until a flag is set.

Two opt-in options.

• incrementalIndexUpdates (default false). When on, an update reads the product's current documents, compares them against the freshly built ones, and writes only the difference: it replaces changed and new documents in place, deletes the ones that no longer exist, and writes nothing when nothing changed. A live document is never removed first, so the product stays searchable during an update. It compares the whole document, so it is correct for any mapping config. A full reindex still writes everything, and a product too large to diff falls back to delete-then-recreate. With the flag off, the write path is the historic one.
• reindexOnStockMovement (default 'always'). With 'onStockStatusChange' the plugin checks, before enqueuing, whether a stock change would flip inStock or productInStock and skips the job when it would not. This covers both order-driven StockMovementEvents and admin variant updates that touched only stock fields, since ProductVariantEvent carries the update input.

Your specific points.

1. Chunking and reindex OOM. Fixed. The builder is now a visitor that yields one document at a time, and the reindex and default paths stream and flush in chunks, so a full reindex never buffers a whole large product. Only the incremental path collects to diff, capped at the result window with a fallback to streaming.

2. stableStringify. Gone, replaced by the existing fast-deep-equal, comparing the built document as the index stores it. The JSON round-trip handles undefined, Dates and key order, which was the Date bug. The diff is a pure function in index-diff.ts with unit tests. targetDocumentsById is gone too, since the builder hands the id and document straight to the diff.

Magic number. Both index reads use a named INDEX_MAX_RESULT_WINDOW and fall back to a full write if a read reaches the window.

@SInCE and docs. Added to both options. Documented both options and how to enable them on the plugin page and regenerated the reference docs. Heads up that the regen also picked up earlier drift from the adapter refactor, since the committed docs were stale.

Tests. The default suite runs with both flags off, so it still covers the historic path. incrementalIndexUpdates has its own spec that reads the document directly rather than paging with take: 200, asserts the exact fixture variant, and proves a skip by checking the document _version stays put on a no-op and increments on a real change. A separate stock-guard spec with no custom mappings proves a non-flipping admin stock edit creates no write while a flipping or non-stock edit does. I ran the full suite locally against both Elasticsearch 9.3.4 and OpenSearch 3.7.0, all green.

On point 2, keeping reindexOnStockMovement. With correctness owned by the whole-document paths, this is a pre-enqueue optimization. It self-disables when a custom mapping is configured, so the stock-booleans-only check runs only when those booleans are provably the only stock-derived fields, and adding a mapping later cannot cause a stale index. The recompute that could drift is gone, both paths share one helper, and it no longer does an index search or a heavy product load on the order path. It is not redundant with the write-path guard: that one runs inside the worker, so by the time it decides nothing changed, the job row is written, the poll waited, and the worker ran. On a database-backed queue that per-change job is the dominant cost, and stock changes are the highest-frequency event in a busy catalog, so skipping the job keeps the queue shallow and lets real updates reach the index sooner.

Thanks again for the careful read. Everything is opt-in, so it is safe to merge and enable per deployment.

[LeftoversTodayAppAdmin]

Hi @biggamesmallworld please review the latest changes! Thank you.

[michaelbromley]

Thanks for the rework. The incremental write path looks good to me: the historic behaviour is the default, it falls back safely, and the diff logic is unit tested. I'm OK with keeping reindexOnStockMovement, but the check it runs needs to scale with the stock movement, not with the size of the product.

The cost today

productStockStatusDiffersFromIndex runs on the server for every StockMovementEvent, and an order typically fires two or three of those (allocation, sale, and cancellation if any). For each product in the event it:

• searches the index for all of the product's documents,
• loads the product with all its variants and their channels,
• calls getSaleableStockLevel for every variant in every channel, then again for every enabled variant inside getProductInStockValue. Each call is a StockLevel query.

That is roughly 2 × variants × channels queries per product per event, run one after another:

Product	Queries per event
5 variants, 1 channel	~13
20 variants, 1 channel	~43
200 variants, 3 channels	~1,200

The subscriber doesn't block the order request, but it shares the database pool with customer requests, so for large products at peak traffic it competes with checkout. (The Sep 13 reply says the guard no longer does an index search or a product load on the order path, but it still does both.)

Moving the check to the worker wouldn't help. The worker uses the same database, and the point of the option is to avoid creating the job.

Proposal: only check the variants that moved

• A variant's inStock can only flip if its own stock moved.
• productInStock can only flip if some variant's inStock flipped.

So if none of the moved variants flipped, nothing the guard cares about changed, and the rest of the product never needs loading. The check becomes:

1. Load the moved variants in one query, for the trackInventory and threshold fields that getSaleableStockLevel needs.
2. Run one search with a terms query on productVariantId for those variants, with _source: ['productVariantId', 'channelId', 'inStock'].
3. For each (variant, channel) pair found, set the channel on the context and call computeVariantInStock once. Return true on the first mismatch, or if a moved variant has no indexed document.

That is one variant query, one search, and one stock query per moved variant per channel. A single order line goes from 13+ queries to about 3, and the cost no longer grows with the size of the product. The product load and getProductInStockValue drop out of the guard, and so does the hand-copied soft-delete and disabled-product logic, which also settles my earlier concern about it drifting from createVariantIndexItem.

One trade-off: the current version also notices a productInStock that is already wrong in the index for some unrelated reason, and the narrower check won't. I think that's fine, since repairing a stale index isn't this guard's job, but it's worth a line in the doc comment.

updateVariantsForVariantEvent uses the same guard, so it gets the same improvement.

Also before merge

• The PR description still describes skipUnchangedIndexUpdates, stableStringify and targetDocumentsById. Please update it to match the current design.
• Both options say @since 2.2.0 while the package is at 2.0.0. Please confirm that's the intended release.

With the guard scoped to the moved variants, I'm happy to approve.

[LeftoversTodayAppAdmin]

Thank you @michaelbromley, that makes sense. I've changed the guard to check only the moved variants, as you suggested:

• One query loads the moved variants with their channels.
• One search, a terms query on productVariantId returning only productVariantId, channelId and inStock.
• One getSaleableStockLevel call per moved variant per indexed channel, returning true on the first mismatch or when a moved variant has no indexed document.

A single order line in one channel is now three small SQL queries and one search, however many variants the product has. The product load, getProductInStockValue and the copied soft-delete and disabled-product logic are gone from the guard. It still enqueues when it can't decide (missing or soft-deleted variant, possibly truncated result, any error), and the custom mapping fallback is unchanged. The doc comment notes the stale productInStock trade-off. updateVariantsForVariantEvent uses the same method, so it gets this too.

You were right about my Sep 13 reply, sorry about that. The guard still did both the search and the product load at that point. Now there's no product load, and the one search covers only the moved variants.

Tests: there's a unit regression check that with 20 variants across 2 channels and one moved, only that variant is loaded and its stock is checked once per channel. The e2e spec now places real orders: a 50 to 49 allocation creates no job, and a sell-out reindexes and flips inStock and productInStock.

The PR description is updated to match the current design. On @SInCE, I kept 2.2.0 because options.ts already has 2.1.7 tags from the core plugin's history, so 2.1.0 would look older than existing options. Happy to switch to 2.1.0 if you prefer.