Description

Summary

IndexWriter#abortMerges() sets the abort flag on running merges and waits for them to finish, relying on merges observing the flag periodically (the comment in abortMerges says "It should not take very long because they periodically check if they are aborted."). That assumption holds for the I/O-bound merge phases, because every write goes through the merge rate limiter, which checks for abort. Since #16264 / #16281, checkIntegrity() is also covered.

HNSW graph construction breaks this assumption: HnswGraphBuilder#addVectors is a pure-CPU loop that performs no writes and never checks OneMerge#isAborted(). For large vector segments this phase runs for 10–25 minutes single-threaded, during which rollback() / abortMerges() block unconditionally.

We believe this is now the last significant blind spot in merge abort handling, and it is the dominant one for vector-heavy indexes: in our measurements graph construction is 93.6% of merge CPU, while the checksum phase addressed by #16281 is 0.1%.

Production impact

Elasticsearch 9.4.2 (Lucene 10.4.0) cluster, HNSW-indexed dense vectors (1M × 384d float32 + 575k × 96d quantized per shard). During a routine rolling restart, each data node's shutdown blocked on abortMerges waiting for an in-flight HNSW merge, for 24–31 minutes per node, exceeding the 30-minute termination grace period, so pods had to be force-killed.

Aborting thread:

"elasticsearch[...][generic][T#…]" TIMED_WAITING
   at java.lang.Object.wait0(java.base/Native Method)
   at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:5571)
   at org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2760)
   at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2514)
   at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2483)

The single merge thread it waits for (RUNNABLE the whole time, no writes):

"elasticsearch[...][merge][T#3]" RUNNABLE
   ... (vector scoring frames elided)
   at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:342)
   at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeInternal(HnswGraphBuilder.java:312)
   at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:354)
   at org.apache.lucene.util.hnsw.HnswGraphBuilder.addVectors(HnswGraphBuilder.java:233)
   at org.apache.lucene.util.hnsw.HnswGraphBuilder.build(HnswGraphBuilder.java:201)
   at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:218)
   at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.mergeOneField(Lucene99HnswVectorsWriter.java:449)
   at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:126)
   at org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105)
   at org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271)
   ...
   at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5323)
   at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4784)

Measurements

Instrumenting one incident-class merge (TieredMergePolicy deletes-purging rewrite of a ~3 GB segment, 1.9M docs, JFR + async-profiler):

Graph construction = 93.6% of merge CPU samples (higher than its wall-time share because the copy phases are I/O-bound). main has the same shape: HnswGraphBuilder#addVectors contains no abort/cancellation hook.

A standalone reproduction below (lucene-core only, 160k × 384d vectors): rollback() called 8s into a forceMerge blocks for 68.6s on stock 10.4.0, which is the entire remaining graph build.

import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.ConcurrentMergeScheduler;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.NoMergePolicy;
import org.apache.lucene.store.FSDirectory;

/** rollback() while an HNSW merge is building its graph blocks until the build finishes. */
public class MiniRepro {
  public static void main(String[] args) throws Exception {
    int dim = 384, segs = 16, perSeg = 10000;
    long sleepMs = args.length > 0 ? Long.parseLong(args[0]) : 8000;
    Path dir = Files.createTempDirectory("hnsw-abort-repro");

    try (FSDirectory d = FSDirectory.open(dir)) {
      IndexWriterConfig cfg = new IndexWriterConfig();
      cfg.setUseCompoundFile(false);
      cfg.setMergePolicy(NoMergePolicy.INSTANCE);
      try (IndexWriter w = new IndexWriter(d, cfg)) {
        Random r = new Random(42);
        for (int s = 0; s < segs; s++) {
          for (int i = 0; i < perSeg; i++) {
            Document doc = new Document();
            float[] v = new float[dim];
            for (int j = 0; j < dim; j++) v[j] = r.nextFloat();
            doc.add(new KnnFloatVectorField("v", v));
            w.addDocument(doc);
          }
          w.flush();
        }
        w.commit();
      }

      IndexWriterConfig cfg2 = new IndexWriterConfig();
      cfg2.setMergeScheduler(new ConcurrentMergeScheduler());
      IndexWriter w2 = new IndexWriter(d, cfg2);
      long m0 = System.nanoTime();
      Thread bg = new Thread(() -> {
        try {
          w2.forceMerge(1);
          System.out.printf("[bg] forceMerge completed normally in %.1fs%n",
              (System.nanoTime() - m0) / 1e9);
        } catch (Throwable t) {
          System.out.printf("[bg] forceMerge exception after %.1fs: %s%n",
              (System.nanoTime() - m0) / 1e9, t);
        }
      });
      bg.start();
      Thread.sleep(sleepMs);
      System.out.printf("calling rollback() at t=%.1fs after merge start%n",
          (System.nanoTime() - m0) / 1e9);
      long r0 = System.nanoTime();
      w2.rollback();
      System.out.printf("### rollback() returned in %.2fs ###%n", (System.nanoTime() - r0) / 1e9);
      bg.join();
    }
  }
}

calling rollback() at t=8.0s after merge start
### rollback() returned in 68.63s ###

calling rollback() at t=8.0s after merge start
[bg] forceMerge exception after 8.2s: ... MergeAbortedException ... [ABORTED]
### rollback() returned in 0.20s ###

Prior reports

• Merges should periodically check for abort [LUCENE-10658] #11694 (LUCENE-10658, 2022): general request for abortable merges, still open. Elasticsearch's side (IndicesClusterStateService#removeIndices blocks on IO while aborting merges elastic/elasticsearch#88055) was closed pointing to it.
• Merges sometimes do lots of work even after being aborted #13354: same pathology, closed after Check if merge is aborted before executing file integrity checks #16264 / Check if merge is aborted during file integrity checksums #16281 made checksumming abortable, which is 0.1% of this workload.
• Add mergeProgress into MergeState for abort in mergeMiddle #13822: proposed plumbing OneMergeProgress into MergeState for exactly this scenario (HNSW stack in the report), but received no review and its plumbing is now superseded by MergeState#oneMerge (Check if merge is aborted before executing file integrity checks #16264).

Proposed fix

Extend the pattern established by #16264 / #16281 to the graph build path:

1. The plumbing already exists on main: MergeState#oneMerge and the null-safe MergeState#checkAborted() (Check if merge is aborted before executing file integrity checks #16264), which KnnVectorsWriter#merge already invokes once per reader before merging. What's missing is handing that same check down from Lucene99HnswVectorsWriter#mergeOneField through the HNSW graph merger into the builders.
2. In HnswGraphBuilder#addVectors, invoke it periodically, e.g. every 1024 nodes. Each node insertion costs hundreds of vector comparisons, so the overhead is unmeasurable. (This and step 3 could likely be consolidated into a single check in the shared node-insertion path addGraphNode/addGraphNodeInternal. Happy to follow whichever reviewers prefer.)
3. Cover the incremental-merge path too (MergingHnswGraphBuilder#build's remainder-insertion loop and updateGraph), which inserts via addGraphNode directly.
4. Same null contract as checkIntegrity(OneMerge): flush-time graph builds pass null and are unaffected.

We validated the approach with a patched 10.4.0 in the affected cluster (same periodic check in addVectors, wired through a ThreadLocal shim only because 10.4 predates MergeState#oneMerge, while the proposal above uses the explicit plumbing instead). Node shutdown during a live graph build went from 24–31 min (grace exceeded, force kill) to 2.2 s, measured between the node's "stopping" and "stopped" log lines. Aborted merges clean up through the normal MergeAbortedException path, and undisturbed merges complete unchanged.

Happy to contribute a PR with a test (rollback during HNSW merge must abort promptly) if this direction sounds right.

Version and environment details

Lucene 10.4.0 (bundled in Elasticsearch 9.4.2). addVectors on main also lacks abort checks.
OpenJDK 26.0.1, Linux aarch64, single-threaded merges (no intra-merge executor).