[BUG] Java: Indexing using multiple threads causing crash

[chatman]

When multiple threads (non-concurrent) are used for building CAGRA indexes, the second such thread crashes as follows:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000077a10bfc2080, pid=208723, tid=208812
#
# JRE version: Java(TM) SE Runtime Environment (22.0.2+9) (build 22.0.2+9-70)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (22.0.2+9-70, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  0x000077a10bfc2080
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/ishan/code/cuvs/java/cuvs-java/core.208723)
#
# An error report file with more information is saved as:
# /home/ishan/code/cuvs/java/cuvs-java/hs_err_pid208723.log
[0.804s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Reproduce: Try running the simple test (CagraBuildAndSearchIT) with multiple threads using the following patch: https://gist.github.com/chatman/620e69cbb73cd0344967dc84f962699f, change the boolean executeInDifferentThreads = true; to false or true.

[chatman]

The /home/ishan/code/cuvs/java/cuvs-java/hs_err_pid208723.log file is here: https://termbin.com/2u0c

FYI @rhdong @cjnolet @narangvivek10 @ChrisHegarty.

[chatman]

Just out of trial and error, this patch passed the above test.
https://termbin.com/x22mb

[chatman]

@ChrisHegarty, does this seem like a correct thing to do? Or, am I missing something bigger?

[chatman]

Interestingly, that test fails/crashes only with the IDE for me (Eclipse), but not via Maven. So, I'm not sure if I was able to reproduce the original problem correctly via the test to begin with. Verifying that..

[chatman]

Ah, doing this resulted in the following error in the downstream application (https://github.com/searchscale/vectorsearch-benchmarks):

[main] INFO com.searchscale.lucene.cuvs.benchmarks.LuceneCuvsBenchmarks - 499 Docs remaining. Calling commit.
Exception in thread "main" org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:916)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:929)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4086)
	at com.searchscale.lucene.cuvs.benchmarks.LuceneCuvsBenchmarks.indexDocuments(LuceneCuvsBenchmarks.java:350)
	at com.searchscale.lucene.cuvs.benchmarks.LuceneCuvsBenchmarks.main(LuceneCuvsBenchmarks.java:162)
Caused by: java.lang.RuntimeException: java.lang.WrongThreadException: Attempted access outside owning thread
	at org.apache.lucene.sandbox.vectorsearch.CuVSVectorsWriter.flush(CuVSVectorsWriter.java:216)
	at org.apache.lucene.index.VectorValuesConsumer.flush(VectorValuesConsumer.java:76)
	at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:305)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:401)
	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:447)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1552)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1837)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1477)
	at com.searchscale.lucene.cuvs.benchmarks.LuceneCuvsBenchmarks.lambda$indexDocuments$2(LuceneCuvsBenchmarks.java:315)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.WrongThreadException: Attempted access outside owning thread
	at java.base/jdk.internal.foreign.MemorySessionImpl.wrongThread(MemorySessionImpl.java:314)
	at java.base/jdk.internal.misc.ScopedMemoryAccess$ScopedAccessError.newRuntimeException(ScopedMemoryAccess.java:113)
	at java.base/jdk.internal.foreign.MemorySessionImpl.checkValidState(MemorySessionImpl.java:209)
	at java.base/jdk.internal.foreign.SegmentFactories.allocateSegment(SegmentFactories.java:142)
	at java.base/jdk.internal.foreign.ArenaImpl.allocateNoInit(ArenaImpl.java:55)
	at java.base/jdk.internal.foreign.ArenaImpl.allocate(ArenaImpl.java:60)
	at java.base/java.lang.foreign.SegmentAllocator.allocate(SegmentAllocator.java:612)
	at com.nvidia.cuvs.internal.panama.CuVSCagraIndexParams.allocate(CuVSCagraIndexParams.java:325)
	at com.nvidia.cuvs.internal.CagraIndexImpl.segmentFromIndexParams(CagraIndexImpl.java:391)
	at com.nvidia.cuvs.internal.CagraIndexImpl.build(CagraIndexImpl.java:159)
	at com.nvidia.cuvs.internal.CagraIndexImpl.<init>(CagraIndexImpl.java:109)
	at com.nvidia.cuvs.internal.CagraIndexImpl$Builder.build(CagraIndexImpl.java:476)
	at com.nvidia.cuvs.internal.CagraIndexImpl$Builder.build(CagraIndexImpl.java:435)
	at org.apache.lucene.sandbox.vectorsearch.CuVSVectorsWriter.createCagraIndex(CuVSVectorsWriter.java:124)
	at org.apache.lucene.sandbox.vectorsearch.CuVSVectorsWriter.flush(CuVSVectorsWriter.java:212)

However, this is happening at a place where there used to be a crash before (using this application).

[ChrisHegarty]

Just focusing on the crash here. This is odd, and appears to be coming from cuvs, or the way that we're interacting with it.

The crash that I see is a # SIGSEGV (0xb) at pc=0x00007ba400073916, pid=59614, tid=59615, with a stack like this:

C  [libcuvs_c.so+0x46916]  rmm::mr::owning_wrapper<rmm::mr::pool_memory_resource<rmm::mr::device_memory_resource>, rmm::mr::device_memory_resource>::do_allocate(unsigned long, rmm::cuda_stream_view)+0x36
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::nep_invoker_blob 0x00007ba4142a3208
j  java.lang.invoke.LambdaForm$MH+0x000000007d1cc000.invoke(Ljava/lang/Object;JJJJJJJJI)J+24 java.base@22.0.2
j  java.lang.invoke.LambdaForm$MH+0x000000007d1d9400.invokeExact_MT(Ljava/lang/Object;JJJJJJJJILjava/lang/Object;)J+35 java.base@22.0.2
j  jdk.internal.foreign.abi.DowncallStub+0x000000007d1cc400.invoke(Ljava/lang/foreign/SegmentAllocator;Ljava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;JJLjava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;I)Ljava/lang/foreign/MemorySegment;+327 java.base@22.0.2
j  java.lang.invoke.LambdaForm$DMH+0x000000007d1cc800.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;JJLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;I)Ljava/lang/Object;+28 java.base@22.0.2
j  java.lang.invoke.LambdaForm$MH+0x000000007d1ce400.invoke(Ljava/lang/Object;Ljava/lang/Object;JJLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;I)Ljava/lang/Object;+62 java.base@22.0.2
j  java.lang.invoke.LambdaForm$MH+0x000000007d1d8c00.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;JJLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object;+33 java.base@22.0.2
j  com.nvidia.cuvs.internal.CagraIndexImpl.build()Lcom/nvidia/cuvs/internal/CagraIndexImpl$IndexReference;+178 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.internal.CagraIndexImpl.<init>(Lcom/nvidia/cuvs/CagraIndexParams;Lcom/nvidia/cuvs/CagraCompressionParams;[[FLcom/nvidia/cuvs/internal/CuVSResourcesImpl;)V+27 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.internal.CagraIndexImpl$Builder.build()Lcom/nvidia/cuvs/internal/CagraIndexImpl;+43 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.internal.CagraIndexImpl$Builder.build()Lcom/nvidia/cuvs/CagraIndex;+1 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.CagraBuildAndSearchIT.lambda$indexAndQueryOnce$0([[F[[FLjava/util/List;Ljava/util/List;)Ljava/lang/Void;+76 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.CagraBuildAndSearchIT$$Lambda+0x000000007d117758.call()Ljava/lang/Object;+16 com.nvidia.cuvs@25.02.0
j  com.nvidia.cuvs.CagraBuildAndSearchIT.testIndexingAndSearchingFlow()V+30 com.nvidia.cuvs@25.02.0

An observation is that a test program creating and searching an index, can execute in a different thread, just that whatever thread does the initial indexing is the only thread that is allowed to do subsequent operations. This is odd, and seems to indicate that we're not releasing or associating resources/streams correctly. ( or some such )

I see the failure, only after the test code create a new index in a thread other than that which created the initial index, which (from debugging output) seems to switch to a new CUDA stream.

[chatman]

@narangvivek10 was able to find the cause of the crash and fix it. Will raise a PR against 25.04 branch soon (I haven't setup 25.04 branch yet, so unable to raise a PR against that branch yet).

[jakirkham]

Closing as fixed by PR: #695