Fix IndexWriter.Dispose leaking file handles, #1284#1311
Open
paulirwin wants to merge 19 commits into
Open
Conversation
NightOwl888
requested changes
May 29, 2026
NightOwl888
reviewed
May 29, 2026
NightOwl888
left a comment
NightOwl888
left a comment
Contributor
There was a problem hiding this comment.
This comment may take precedent over many of the other comments, since it calls into question whether there is any real benefit to porting CountDownLatch.
4 tasks
4 tasks
apache#1284 Backport of LUCENE-5871 (upstream commit 2cfcdcc on branch_4x). If anything before the cleanup steps in the close path throws — for example CommitInternal due to an I/O failure — the old straight-line CloseInternal skipped DropAll, deleter.Dispose(), and writeLock.Dispose(), leaking the reader pool, deleter, and write.lock file handle. Replaces CloseInternal with a Shutdown(bool) method that runs Flush, FinishMerges, CommitInternal, then RollbackInternal in a try/finally so RollbackInternal always runs and releases all resources. Also moves the numDocsInRAM accounting in DocumentsWriter into a finally block so a partially-applied addDocument is still counted correctly. Tests added: - TestHasUncommittedChangesAfterException, TestDoubleClose, TestRollbackThenClose, TestCloseThenRollback, TestRollbackWhileMergeIsRunning, TestCloseDuringCommit (TestIndexWriter) - TestOutOfMemoryErrorRollback (TestIndexWriterExceptions) - TestDisposeDoesNotLeakWriteLockOnCommitFailure ([LuceneNetSpecific] regression test that explicitly verifies write.lock is released after a CommitInternal IOException) TestOtherFiles2 was removed in the upstream commit — its assertion no longer holds because a graceful close now checkpoints unreferenced files. Kept commented out at its original location for porting reference. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Follow-up to 7ec8790. Audit against upstream 2cfcdcc found four hunks from the same commit that weren't ported and three doc/annotation changes on the close() overloads. This commit closes all seven gaps. Test/test-framework hunks (from 2cfcdcc): - TestIndexWriterCommit.TestPrepareCommitThenClose: new test, pins the Shutdown contract that close() with an outstanding prepareCommit throws IllegalStateException. - RandomIndexWriter.Dispose: add `IndexWriter.IsClosed == false` guard around the 1-in-8 random force-merge. Without it the new Shutdown contract can leave the writer closed when the test's user-visible call threw, and the force-merge in teardown then throws AlreadyClosedException. - TestIndexWriterExceptions: w.Commit() before w.Dispose() in the random- close branch so Shutdown doesn't throw IllegalStateException there. New test file (from LUCENE-5644 at the 4.8.1 release point, with the 2cfcdcc tweaks layered on TestManyThreadsClose): - TestIndexWriterThreadsToSegments: four tests covering segment-flush- per-thread invariants. Uses the 4.8.1-era APIs (NoMergePolicy.NO_COMPOUND_FILES, Lucene46 codec, 4-arg SegmentCommitInfo). IndexWriter doc + annotation changes (from 2cfcdcc): - Dispose(): replace the pre-fix doc text (which described the old "write lock may remain held / call close() again" recovery flow) with the new two-cases bullet list. - Dispose(bool): replace the OOME warning with a cross-reference to Dispose() for exception behavior; expand the merge-starvation note. - Dispose(bool): mark [Obsolete] mirroring upstream's @deprecated. The message and intent are identical: prefer Commit() + Rollback() to abort merges and then close. LUCENE-5871 and LUCENE-5644 both first appear in 4.10.0 (post-4.8.1), but are in scope because the project tracks branch_4x at 4.8.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pache#1284 Follow-up to f404981. Renames the four extracted Java anonymous Thread classes in TestIndexWriterThreadsToSegments to match the Lucene.NET convention of <BaseType>AnonymousClass with a method-specific suffix when needed: - SegmentCountOnFlushBasicThread -> ThreadAnonymousClassForSegmentCountOnFlushBasic - SegmentCountOnFlushRandomThread -> ThreadAnonymousClassForSegmentCountOnFlushRandom - ManyThreadsCloseThread -> ThreadAnonymousClassForManyThreadsClose - DocsStuckInRAMForeverThread -> ThreadAnonymousClassForDocsStuckInRAMForever CheckSegmentCount is left as-is because it is a named static class in upstream Java, not an anonymous inner class. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…vals, apache#1284 Upstream 2cfcdcc rewrote close(boolean) to delegate to shutdown(boolean) but kept assertEventQueueAfterClose() and closeInternal(boolean, boolean) as private dead code. The original apache#1284 port (7ec8790) dropped both, which matches trunk LUCENE-4246's eventual cleanup (8559eaf, Lucene 5.0) but diverges from the conservative branch_4x state at 2cfcdcc. Adds two LUCENENET breadcrumbs around ShouldClose() so a future porter diffing against 2cfcdcc can find the removal rationale at the position where each method used to live: - Full note before ShouldClose() covering both, since assertEventQueueAfterClose() sat there in upstream. - One-liner after ShouldClose() pointing back to the full note, since closeInternal() sat immediately after. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apache#1284 Follow-up to 08d82e8. The previous rename pass only covered the four classes in TestIndexWriterThreadsToSegments and missed three classes that the original apache#1284 port extracted from anonymous inner classes in TestIndexWriter: - RollbackWhileMergeInfoStream -> InfoStreamAnonymousClassForRollbackWhileMergeIsRunning - RollbackWhileMergeScheduler -> ConcurrentMergeSchedulerAnonymousClassForRollbackWhileMergeIsRunning - SlowCommittingInfoStream -> InfoStreamAnonymousClassForCloseDuringCommit The ForCloseDuringCommit suffix matches the existing ThreadAnonymousClassForCloseDuringCommit precedent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a Lucene.Net.Support.Threading.CountDownLatch class as a 1:1 port of java.util.concurrent.CountDownLatch (adapted from Apache Harmony), along with a JSR166-style CountDownLatchTest port. Why this exists alongside System.Threading.CountdownEvent: - CountdownEvent.Signal() throws InvalidOperationException if called after the count reaches zero. Java's CountDownLatch.countDown() is idempotent at zero. Test code ported from upstream Lucene Java routinely calls countDown() past zero (e.g. one thread per loop iteration when only the first N matter); the C# port has to either wrap each call in a CurrentCount > 0 guard or eat sporadic exceptions during teardown. - This bug was hit by TestConcurrentMergeScheduler.TestMaxMergeCount on CI after the LUCENE-5871 / apache#1284 port made Shutdown(false) forward-flush merges more aggressively, pushing more merges through DoMerge than maxMergeCount and tripping the Signal() overshoot. The new CountDownLatch wraps a ManualResetEventSlim + Interlocked int. CountDown is safe past zero (early return + idempotent Set). Count clamps at zero to match Java's getCount() semantics. Await(TimeSpan) returns a bool matching await(long, TimeUnit). Thread interrupt is NOT honored (consistent with the rest of Lucene.NET's threading; see the ReentrantLockTest precedent), and the two interrupt-dependent tests in CountDownLatchTest are [Ignore]'d with a LUCENENET note. CountDownLatchTest extends the existing JSR166TestCase. Anonymous Java inner classes are extracted as ThreadAnonymousClassFor<TestName> per the established naming convention. 7 of 9 upstream tests pass; 2 are skipped per the interrupt-semantics note above. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e#1284 Replaces System.Threading.CountdownEvent with the new Lucene.Net.Support.Threading.CountDownLatch across 22 test files where the upstream Java equivalent uses java.util.concurrent.CountDownLatch. Mechanical changes: - type rename: CountdownEvent -> CountDownLatch - .Signal() -> .CountDown() - .Wait() / .Wait(TimeSpan) -> .Await() / .Await(TimeSpan) - .CurrentCount -> .Count - .IsSet -> .Count == 0 - added using Lucene.Net.Support.Threading where needed Workarounds for CountdownEvent.Signal()'s "throws past zero" behavior that are now redundant (CountDown is idempotent at zero, matching Java): - TestConcurrentMergeScheduler.TestMaxMergeCount: removed the CurrentCount > 0 guard. This was the trigger for the CI failure on Windows .NET Framework and Ubuntu .NET 8 after the LUCENE-5871 port made Shutdown(false) flush more aggressively. - TestIndexWriterWithThreads: removed the same pre-existing guard at iwConstructed. - TestDocumentsWriterStallControl: removed if (!updateJoin.IsSet) guard around CountDown(); upstream calls countDown() unconditionally. - TestControlledRealTimeReopenThread: replaced two signal.Reset(Count - 1)/latch.Reset(Count - 1) hacks with plain CountDown(), matching upstream signal.countDown()/latch.countDown(). Other adjustments: - BaseDocValuesFormatTestCase: dropped `using` keyword on two latches. CountdownEvent was IDisposable; CountDownLatch isn't (Java's isn't either), so the latches now go out of scope without explicit cleanup. - TestControlledRealTimeReopenThread.LatchedIndexWriter: ctor changed from public to internal so CountDownLatch (internal) is reachable from a public class member. Files touched: 22 test files plus 2 in TestFramework. All ported sites now read the same as the upstream Java; no LUCENENET notes were added because the code is now equivalent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…che#1284 Align the latch's naming with .NET's CountdownEvent: rename the type CountDownLatch -> CountdownLatch, Await()/Await(TimeSpan) -> Wait()/Wait(TimeSpan), and CountDown() -> Signal(). Upstream-Java references in comments are left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Well, one of the runs prior to the most recent one had a random failure: https://github.com/apache/lucenenet/actions/runs/27988259483/job/82834745311 I guess we're not quite out of the woods yet. Converting to draft while I investigate. Logs: |
paulirwin
marked this pull request as draft
June 22, 2026 22:43
4 tasks
paulirwin
marked this pull request as ready for review
June 23, 2026 15:01
Contributor
Author
|
Ready for review again; flaky test was a known issue. Fixed in #1382 separately. |
NightOwl888
requested changes
Jun 28, 2026
NightOwl888
left a comment
NightOwl888
left a comment
Contributor
There was a problem hiding this comment.
Since we are touching every CountdownLatch line now, this is the best place to see all of the missing Dispose() calls to it and fix them. This is the sort of thing that could come back to bite us if we don't release the resources. Someday, it could produce strange results that are difficult to reproduce and debug that ultimately amount to resource starvation.
NightOwl888
requested changes
Jul 4, 2026
NightOwl888
left a comment
NightOwl888
left a comment
Contributor
There was a problem hiding this comment.
I queued up 30 runs and out of the first 15 there were several failures. After disregarding:
TestIndexWriterOnJRECrashTestFlushExceptionsTestOpenInputConcurrentFileExtension_Issue1090
TestNoWaitClose is now failing and I don't think it failed before.
First TestNoWaitClose Failure
- OS: Windows
- TFM: net472
- Platform: x64
System.IO.IOException : file "_av.fdx" was already written to
(Test: Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0x68968eeb694ef563:0x1bbc85657306f1c9")]
[assembly: NUnit.Framework.SetCulture("ff-Adlm-GN")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0x68968eeb694ef563:0x1bbc85657306f1c9" />
<Parameter name="tests:culture" value="ff-Adlm-GN" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0x68968eeb694ef563:0x1bbc85657306f1c9",
"culture": "ff-Adlm-GN"
}
}
Fixture Test Values
Random Seed: 0x68968eeb694ef563:0x1bbc85657306f1c9
Culture: ff-Adlm-GN
Time Zone: (UTC+03:00) Amman
Default Codec: Lucene46 (RandomCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=no): {termVector=DFR I(n)L3(800), id=IB LL-D1}
System Properties
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace
at Lucene.Net.Store.MockDirectoryWrapper.CreateOutput(String name, IOContext context) in /_/src/Lucene.Net.TestFramework/Store/MockDirectoryWrapper.cs:line 669
at Lucene.Net.Codecs.Compressing.CompressingStoredFieldsWriter..ctor(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, Int32 chunkSize) in /_/src/Lucene.Net/Codecs/Compressing/CompressingStoredFieldsWriter.cs:line 108
at Lucene.Net.Codecs.Compressing.CompressingStoredFieldsFormat.FieldsWriter(Directory directory, SegmentInfo si, IOContext context) in /_/src/Lucene.Net/Codecs/Compressing/CompressingStoredFieldsFormat.cs:line 114
at Lucene.Net.Index.StoredFieldsProcessor.InitFieldsWriter(IOContext context) in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 105
at Lucene.Net.Index.StoredFieldsProcessor.FinishDocument() in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 146
at Lucene.Net.Index.TwoStoredFieldsConsumers.FinishDocument() in /_/src/Lucene.Net/Index/TwoStoredFieldsConsumers.cs:line 77
at Lucene.Net.Index.DocFieldProcessor.FinishDocument() in /_/src/Lucene.Net/Index/DocFieldProcessor.cs:line 293
at Lucene.Net.Index.DocumentsWriterPerThread.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriterPerThread.cs:line 316
at Lucene.Net.Index.DocumentsWriter.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriter.cs:line 592
at Lucene.Net.Index.IndexWriter.UpdateDocument(Term term, IEnumerable`1 doc, Analyzer analyzer) in /_/src/Lucene.Net/Index/IndexWriter.cs:line 1829
[at Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose() in /_/src/Lucene.Net.Tests/Index/TestIndexWriterMerging.cs:line 408](https://dev.azure.com/shad0962/Experiments/_git/4882ad91-3a7d-453c-bd92-8068ed8f2711?path=%2F_%2Fsrc%2FLucene.Net.Tests%2FIndex%2FTestIndexWriterMerging.cs&version=GBissue%2F1284&_a=contents&line=408&lineEnd=409&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)Second TestNoWaitClose Failure
- OS: Windows
- TFM: net10.0
- Platform: x64
System.IO.IOException : file "_91.fdt" was already written to
(Test: Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0x92f2863df141329a:0x918f78dad328fbab")]
[assembly: NUnit.Framework.SetCulture("ff-Latn-CM")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0x92f2863df141329a:0x918f78dad328fbab" />
<Parameter name="tests:culture" value="ff-Latn-CM" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0x92f2863df141329a:0x918f78dad328fbab",
"culture": "ff-Latn-CM"
}
}
Fixture Test Values
Random Seed: 0x92f2863df141329a:0x918f78dad328fbab
Culture: ff-Latn-CM
Time Zone: (UTC+01:00) Belgrade, Bratislava, Budapest, Ljubljana, Prague
Default Codec: CheapBastard (CheapBastardCodec)
Default Similarity: DefaultSimilarity
System Properties
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace
at Lucene.Net.Store.MockDirectoryWrapper.CreateOutput(String name, IOContext context) in /_/src/Lucene.Net.TestFramework/Store/MockDirectoryWrapper.cs:line 669
at Lucene.Net.Index.StoredFieldsProcessor.FinishDocument() in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 145
at Lucene.Net.Index.DocFieldProcessor.FinishDocument() in /_/src/Lucene.Net/Index/DocFieldProcessor.cs:line 293
at Lucene.Net.Index.IndexWriter.UpdateDocument(Term term, IEnumerable`1 doc, Analyzer analyzer) in /_/src/Lucene.Net/Index/IndexWriter.cs:line 1829
at Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose() in /_/src/Lucene.Net.Tests/Index/TestIndexWriterMerging.cs:line 411
at System.RuntimeMethodHandle.InvokeMethod(ObjectHandleOnStack target, Void** arguments, ObjectHandleOnStack sig, BOOL isConstructor, ObjectHandleOnStack result)
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)Third TestNoWaitClose Failure
- OS: Windows
- TFM: net9.0
- Platform: x64
System.IO.IOException : file "_8x.fdx" was already written to
(Test: Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0x38e7aee2fdab6a45:0xe8c7bfb07d922c8a")]
[assembly: NUnit.Framework.SetCulture("ar-SO")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0x38e7aee2fdab6a45:0xe8c7bfb07d922c8a" />
<Parameter name="tests:culture" value="ar-SO" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0x38e7aee2fdab6a45:0xe8c7bfb07d922c8a",
"culture": "ar-SO"
}
}
Fixture Test Values
Random Seed: 0x38e7aee2fdab6a45:0xe8c7bfb07d922c8a
Culture: ar-SO
Time Zone: (UTC-08:00) Coordinated Universal Time-08
Default Codec: Lucene45 (Lucene45RWCodec)
Default Similarity: DefaultSimilarity
System Properties
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace
at Lucene.Net.Store.MockDirectoryWrapper.CreateOutput(String name, IOContext context) in /_/src/Lucene.Net.TestFramework/Store/MockDirectoryWrapper.cs:line 669
at Lucene.Net.Codecs.Compressing.CompressingStoredFieldsWriter..ctor(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, Int32 chunkSize) in /_/src/Lucene.Net/Codecs/Compressing/CompressingStoredFieldsWriter.cs:line 108
at Lucene.Net.Index.StoredFieldsProcessor.InitFieldsWriter(IOContext context) in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 100
at Lucene.Net.Index.StoredFieldsProcessor.FinishDocument() in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 145
at Lucene.Net.Index.DocFieldProcessor.FinishDocument() in /_/src/Lucene.Net/Index/DocFieldProcessor.cs:line 293
at Lucene.Net.Index.DocumentsWriterPerThread.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriterPerThread.cs:line 315
at Lucene.Net.Index.DocumentsWriter.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriter.cs:line 592
at Lucene.Net.Index.IndexWriter.UpdateDocument(Term term, IEnumerable`1 doc, Analyzer analyzer) in /_/src/Lucene.Net/Index/IndexWriter.cs:line 1829
at Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose() in /_/src/Lucene.Net.Tests/Index/TestIndexWriterMerging.cs:line 411
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)Fourth TestNoWaitClose Failure
- OS: Windows
- TFM: net8.0
- Platform: x64
System.IO.IOException : file "_bo.fdx" was already written to
(Test: Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0x52794ef19a953d78:0x9230e2aab3306407")]
[assembly: NUnit.Framework.SetCulture("mg")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0x52794ef19a953d78:0x9230e2aab3306407" />
<Parameter name="tests:culture" value="mg" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0x52794ef19a953d78:0x9230e2aab3306407",
"culture": "mg"
}
}
Fixture Test Values
Random Seed: 0x52794ef19a953d78:0x9230e2aab3306407
Culture: mg
Time Zone: (UTC-03:00) Saint Pierre and Miquelon
Default Codec: Lucene45 (Lucene45RWCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes): {termVector=DFR I(ne)L2, id=DFR I(F)BZ(0.3)}
System Properties
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace
at Lucene.Net.Store.MockDirectoryWrapper.CreateOutput(String name, IOContext context) in /_/src/Lucene.Net.TestFramework/Store/MockDirectoryWrapper.cs:line 669
at Lucene.Net.Codecs.Compressing.CompressingStoredFieldsWriter..ctor(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, Int32 chunkSize) in /_/src/Lucene.Net/Codecs/Compressing/CompressingStoredFieldsWriter.cs:line 108
at Lucene.Net.Index.StoredFieldsProcessor.InitFieldsWriter(IOContext context) in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 100
at Lucene.Net.Index.StoredFieldsProcessor.FinishDocument() in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 145
at Lucene.Net.Index.DocFieldProcessor.FinishDocument() in /_/src/Lucene.Net/Index/DocFieldProcessor.cs:line 293
at Lucene.Net.Index.DocumentsWriterPerThread.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriterPerThread.cs:line 315
at Lucene.Net.Index.DocumentsWriter.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriter.cs:line 592
at Lucene.Net.Index.IndexWriter.UpdateDocument(Term term, IEnumerable`1 doc, Analyzer analyzer) in /_/src/Lucene.Net/Index/IndexWriter.cs:line 1829
at Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose() in /_/src/Lucene.Net.Tests/Index/TestIndexWriterMerging.cs:line 411
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)Fifth TestNoWaitClose Failure
- OS: macOS
- TFM: net9.0
- Platform: x64
System.IO.IOException : file "_3w.fdt" was already written to
(Test: Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0xa14e591e25ee3dc1:0x8f7f834083c5ece0")]
[assembly: NUnit.Framework.SetCulture("syr-IQ")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0xa14e591e25ee3dc1:0x8f7f834083c5ece0" />
<Parameter name="tests:culture" value="syr-IQ" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0xa14e591e25ee3dc1:0x8f7f834083c5ece0",
"culture": "syr-IQ"
}
}
Fixture Test Values
Random Seed: 0xa14e591e25ee3dc1:0x8f7f834083c5ece0
Culture: syr-IQ
Time Zone: (UTC+07:00) Krasnoyarsk Standard Time (Novokuznetsk)
Default Codec: CheapBastard (CheapBastardCodec)
Default Similarity: DefaultSimilarity
System Properties
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace
[at Lucene.Net.Store.MockDirectoryWrapper.CreateOutput(String name, IOContext context) in /_/src/Lucene.Net.TestFramework/Store/MockDirectoryWrapper.cs:line 669](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net.TestFramework%2FStore%2FMockDirectoryWrapper.cs&version=GBissue%2F1284&_a=contents&line=669&lineEnd=670&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Codecs.Lucene40.Lucene40StoredFieldsWriter..ctor(Directory directory, String segment, IOContext context) in /_/src/Lucene.Net/Codecs/Lucene40/Lucene40StoredFieldsWriter.cs:line 102](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FCodecs%2FLucene40%2FLucene40StoredFieldsWriter.cs&version=GBissue%2F1284&_a=contents&line=102&lineEnd=103&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.StoredFieldsProcessor.InitFieldsWriter(IOContext context) in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 100](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FStoredFieldsProcessor.cs&version=GBissue%2F1284&_a=contents&line=100&lineEnd=101&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.StoredFieldsProcessor.FinishDocument() in /_/src/Lucene.Net/Index/StoredFieldsProcessor.cs:line 145](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FStoredFieldsProcessor.cs&version=GBissue%2F1284&_a=contents&line=145&lineEnd=146&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.DocFieldProcessor.FinishDocument() in /_/src/Lucene.Net/Index/DocFieldProcessor.cs:line 293](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FDocFieldProcessor.cs&version=GBissue%2F1284&_a=contents&line=293&lineEnd=294&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.DocumentsWriterPerThread.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriterPerThread.cs:line 315](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FDocumentsWriterPerThread.cs&version=GBissue%2F1284&_a=contents&line=315&lineEnd=316&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.DocumentsWriter.UpdateDocument(IEnumerable`1 doc, Analyzer analyzer, Term delTerm) in /_/src/Lucene.Net/Index/DocumentsWriter.cs:line 592](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FDocumentsWriter.cs&version=GBissue%2F1284&_a=contents&line=592&lineEnd=593&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.IndexWriter.UpdateDocument(Term term, IEnumerable`1 doc, Analyzer analyzer) in /_/src/Lucene.Net/Index/IndexWriter.cs:line 1829](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net%2FIndex%2FIndexWriter.cs&version=GBissue%2F1284&_a=contents&line=1829&lineEnd=1830&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Lucene.Net.Index.TestIndexWriterMerging.TestNoWaitClose() in /_/src/Lucene.Net.Tests/Index/TestIndexWriterMerging.cs:line 411](https://dev.azure.com/lucene-net-temp4/Main/_git/fdfefc9a-2f87-4b6c-b69a-6efa0c048afe?path=%2F_%2Fsrc%2FLucene.Net.Tests%2FIndex%2FTestIndexWriterMerging.cs&version=GBissue%2F1284&_a=contents&line=411&lineEnd=412&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)I compared TestNoWaitClose() line by line against Lucene 4.10.0 and the test is identical.
Contributor
Author
|
Comments resolved. I've kicked off some agents attempting to reproduce the test failures ( Update: can reproduce with |
Contributor
Author
|
It turns out this TestNoWaitClose failure is an upstream bug! It is reproducible in Java Lucene 4.8.1 and 4.10.4. Not only that, but it is a latent, existing bug in our code too. This PR just widens the window in which it can occur because of the Shutdown change. It was fixed in Lucene 7.7.0 via LUCENE-8639: "Newly created threadstates while flushing / refreshing can cause duplicated sequence IDs on IndexWriter." Java repro (generated with Claude Code, Opus 4.8): import java.util.ArrayList;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
import org.apache.lucene.index.LogMergePolicy;
import org.apache.lucene.store.AlreadyClosedException;
import org.apache.lucene.store.MockDirectoryWrapper;
import org.apache.lucene.util.LuceneTestCase;
import com.carrotsearch.randomizedtesting.annotations.Repeat;
/**
* Aggressive port of the C# LuceneNet-specific repro: 6 concurrent adder threads,
* maxBufferedDocs=2, mergeFactor stress, 500 close(false)/reopen cycles in one method.
* If upstream 4.10.4 regresses the segment-name counter the same way the .NET port does,
* this will fail with MockDirectoryWrapper "already written to" or "still open locks".
*/
public class TestNoWaitCloseStress4104 extends LuceneTestCase {
static final int CYCLES = 500;
static final int ADDERS = 6;
@Repeat(iterations = 20)
public void testNoWaitCloseStress() throws Throwable {
MockDirectoryWrapper directory = newMockDirectory();
final Document doc = new Document();
FieldType customType = new FieldType(TextField.TYPE_STORED);
customType.setTokenized(false);
final Field idField = newField("id", "", customType);
doc.add(idField);
IndexWriter writer = new IndexWriter(directory,
newIndexWriterConfig(new MockAnalyzer(random()))
.setOpenMode(OpenMode.CREATE)
.setMaxBufferedDocs(2)
.setMergePolicy(newLogMergePolicy()));
((LogMergePolicy) writer.getConfig().getMergePolicy()).setMergeFactor(100);
for (int cycle = 0; cycle < CYCLES; cycle++) {
for (int j = 0; j < 40; j++) {
idField.setStringValue(Integer.toString(cycle * 201 + j));
writer.addDocument(doc);
}
((LogMergePolicy) writer.getConfig().getMergePolicy()).setMergeFactor(2);
final IndexWriter finalWriter = writer;
final ArrayList<Throwable> failure = new ArrayList<>();
Thread[] adders = new Thread[ADDERS];
for (int t = 0; t < ADDERS; t++) {
adders[t] = new Thread() {
@Override
public void run() {
boolean done = false;
while (!done) {
for (int i = 0; i < 100; i++) {
try {
finalWriter.addDocument(doc);
} catch (AlreadyClosedException e) {
done = true;
break;
} catch (NullPointerException e) {
done = true;
break;
} catch (Throwable e) {
e.printStackTrace(System.out);
failure.add(e);
done = true;
break;
}
}
Thread.yield();
}
}
};
}
if (failure.size() > 0) {
throw failure.get(0);
}
for (Thread a : adders) {
a.start();
}
writer.close(false);
for (Thread a : adders) {
a.join();
}
IndexReader reader = DirectoryReader.open(directory);
reader.close();
writer = new IndexWriter(directory,
newIndexWriterConfig(new MockAnalyzer(random()))
.setOpenMode(OpenMode.APPEND)
.setMaxBufferedDocs(2)
.setMergePolicy(newLogMergePolicy()));
((LogMergePolicy) writer.getConfig().getMergePolicy()).setMergeFactor(100);
}
writer.close();
directory.close();
}
}I'm going to work on backporting LUCENE-8639 to fix this. |
Contributor
Author
|
LUCENE-8639 was the wrong direction for the fix. There are actually two related things here that compounded into this surfacing. After attempting a backport and determining that it wasn't fixed sufficiently, I bisected the releases between 7.7.0 (when it for sure was fixed) and 4.10.4 (when it for sure was not fixed), and found that this particular test failure was fixed in Lucene 6.0.0, as a part of a much larger item LUCENE-5438 ("add near-real-time replication"), commit 6369012d. This change included a one line fix to MockDirectoryWrapper to add But the segment-name reuse itself was never "fixed" in Lucene production code, even in 10.x. The reopened writer can still reissue a segment name that a previous cycle had used. Fortunately, this is benign, hence the effective suppression in the test code. So I don't think it's worth filing a bug upstream, as it seems to be an acceptable state. The orphaned files are always gone before reuse, and a live/committed segment is never overwritten. So the test-side change is safe to suppress this. |
4 tasks
NightOwl888
requested changes
Jul 9, 2026
Contributor
There was a problem hiding this comment.
Alright, with the latest changes in #1373 and my test against master-1013-1284 (combined), I discovered that there is only one failing test remaining. Furthermore, I have identified this branch (rebased against master) as the one that is causing the failure to reappear. I also confirmed that it has already been fixed on the master branch, so this is a regression.
TestIndexWriterOnJRECrash.TestNRTThreads_Mem
TestNRTThreads_Mem
Source: TestIndexWriterOnJRECrash.cs line 54
Duration: 3.1 min
Message:
Repeat failed on iteration '14'.
Lucene.Net.Util.LuceneSystemException : CheckIndex failed
Data:
Lucene_SuppressedExceptions: [Lucene.Net.Util.LuceneSystemException: MockDirectoryWrapper: cannot close: there are still open files: {_1_MockRandom_0.doc=1}
---> Lucene.Net.Util.LuceneSystemException: unclosed IndexInput: _1_MockRandom_0.doc
--- End of inner exception stack trace ---
at Lucene.Net.Store.MockDirectoryWrapper.Dispose(Boolean disposing) in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Store\MockDirectoryWrapper.cs:line 956
at Lucene.Net.Store.Directory.Dispose() in F:\Projects\lucenenet\src\Lucene.Net\Store\Directory.cs:line 134
at Lucene.Net.Util.IOUtils.DisposeWhileHandlingException(Exception priorException, IDisposable[] objects) in F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 292]
(Test: Lucene.Net.Index.TestIndexWriterOnJRECrash.TestNRTThreads_Mem)
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
[assembly: Lucene.Net.Util.RandomSeed("0x03c0703dd1d74190:0xfc8ded704ed908f9")]
[assembly: NUnit.Framework.SetCulture("arn")]
Option 2:
Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0x03c0703dd1d74190:0xfc8ded704ed908f9" />
<Parameter name="tests:culture" value="arn" />
</TestRunParameters>
</RunSettings>
Option 3:
Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:
{
"tests": {
"seed": "0x03c0703dd1d74190:0xfc8ded704ed908f9",
"culture": "arn"
}
}
Fixture Test Values
=================
Random Seed: 0x03c0703dd1d74190:0xfc8ded704ed908f9
Culture: arn
Time Zone: (UTC+02:00) Windhoek
Default Codec: CheapBastard (CheapBastardCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=True,coord=yes): {}
System Properties
=================
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: False
Random Multiplier: 1
Stack Trace:
TestUtil.CheckIndex(Directory dir, Boolean crossCheckTermVectors) line 176
TestUtil.CheckIndex(Directory dir) line 161
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 416
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 433
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 433
TestIndexWriterOnJRECrash.TestNRTThreads_Mem() line 105
InvokeStub_TestIndexWriterOnJRECrash.TestNRTThreads_Mem(Object, Object, IntPtr*)
RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
--- End of stack trace from previous location ---
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 428
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 433
TestIndexWriterOnJRECrash.CheckIndexes(FileSystemInfo file) line 433
TestIndexWriterOnJRECrash.TestNRTThreads_Mem() line 105
InvokeStub_TestIndexWriterOnJRECrash.TestNRTThreads_Mem(Object, Object, IntPtr*)
RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Standard Output:
CheckIndex failed
Segments file=segments_2 numSegments=4 version=4.8 format=
1 of 4: name=_0 docCount=1
codec=Lucene46
compound=False
numFiles=29
size (MB)=0.00376129150390625
diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=10.0.8, java.vendor=Microsoft, timestamp=2026-07-09 14:13:31}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [7 fields]
test: field norms.........OK [3 fields]
test: terms, freq, prox...ERROR: Lucene.Net.Diagnostics.AssertionException: Exception of type 'Lucene.Net.Diagnostics.AssertionException' was thrown.
at Lucene.Net.Codecs.IntBlock.FixedInt32BlockIndexInput.Index.Read(DataInput indexIn, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\FixedIntBlockIndexInput.cs:line 175
at Lucene.Net.Codecs.Sep.SepPostingsReader.DecodeTerm(Int64[] empty, DataInput input, FieldInfo fieldInfo, BlockTermState termState, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 197
at Lucene.Net.Codecs.Memory.FSTTermsReader.TermsReader.SegmentTermsEnum.DecodeMetaData() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Memory\FSTTermsReader.cs:line 343
at Lucene.Net.Codecs.Memory.FSTTermsReader.TermsReader.BaseTermsEnum.Docs(IBits liveDocs, DocsEnum reuse, DocsFlags flags) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Memory\FSTTermsReader.cs:line 286
at Lucene.Net.Index.CheckIndex.CheckFields(Fields fields, IBits liveDocs, Int32 maxDoc, FieldInfos fieldInfos, Boolean doPrint, Boolean isVectors, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1108
at Lucene.Net.Index.CheckIndex.TestPostings(AtomicReader reader, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1651
Lucene.Net.Diagnostics.AssertionException: Exception of type 'Lucene.Net.Diagnostics.AssertionException' was thrown.
at Lucene.Net.Codecs.IntBlock.FixedInt32BlockIndexInput.Index.Read(DataInput indexIn, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\IntBlock\FixedIntBlockIndexInput.cs:line 175
at Lucene.Net.Codecs.Sep.SepPostingsReader.DecodeTerm(Int64[] empty, DataInput input, FieldInfo fieldInfo, BlockTermState termState, Boolean absolute) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 197
at Lucene.Net.Codecs.Memory.FSTTermsReader.TermsReader.SegmentTermsEnum.DecodeMetaData() in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Memory\FSTTermsReader.cs:line 343
at Lucene.Net.Codecs.Memory.FSTTermsReader.TermsReader.BaseTermsEnum.Docs(IBits liveDocs, DocsEnum reuse, DocsFlags flags) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Memory\FSTTermsReader.cs:line 286
at Lucene.Net.Index.CheckIndex.CheckFields(Fields fields, IBits liveDocs, Int32 maxDoc, FieldInfos fieldInfos, Boolean doPrint, Boolean isVectors, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1108
at Lucene.Net.Index.CheckIndex.TestPostings(AtomicReader reader, TextWriter infoStream, Boolean verbose) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 1651
test: stored fields.......OK [5 total field count; avg 5 fields per doc]
test: term vectors........OK [2 total vector count; avg 2.0 term/freq vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
Lucene.Net.Util.LuceneSystemException: Term Index test failed
at Lucene.Net.Index.CheckIndex.DoCheckIndex(IList`1 onlySegments) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 849
2 of 4: name=_1 docCount=1
codec=Lucene46
compound=False
numFiles=30
size (MB)=0.006054878234863281
diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=10.0.8, java.vendor=Microsoft, timestamp=2026-07-09 14:13:31}
no deletions
test: open reader.........FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
System.IO.IOException: codec header mismatch: actual header=-989462528 vs expected header=1071082519 (resource: MockIndexInputWrapper(NIOFSIndexInput(path="C:\Users\shad\AppData\Local\Temp\LuceneTemp\netcrash-bgoylkrp\LuceneTemp\TestNRTThreads-vcpk0vdh\_1_MockRandom_0.doc")))
at Lucene.Net.Codecs.MockSep.MockSingleInt32IndexInput..ctor(Directory dir, String fileName, IOContext context) in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockSep\MockSingleIntIndexInput.cs:line 38
at Lucene.Net.Codecs.MockSep.MockSingleInt32Factory.OpenInput(Directory dir, String fileName, IOContext context) in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockSep\MockSingleIntFactory.cs:line 32
at Lucene.Net.Codecs.MockRandom.MockRandomPostingsFormat.MockInt32StreamFactory.OpenInput(Directory dir, String fileName, IOContext context) in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockRandom\MockRandomPostingsFormat.cs:line 105
at Lucene.Net.Codecs.Sep.SepPostingsReader..ctor(Directory dir, FieldInfos fieldInfos, SegmentInfo segmentInfo, IOContext context, Int32StreamFactory intFactory, String segmentSuffix) in F:\Projects\lucenenet\src\Lucene.Net.Codecs\Sep\SepPostingsReader.cs:line 56
at Lucene.Net.Codecs.MockRandom.MockRandomPostingsFormat.FieldsProducer(SegmentReadState state) in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Codecs\MockRandom\MockRandomPostingsFormat.cs:line 386
at Lucene.Net.Codecs.PerField.PerFieldPostingsFormat.FieldsReader..ctor(SegmentReadState readState) in F:\Projects\lucenenet\src\Lucene.Net\Codecs\PerField\PerFieldPostingsFormat.cs:line 237
at Lucene.Net.Index.SegmentCoreReaders..ctor(SegmentReader owner, Directory dir, SegmentCommitInfo si, IOContext context, Int32 termsIndexDivisor) in F:\Projects\lucenenet\src\Lucene.Net\Index\SegmentCoreReaders.cs:line 111
at Lucene.Net.Index.SegmentReader..ctor(SegmentCommitInfo si, Int32 termInfosIndexDivisor, IOContext context) in F:\Projects\lucenenet\src\Lucene.Net\Index\SegmentReader.cs:line 90
at Lucene.Net.Index.CheckIndex.DoCheckIndex(IList`1 onlySegments) in F:\Projects\lucenenet\src\Lucene.Net\Index\CheckIndex.cs:line 743
3 of 4: name=_3 docCount=19
codec=Lucene46
compound=False
numFiles=31
size (MB)=0.04930305480957031
diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=10.0.8, java.vendor=Microsoft, timestamp=2026-07-09 14:13:32}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [9 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [1204 terms; 1389 terms/docs pairs; 1636 tokens]
test: stored fields.......OK [102 total field count; avg 5.368421 fields per doc]
test: term vectors........OK [38 total vector count; avg 2.0 term/freq vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
4 of 4: name=_2 docCount=23
codec=Lucene46
compound=False
numFiles=34
size (MB)=0.18881893157958984
diagnostics = {source=flush, lucene.version=4.8.0, os=Microsoft Windows, os.arch=X64, os.version=10.0.19045, java.version=10.0.8, java.vendor=Microsoft, timestamp=2026-07-09 14:13:32}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [11 fields]
test: field norms.........OK [6 fields]
test: terms, freq, prox...OK [3905 terms; 4184 terms/docs pairs; 10885 tokens]
test: stored fields.......OK [121 total field count; avg 5.2608695 fields per doc]
test: term vectors........OK [47 total vector count; avg 2.0434783 term/freq vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 0 NUMERIC; 1 SORTED; 0 SORTED_SET]
WARNING: 2 broken segments (containing 2 documents) detectedNote that this failure happened locally when testing in Release mode. I used the following settings:
{
"assert": "true",
"tests": {
"awaitsfix": "false",
"verbose": "false"
}
}I ran the test with [Repeat(50)] which failed only here (rebased against master), but did not fail on master or #1373 rebased against master.
…ilarity selection MockRandomPostingsFormat, RandomCodec, and RandomSimilarityProvider chose a per-file/per-field int-stream factory, postings/docvalues format, and similarity by indexing into a collection with `string.GetHashCode()`. On .NET, string hash codes are randomized per process, so a choice made when writing an index (e.g. in the forked child process of TestIndexWriterOnJRECrash) differed from the choice made when reading it back in another process, producing spurious "codec header mismatch" / CheckIndex failures. Java is unaffected because String.hashCode() is deterministic across JVMs. Use J2N's CharSequenceComparer.Ordinal.GetHashCode(), which is deterministic and produces the same value as Java's String.hashCode(), keeping writer and reader in sync. This was a pre-existing test-framework bug (reproduces on master). Verified: TestIndexWriterOnJRECrash.TestNRTThreads_Mem (previously a deterministic failure under seed 0x03c0703dd1d74190:0xfc8ded704ed908f9) now passes on fixed and random seeds; RandomCodec- and similarity-heavy suites remain green. Confirmed against a native build of upstream Lucene 4.8.1, which passes this test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
The root cause of this failure was unrelated to this PR; with that seed it was reliably reproducible on master. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix IndexWriter.Dispose leaking file handles by backporting fix from Lucene 4.10.0
Fixes #1284
Description
See #1284 and #1284 (comment) for more information.
This PR backports apache/lucene@2cfcdcc from Lucene 4.10.0 to fix this issue.
Additionally, CI surfaced increased random failures of
TestMaxMergeCount, caused by a pre-existing latent porting issue rather than something new in this PR. The fix in this PR is upstream-faithful:Shutdown(false)(replacing the oldCloseInternal(false, true)path) now triggers a fresh round of merges during shutdown that the old code skipped, which surfaces a long-dormant problem in how the test was translated.CountdownEventis not a drop-in replacement for Java'sCountDownLatch:CountdownEvent.Signal()throws if the count is already zero, whereas Java'scountDown()silently does nothing past zero, and Lucene's test code relies on the Java semantics.This PR adds a port of
CountDownLatchfrom Apache Harmony, adapted to useInterlockedandManualResetEventSlimlike .NET'sCountdownEventdoes (which is simpler than the HarmonySync-based implementation). All Harmony tests were ported; two are[Ignore]'d because we don't need thread interrupt support.