Search crash

[Kiliggggg]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

It's sporadic. We haven't found a pattern yet.
Error:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at System.IO.UnmanagedMemoryAccessor.ReadByte(Int64)
at Lucene.Net.Store.DataInput.ReadVInt32()
at Lucene.Net.Codecs.Lucene41.Lucene41PostingsReader.ReadVInt32Block(Lucene.Net.Store.IndexInput, Int32[], Int32[], Int32, Boolean)
at Lucene.Net.Codecs.Lucene41.Lucene41PostingsReader+BlockDocsEnum.RefillDocs()
at Lucene.Net.Codecs.Lucene41.Lucene41PostingsReader+BlockDocsEnum.NextDoc()
at Lucene.Net.Search.Weight+DefaultBulkScorer.ScoreAll(Lucene.Net.Search.ICollector, Lucene.Net.Search.Scorer)
at Lucene.Net.Search.IndexSearcher.Search(System.Collections.Generic.IList`1<Lucene.Net.Index.AtomicReaderContext>, Lucene.Net.Search.Weight, Lucene.Net.Search.ICollector)
at Lucene.Net.Search.IndexSearcher.Search(Lucene.Net.Search.Weight, Lucene.Net.Search.ScoreDoc, Int32)
at EDiscovery.Core.Search.Searcher.Search(EDiscovery.Core.Query.Query, EDiscovery.Core.Query.Sort, EDiscovery.Core.Query.ScoreDoc, Int32)

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

Lucene.NET Version

4.8.0-beta00016

.NET Version

.NET 8

Operating System

Linux

Anything else?

No response

[NightOwl888]

Could you try upgrading to 4.8.0-beta00017 to see if it still happens?

[Kiliggggg]

Could you try upgrading to 4.8.0-beta00017 to see if it still happens?

Got it, I will try it then update the result, thank you.

[Kiliggggg]

We are using the 4.8.0-beta00017 but the error still exists.

[florisrobbemont]

I'm also getting this error with 4.8.0-beta00017. Even when I clone the latest master it keeps crashing.

My setup is not that complex:

• Multiple threads using a single SearchManager
• Simple FacetsCollector + FacetCounter (FastTaxonomyFacetCounts)

public static Dictionary<string, FacetResult> GetFacetResults(this BooleanQuery facetQuery, ILuceneIndex index, FacetsConfig facetsConfig)
    {
        var results = new Dictionary<string, FacetResult>();

        var searcher = index.SearcherManager.Acquire();
        try
        {
            // This collector accumulates all hits for faceting
            var fc = new FacetsCollector();
            FacetsCollector.Search(searcher, facetQuery, int.MaxValue, fc);
                        
            // Get the facet counts
            var facetCounts = new FastTaxonomyFacetCounts(index.FacetReader, facetsConfig, fc);

            // For each facetable field, gather its facet
            foreach (var facetField in facetsConfig.DimConfigs.Keys)
            {
                var fr = facetCounts.GetTopChildren(int.MaxValue, facetField);

                if (fr != null)
                {
                    // store it using the same "key" in results:
                    results[facetField] = fr;
                }
            }
        }
        finally
        {
            index.SearcherManager.Release(searcher);
            searcher = null;
        }

        return results;
    }

At first the exception was random, but when I run this method 10 times simultaneously the error starts popping up more often.

When I attach a debugger it always seems to crash on something in PriorityQueue. But I can't figure out if this is a multi-threading issue (PriorityQueue doesn't seem to be shared, because even when I synclock all the different methods the crash still happen).

So it doesn't feel like a concurrency bug, although it gets triggered more when doing concurrent searches.

Let me know if you need anything from me. We're about 3-4 weeks from going into production with this, so we're highly motivated to make this work ;)

Stack trace (will sometimes differ):

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Lucene.Net.Facet.FacetsCollector.DoSearch(Lucene.Net.Search.IndexSearcher, Lucene.Net.Search.ScoreDoc, Lucene.Net.Search.Query, Lucene.Net.Search.Filter, Int32, Lucene.Net.Search.Sort, Boolean, Boolean, Lucene.Net.Search.ICollector)
   at Lucene.Net.Facet.FacetsCollector.Search(Lucene.Net.Search.IndexSearcher, Lucene.Net.Search.Query, Int32, Lucene.Net.Search.ICollector)
   at Application.Data.ProductSearch.Index.QueryExtensions.GetFacetResults(Lucene.Net.Search.BooleanQuery, Application.Data.ProductSearch.Index.ILuceneIndex, Lucene.Net.Facet.FacetsConfig)
   at Application.Services.ProductService.Services.Searching.AbstractQueryService`3+<Search>d__15[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
   at Application.Data.AbstractDataService+<ExecuteListAsync>d__18`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)
   at Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Dapper, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishSlow(Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.ContinueWithTaskContinuation.Run(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.UnwrapPromise`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetFromTask(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.UnwrapPromise`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Invoke(System.Threading.Tasks.Task)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishSlow(Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

[Kiliggggg]

Do we have any update

[paulirwin]

I don't have an update at this time, but I've added it to the beta 18 milestone to track the progress. This would be a great issue to get help from the community to look into the possible cause.

[paulirwin]

@NightOwl888 I wonder if this is related to #1090. UnmanagedMemoryAccessor (in the original issue stack trace) is returned by MemoryMappedFile.CreateViewAccessor, so that smells like it's related.

@Kiliggggg @florisrobbemont Can either of you confirm if you're using MMapDirectory? If so, if you change it to another directory implementation (SimpleFSDirectory or NIOFSDirectory, or even RAMDirectory if the index can fit in memory), does it still happen?

[NightOwl888]

Given that the crash happens in PriorityQueue (could it be ValuePriorityQueue?), it seems possible that this is due to running on a machine with very little stack space, although I would expect there to be a StackOverflowException not an AccessViolationException. The default value that we provided in beta 17 is pretty high, 2048 bytes (or 1024 characters). This may not be optimal for all environments.

This is just the maximum value that is allowed on the stack before falling back to using array pool on the heap, so it can be lowered to a more acceptable value if your system is running into problems without any ill effects (I wouldn't recommend going below 32, though).

The value can be changed, since it is a system property named maxStackByteLimit.

Option 1 - Environment Variables

Configure the maxStackByteLimit variable, which must be prefixed with lucene:.

Name: lucene:maxStackByteLimit
Value: 256

Option 2 - Self-Contained Custom Configuration Provider

Here is a minimal self-contained implementation, that requires references on:

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.3" />
    <PackageReference Include="Microsoft.Extensions.FileProviders.Abstractions" Version="9.0.3" />
  </ItemGroup>

    public sealed class SystemConfigurationFactory : IConfigurationFactory
    {
        private static readonly IConfiguration config = new SystemConfiguration();

        public IConfiguration GetConfiguration() => config;
    }

    public sealed class SystemConfiguration : IConfiguration
    {
        private readonly ConcurrentDictionary<string, string?> systemProperties = new ConcurrentDictionary<string, string?>
        {
            ["maxStackByteLimit"] = "256",
        };

        public string? this[string key]
        {
            get => systemProperties.TryGetValue(key, out var value) ? value : null;
            set => systemProperties[key] = value;
        }

        public IEnumerable<IConfigurationSection> GetChildren()
        {
            // Return only the first-level keys as sections
            var sectionKeys = systemProperties.Keys
                .Select(k => k.Split([':'], 2)[0])
                .Distinct();

            return sectionKeys.Select(k => new ConfigurationSection(this, k));
        }

        public IChangeToken GetReloadToken() => NullChangeToken.Singleton;

        public IConfigurationSection GetSection(string key)
        {
            return new ConfigurationSection(this, key);
        }

        private class ConfigurationSection : IConfigurationSection
        {
            private readonly SystemConfiguration parent;
            public string Key { get; }
            public string Path => Key;
            public string? Value
            {
                get => parent[Key];
                set => parent[Key] = value;
            }

            public ConfigurationSection(SystemConfiguration parent, string key)
            {
                this.parent = parent;
                Key = key;
            }

            public IEnumerable<IConfigurationSection> GetChildren()
            {
                var sectionPrefix = Key + ":";
                var childKeys = parent.systemProperties.Keys
                    .Where(k => k.StartsWith(sectionPrefix))
                    .Select(k => k.Substring(sectionPrefix.Length).Split(':')[0])
                    .Distinct();

                return childKeys.Select(k => new ConfigurationSection(parent, sectionPrefix + k));
            }

            public IChangeToken GetReloadToken() => NullChangeToken.Singleton;

            public IConfigurationSection GetSection(string key)
            {
                return new ConfigurationSection(parent, $"{Key}:{key}");
            }

            // Explicit implementation for IConfiguration members
            string? IConfiguration.this[string key]
            {
                get => parent[$"{Key}:{key}"];
                set => parent[$"{Key}:{key}"] = value;
            }

            IEnumerable<IConfigurationSection> IConfiguration.GetChildren() => GetChildren();

            IChangeToken IConfiguration.GetReloadToken() => GetReloadToken();

            IConfigurationSection IConfiguration.GetSection(string key) => GetSection(key);
        }
    }

And your application startup would look like:

ConfigurationSettings.SetConfigurationFactory(new SystemConfigurationFactory());

Note that I haven't tested this, so there may be some kinks to work out. Also, be aware that this will disable the built-in environment variable provider so none of the lucene: prefixed environment variables will work.

Option 3 - Standard (or Custom) Configuration Providers

Alternatively, you could use existing Microsoft.Extension.Configuration providers by providing a pass-through implementation of IConfiguarationFactory, as follows.

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.3" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="9.0.3" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="9.0.3" />
  </ItemGroup>

    public sealed class ConfigurationFactory : IConfigurationFactory
    {
        private readonly IConfiguration configuration;

        public ConfigurationFactory(IConfiguration configuration)
        {
            this.configuration = configuration ?? throw new ArgumentNullException(nameof(configuration));
        }

        public IConfiguration GetConfiguration()
        {
            return configuration;
        }
    }

This can be used like so:

var configuration = new ConfigurationBuilder() // Note there should only be 1 instance of ConfigurationBuilder for your entire app
    .AddEnvironmentVariables(prefix: "lucene:") // Use a custom prefix to only load Lucene.NET settings
    .AddJsonFile("appsettings.json", optional: true)
    .Build();
ConfigurationSettings.SetConfigurationFactory(new ConfigurationFactory(configuration));

Then you could put your application settings in the appsettings.json file, like so:

{
    "maxStackByteLimit": "256"
}

Do note that the above implementation won't detect any changes to the file at runtime, though.

You could alternatively use one or more of the other providers, for example the Microsoft.Extensions.Configuration contains an in-memory provider.

---

All of that said, changing the directory implementation could be a way to quickly rule out MMapDirectory as the cause, although the other implementations are not designed to scale to as many reads as MMapDirectory.