Update docs for v4.0.0 and fix the DocFX publishing pipeline (#588)

The docs site has not deployed since 2023-03-08 and the content on `dev`
had drifted from the v4 API.

Publishing pipeline:

- Trigger `docfx-gh-pages.yml` on push/PR to `dev` (scoped to `docs/**`,
  `src/**` and the workflow file) instead of `feature/docfx`, which never
  fired. Keep `workflow_dispatch`.
- Install the .NET 8/9/10 SDKs. The DocFX `metadata` step MSBuild-loads
  `src/**.csproj`, which now multi-targets net8.0;net9.0;net10.0, so the
  pinned .NET 6 SDK could not restore or build it.
- Replace `nunit/docfx-action@v2.10.0` with `dotnet tool install -g docfx`
  so the `modern` template builds against a current DocFX.
- Bump `actions/checkout` v3 -> v7, `actions/setup-dotnet` v3 -> v6,
  `actions/configure-pages` v3 -> v6, `actions/upload-pages-artifact`
  v1 -> v5 and `actions/deploy-pages` v1 -> v5. The v1 Pages actions are
  retired.
- Guard the `deploy` job so pull requests build but do not publish.
- Point `_gitContribute.branch` at `dev` so "Edit this page" links resolve,
  and exclude `Examine.Benchmarks` from API reference generation.

Content:

- index.md: correct the Minimum Requirements table (V4 is .NET 8.0/9.0/10.0,
  V3 is .NET 6.0/8.0 - it was claiming .NET Standard 2.0), correct the
  install version to 4.0.0, and add a "What's new in V4" section linking to
  the v3-to-v4 API change report and the release notes.
- configuration.md: rewrite the facet and taxonomy registration samples.
  They used `facetsConfig:` and `useTaxonomyIndex:` named parameters on
  `AddExamineLuceneIndex` that do not exist; these are properties on
  `LuceneIndexOptions`. Replace the v3-era "After construction" section,
  which used a removed `LuceneIndex` constructor and a
  `FieldDefinitionCollection` that is no longer publicly mutable. Note that
  `UseTaxonomyIndex` defaults to true. Document the NRT options,
  `IndexDeletionPolicy`, `UnlockIndex` and `DirectoryFactory`.
- searching.md: rewrite the Faceting section - every sample called
  `facets.Facet(...)`, which does not exist; the API is `FacetString`,
  `FacetLongRange`, `FacetDoubleRange` and `FacetFloatRange`. Cover
  `GetFacets()`, `TryGetFacet` and nullable results, and taxonomy-side
  faceting. Fill in the "Booleans, Groups & Sub Groups" section, which was
  a TODO. Document `SelectField`/`SelectFields`/`SelectAllFields`,
  `WithBoost` and `Phrase()`.
- paging.md: document deep paging with `SearchAfterOptions` and
  `ExecuteWithLucene`, plus skip/take limits, scoring options and facet
  sampling. Correct `DefaultMaxResults` from 500 to 100.
- Replace `indexer.GetSearcher()` with the `Searcher` property throughout.
- Fix several samples that assigned `CreateQuery()` to a variable and then
  called `Or()`, `And()` or `Execute()` on it. `IQuery` exposes none of
  those - they are on `IBooleanOperation`, so the samples did not compile.
- Add replication.md covering `ExamineReplicator`, taxonomy replication,
  scheduled replication and `IsReplicationHealthy`. Replication had no
  documentation.
- developerguides: correct the project paths in codestructure.md, add
  `Examine.Benchmarks`, note the .NET 10 SDK requirement in buildtest.md,
  clean up the DocFX build instructions in docsite.md and add the 4.0.0 GA
  entry to roadmap.md.
- Sweep stale links: Lucene.NET `4.8.0-beta00016` -> `4.8.0-beta00018`
  (the dependency did not change in v4, the docs were just wrong), and
  `blob/master` / `blob/release/3.0` source links -> `blob/dev`.

Every code sample was compiled against the v4 assemblies to verify it uses
real APIs. `docs/docs-v1-v2/` is intentionally left untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Author: Shazwazza

.github/workflows/docfx-gh-pages.yml

@@ -3,10 +3,18 @@ name: Deploy DocFX with GitHub Pages dependencies preinstalled
 on:
   # Runs on pushes targeting the default branch
   push:
-    branches: ["feature/docfx"]
+    branches: ["dev"]
+    paths:
+      - 'docs/**'
+      - 'src/**'
+      - '.github/workflows/docfx-gh-pages.yml'
   pull_request:
     branches:
-      - 'feature/docfx'
+      - 'dev'
+    paths:
+      - 'docs/**'
+      - 'src/**'
+      - '.github/workflows/docfx-gh-pages.yml'
 
   # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
@@ -28,24 +36,33 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
-      - name: Instal DotNet
-        uses: actions/setup-dotnet@v3
+        uses: actions/checkout@v7
+      - name: Install DotNet
+        uses: actions/setup-dotnet@v6
         with:
-          dotnet-version: '6.0.x'
+          # DocFX builds the API reference from src/**.csproj, which multi-targets
+          # net8.0;net9.0;net10.0, so every targeted SDK must be present.
+          dotnet-version: |
+            8.0.x
+            9.0.x
+            10.0.x
       - name: Setup Github Pages
-        uses: actions/configure-pages@v3        
+        uses: actions/configure-pages@v6
+      - name: Install DocFX
+        run: |
+          dotnet tool install -g docfx
+          echo "$HOME/.dotnet/tools" >> "$GITHUB_PATH"
       - name: Build DocFX Site
-        uses: nunit/docfx-action@v2.10.0
-        with:
-         args: docs/docfx.json
+        run: docfx docs/docfx.json
       - name: Upload docfx built site artifact
-        uses: actions/upload-pages-artifact@v1
+        uses: actions/upload-pages-artifact@v5
         with:
          path: docs/_site
 
   # Deployment job
   deploy:
+    # Never deploy from a pull request, only from pushes and manual runs
+    if: github.event_name != 'pull_request'
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
@@ -54,4 +71,4 @@ jobs:
     steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v1
+        uses: actions/deploy-pages@v5

docs/articles/configuration.md

@@ -25,8 +25,8 @@ Configuration of Examine indexes is done with [.NET's Options pattern](https://d
 There are several options that can be configured, the most common ones are:
 
 * __FieldDefinitions__ _[`FieldDefinitionCollection`](xref:Examine.FieldDefinitionCollection)_ - Manages the mappings between a field name and it's index value type
-* __Analyzer__ _`Analyzer`_ - The default Lucene Analyzer to use for each field (default = [`StandardAnalyzer`](https://lucenenet.apache.org/docs/4.8.0-beta00016/api/analysis-common/Lucene.Net.Analysis.Standard.StandardAnalyzer.html))
-* __Validator__ _[`IValueSetValidator`]([`IValueSetValidator`](xref:Examine.IValueSetValidator))_ - Used to validate a value set to be indexed, if validation fails it will not be indexed
+* __Analyzer__ _`Analyzer`_ - The default Lucene Analyzer to use for each field (default = [`StandardAnalyzer`](https://lucenenet.apache.org/docs/4.8.0-beta00018/api/analysis-common/Lucene.Net.Analysis.Standard.StandardAnalyzer.html))
+* __Validator__ _[`IValueSetValidator`](xref:Examine.IValueSetValidator)_ - Used to validate a value set to be indexed, if validation fails it will not be indexed
 * __[IndexValueTypesFactory](xref:Examine.Lucene.IFieldValueTypeFactory)__ _`IReadOnlyDictionary<string, IFieldValueTypeFactory>`_ - Allows you to define custom Value Types
 
 ```cs
@@ -35,7 +35,7 @@ There are several options that can be configured, the most common ones are:
 /// </summary>
 public sealed class ConfigureIndexOptions : IConfigureNamedOptions<LuceneDirectoryIndexOptions>
 {
-    public void Configure(string name, LuceneDirectoryIndexOptions options)
+    public void Configure(string? name, LuceneDirectoryIndexOptions options)
     {
         switch (name)
         {
@@ -52,39 +52,22 @@ public sealed class ConfigureIndexOptions : IConfigureNamedOptions<LuceneDirecto
 }
 ```
 
-### After construction
+### At registration
 
-You can modify the field definitions [FieldDefinitionCollection](xref:Examine.FieldDefinitionCollection) for an index after it is constructed by using any of the following methods:
-
-* `myIndex.FieldDefinitionCollection.TryAdd`
-* `myIndex.FieldDefinitionCollection.AddOrUpdate`
-* `myIndex.FieldDefinitionCollection.GetOrAdd`
-
-These modifications __must__ be done before any indexing or searching is executed.
-
-### Add a field value type after construction
-
-It is possible to add custom field value types after the construction of the index, but this must be done before the index is used. Some people may prefer this method of adding custom field value types. Generally, these should be modified directly after the construction of the index.
+The same options can be set inline when the index is registered, without a separate
+[`IConfigureNamedOptions`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.options.iconfigurenamedoptions-1)
+implementation:
 
 ```cs
-// Create the index with all of the defaults
-var myIndex = new LuceneIndex(
-    "MyIndex",
-    new SimpleFSDirectory(new DirectoryInfo("C:\\TestIndexes")));
-
-// Add a custom field value type
-myIndex.FieldValueTypeCollection.ValueTypeFactories
-    .TryAdd(
-        "phonenumber", 
-        name => new GenericAnalyzerFieldValueType(
-            name, 
-            new PhoneNumberAnalyzer()));
-
-// Map a field to use the custom field value type
-myIndex.FieldDefinitionCollection.TryAdd(
-    new FieldDefinition("Phone", "phonenumber"));
+services.AddExamineLuceneIndex("MyIndex", options =>
+{
+    options.FieldDefinitions = new FieldDefinitionCollection(
+        new FieldDefinition("Price", FieldDefinitionTypes.Double));
+});
 ```
 
+An index's field definitions are fixed once the index is in use. [`IIndex.FieldDefinitions`](xref:Examine.IIndex#Examine_IIndex_FieldDefinitions) exposes them as a [`ReadOnlyFieldDefinitionCollection`](xref:Examine.ReadOnlyFieldDefinitionCollection) for inspection only - all configuration __must__ be done through the options above, before any indexing or searching is executed.
+
 ## Value types
 
 Value types are responsible for:
@@ -163,7 +146,7 @@ public sealed class ConfigureIndexOptions : IConfigureNamedOptions<LuceneDirecto
     public ConfigureIndexOptions(ILoggerFactory loggerFactory)
         => _loggerFactory = loggerFactory;
 
-    public void Configure(string name, LuceneDirectoryIndexOptions options)
+    public void Configure(string? name, LuceneDirectoryIndexOptions options)
     {
         switch (name)
         {
@@ -215,71 +198,88 @@ That returns an result [`ValueSetValidationResult`](xref:Examine.ValueSetValidat
 
 Examine only has one implementation: [`ValueSetValidatorDelegate`](xref:Examine.Lucene.Providers.ValueSetValidatorDelegate) which can be used by developers as a simple way to create a validator based on a callback, else developers can implement this interface if required. By default, no ValueSet validation is done with Examine.
 
+## Index behavior options
+
+[`LuceneIndexOptions`](xref:Examine.Lucene.LuceneIndexOptions) also controls how the underlying Lucene index reader and writer behave. These are set the same way as any other option.
+
+* __NrtEnabled__ _`bool`_ - Whether Near Real-Time (NRT) searching is enabled. Default `true`. With NRT enabled, searchers see indexed documents without waiting for a commit.
+* __NrtTargetMinStaleSec__ _`double`_ - The lower staleness bound for NRT reopens, in seconds. Default `1.0`.
+* __NrtTargetMaxStaleSec__ _`double`_ - The upper staleness bound for NRT reopens, in seconds. Default `60.0`.
+* __NrtCacheMaxMergeSizeMB__ _`double`_ - Maximum merge size in megabytes held in the NRT cache. Default `5.0`.
+* __NrtCacheMaxCachedMB__ _`double`_ - Maximum total megabytes held in the NRT cache. Default `60.0`.
+* __IndexDeletionPolicy__ _`IndexDeletionPolicy`_ - The Lucene index deletion policy. Required when [replicating](xref:replication) an index.
+* __UnlockIndex__ _`bool`_ - On [`LuceneDirectoryIndexOptions`](xref:Examine.Lucene.LuceneDirectoryIndexOptions). If `true`, forcibly unlocks the index directory on startup.
+* __DirectoryFactory__ _[`IDirectoryFactory`](xref:Examine.Lucene.Directories.IDirectoryFactory)_ - On [`LuceneDirectoryIndexOptions`](xref:Examine.Lucene.LuceneDirectoryIndexOptions). Determines the Lucene `Directory` backing the index.
+
+```cs
+services.AddExamineLuceneIndex("MyIndex", options =>
+{
+    // Reopen the searcher more aggressively than the default
+    options.NrtTargetMinStaleSec = 0.5;
+    options.NrtTargetMaxStaleSec = 10.0;
+});
+```
+
+Lowering the staleness targets makes newly indexed documents searchable sooner at the cost of more frequent reader reopens.
+
 ## Facets configuration
 
 When using the facets feature it's possible to add facets configuration to change the behavior of the indexing.
 
 For example, you can allow multiple values in an indexed field with the configuration below.
 ```csharp
-// Create a config
-var facetsConfig = new FacetsConfig();
-
-// Set field to be able to contain multiple values (This is default for a field in Examine. But you only need this if you are actually using multiple values for a single field)
-facetsConfig.SetMultiValued("MultiIdField", true);
-
-services.AddExamineLuceneIndex("MyIndex",
+services.AddExamineLuceneIndex("MyIndex", options =>
+{
     // Set the indexing of your fields to use the facet type
-    fieldDefinitions: new FieldDefinitionCollection(
+    options.FieldDefinitions = new FieldDefinitionCollection(
         new FieldDefinition("Timestamp", FieldDefinitionTypes.FacetDateTime),
+        new FieldDefinition("MultiIdField", FieldDefinitionTypes.FacetFullText));
 
-        new FieldDefinition("MultiIdField", FieldDefinitionTypes.FacetFullText)
-        ),
-    // Pass your config
-    facetsConfig: facetsConfig
-    );
+    // Set field to be able to contain multiple values. (This is default for a field in Examine,
+    // so you only need this if you are actually using multiple values for a single field.)
+    options.FacetsConfig.SetMultiValued("MultiIdField", true);
+});
 ```
 
 Without this configuration for multiple values, you'll notice that your faceted search breaks or behaves differently than expected.
 
 ### Hierarchical and Taxonomy Facets configuration
 
-To enable support for hierarchical facets as well as supporting faster faceting the Taxonomy Facet sidecar index can be enabled.
+To enable support for hierarchical facets as well as supporting faster faceting the Taxonomy Facet sidecar index can be used.
+
+[`LuceneIndexOptions.UseTaxonomyIndex`](xref:Examine.Lucene.LuceneIndexOptions#Examine_Lucene_LuceneIndexOptions_UseTaxonomyIndex) is __`true` by default__. Setting it to `false` switches faceting over to `SortedSetDocValues` and no sidecar index is written.
 
-1. Set LuceneIndexOptions.UseTaxonomyIndex = true; for the index. This enables the use of the Taxonomy sidecar index.
-2. Change the Field Definitions to use the "FacetTaxonomy" Field Definition Types instead of the "Facet" types. E.g. FieldDefinitionTypes.FacetFullText => FieldDefinitionTypes.FacetTaxonomyFullText.
-3. To enable hierarchical facets on a field, call FacetsConfig.SetHierarchical("facetfieldname", true);
+1. Leave `LuceneIndexOptions.UseTaxonomyIndex` as `true` for the index (or set it explicitly).
+2. Change the Field Definitions to use the "FacetTaxonomy" Field Definition Types instead of the "Facet" types. E.g. `FieldDefinitionTypes.FacetFullText` => `FieldDefinitionTypes.FacetTaxonomyFullText`.
+3. To enable hierarchical facets on a field, call `FacetsConfig.SetHierarchical("facetfieldname", true)`.
 
 Example:
 
 ```csharp
-// Create a config
-var facetsConfig = new FacetsConfig();
-
-// Set field to be able to support hierarchical facets
-facetsConfig.SetHierarchical("hierarchyFacetfield", true);
-
-// Set field to be able to contain multiple values (This is default for a field in Examine. But you only need this if you are actually using multiple values for a single field)
-facetsConfig.SetMultiValued("MultiIdField", true);
-
-services.AddExamineLuceneIndex("MyIndex",
+services.AddExamineLuceneIndex("MyIndex", options =>
+{
     // Set the indexing of your fields to use the facet Taxonomy type
-    fieldDefinitions: new FieldDefinitionCollection(
+    options.FieldDefinitions = new FieldDefinitionCollection(
         new FieldDefinition("Timestamp", FieldDefinitionTypes.FacetTaxonomyDateTime),
         new FieldDefinition("hierarchyFacetfield", FieldDefinitionTypes.FacetTaxonomyFullText),
+        new FieldDefinition("MultiIdField", FieldDefinitionTypes.FacetTaxonomyFullText));
+
+    // Use the Taxonomy sidecar index (this is the default)
+    options.UseTaxonomyIndex = true;
+
+    // Set field to be able to support hierarchical facets
+    options.FacetsConfig.SetHierarchical("hierarchyFacetfield", true);
 
-        new FieldDefinition("MultiIdField", FieldDefinitionTypes.FacetTaxonomyFullText)
-        ),
-    // Pass your config
-    facetsConfig: facetsConfig,
-    // Enable the Taxonomy sidecar index
-    useTaxonomyIndex: true
-    );
+    // Set field to be able to contain multiple values. (This is default for a field in Examine,
+    // so you only need this if you are actually using multiple values for a single field.)
+    options.FacetsConfig.SetMultiValued("MultiIdField", true);
+});
 ```
 
 **Note: See more examples of how facets configuration can be used under [Searching](xref:searching)**
 
 To explore other configuration settings see the links below:
-- [FacetsConfig API docs](https://lucenenet.apache.org/docs/4.8.0-beta00016/api/facet/Lucene.Net.Facet.FacetsConfig.html#methods)
+- [FacetsConfig API docs](https://lucenenet.apache.org/docs/4.8.0-beta00018/api/facet/Lucene.Net.Facet.FacetsConfig.html#methods)
 - [Facets with lucene](https://norconex.com/facets-with-lucene/). See how the config is used in the code examples.
 
 ## Luke

docs/articles/indexing.md

@@ -208,9 +208,9 @@ private void IndexingError(object sender, IndexingErrorEventArgs e)
 
 #### [LuceneIndex.DocumentWriting](xref:Examine.Lucene.Providers.LuceneIndex#Examine_Lucene_Providers_LuceneIndex_DocumentWriting)
 
-If using Examine with the default Lucene implementation then the [`IIndex`](xref:Examine.IIndex) implementation will be [`LuceneIndex`](xref:Examine.Lucene.Providers.LuceneIndex). This event provides access to the Lucene [`Document`](https://lucenenet.apache.org/docs/4.8.0-beta00016/api/core/Lucene.Net.Documents.Document.html) object before it gets added to the Lucene Index.
+If using Examine with the default Lucene implementation then the [`IIndex`](xref:Examine.IIndex) implementation will be [`LuceneIndex`](xref:Examine.Lucene.Providers.LuceneIndex). This event provides access to the Lucene [`Document`](https://lucenenet.apache.org/docs/4.8.0-beta00018/api/core/Lucene.Net.Documents.Document.html) object before it gets added to the Lucene Index.
 
-You can use this event to entirely customize how the data is stored in the Lucene index, including adding custom boosting profiles, changing the [`Document`](https://lucenenet.apache.org/docs/4.8.0-beta00016/api/core/Lucene.Net.Documents.Document.html)'s field values or types, etc...
+You can use this event to entirely customize how the data is stored in the Lucene index, including adding custom boosting profiles, changing the [`Document`](https://lucenenet.apache.org/docs/4.8.0-beta00018/api/core/Lucene.Net.Documents.Document.html)'s field values or types, etc...
 
 Example of how to listen the event: