v8.0.0-beta1: NEST → Elastic.Clients.Elasticsearch

v8.0.0-beta1

This is the first beta of Foundatio.Repositories v8.0.0, focused on migrating the Elasticsearch implementation from the legacy NEST client stack to the official Elasticsearch .NET client (Elastic.Clients.Elasticsearch, 8.x generation), coordinated with Foundatio.Parsers.ElasticQueries builds that target that same client generation. The v8 package version reflects breaking changes in this repository library only; it is unrelated to how you schedule Elasticsearch cluster upgrades. CI runs against Elasticsearch 9.4 (docker-compose.yml) for compatibility coverage. In production you can run Elastic.Clients.Elasticsearch 8.x against Elasticsearch 8 and defer Elasticsearch 9 until you are ready.

Staged migration (client vs cluster)

• Elastic.Clients.Elasticsearch 8.x is the practical bridge off NEST: move application code to this client generation while clusters stay on Elasticsearch 8, without bundling a forced Elasticsearch 9 server upgrade into the same cutover.
• The Elasticsearch .NET 9 client generation pairs with Elasticsearch 9 server expectations; deferring that pair until after you have stabilized on client 8.x shrinks blast radius and scheduling risk.
• When you later upgrade Elasticsearch 8 → 9, no code changes are required. This codebase has already been exercised against a Elastic.Clients.Elasticsearch 9.x client with minimal follow-up changes, so most teams should not need another sweeping rewrite for that step alone to support Elasticsearch 10.

The same Elasticsearch .NET migration for Foundatio.Parsers.ElasticQueries (build APIs, visitors, System.Text.Json vs Json.NET assumptions, and related breaking changes) is documented on the companion Foundatio.Parsers v8.0.0 release; read that alongside this beta when you touch searchable query or aggregation code paths.

Migration (Repositories): See the in-repo checklist at docs/guide/upgrading-to-elastic-clients-elasticsearch.md for connection settings, mapping configuration (ConfigureIndex / ConfigureIndexMapping), response validation (IsValidResponse), and other repository- and client-facing breaking changes.

Migration (ElasticQueries / parsers): Follow Foundatio.Parsers v8.0.0 and the docs site linked there (including Upgrading to Elastic.Clients.Elasticsearch) for parser and query-builder migration detail.

Themes

• Staged upgrades: Use Elastic.Clients.Elasticsearch 8.x with Elasticsearch 8 before you adopt Elasticsearch 9 and a later 9 client generation (see Staged migration above).
• Official Elasticsearch .NET client: End-to-end adoption of Elastic.Clients.Elasticsearch (via Foundatio.Parsers.ElasticQueries packages built for that client) replacing NEST 7.17.x for query building and low-level Elasticsearch operations.
• Repository + infrastructure parity: Broad updates across index lifecycle, bulk/reindex paths, aggregations, JSON patch, caching/serialization edges, and test harnesses to match new client behavior and server semantics.
• Reliability & correctness: Follow-up hardening in bulk/reindex, aggregation metadata casting, paging with projections, bounded retries, and JSON utilities surfaced during the migration.
• Repository package semver: Foundatio.Repositories v8 signals breaking changes in this Elasticsearch implementation; see MinVerMinimumMajorMinor 8.0. That is independent of Elasticsearch server or Elastic.Clients.Elasticsearch client upgrade timing.

Breaking changes

• Elasticsearch client generation: Consumers that depended on NEST / Elasticsearch.Net types flowing from Foundatio.Parsers / repository internals must migrate to Elastic.Clients.Elasticsearch APIs (or isolate those dependencies behind your own abstractions). Foundatio.Parsers 7.18.x (NEST 7.17.5) → 8.0.0 (Elastic.Clients.Elasticsearch 8.19.x in current Parsers source). Parser-specific return-type and visitor changes are summarized on Foundatio.Parsers v8.0.0. by @niemyjski in #216 (82296c0c)
• Foundatio.Parsers.ElasticQueries major-line bump: 7.18.3 → 8.0.0 on Foundatio.Repositories.Elasticsearch.csproj. Align your solution on compatible Foundatio.Parsers pre-releases when using searchable repositories; when you move to Foundatio.Parsers 8.0.0 stable, use Foundatio.Parsers v8.0.0 as the dependency and migration reference. by @niemyjski in 1c9be060
• Public / behavioral API shifts (high level; see migration doc for exhaustive detail):
  • Index and alias operations updated for new client patterns (e.g. index name / Indices usage, settings updates requiring reopen where applicable).
  • Bulk and hit validity checks aligned with client response models (e.g. IsValid / validity semantics rather than legacy status-code assumptions).
  • Reindex/versioning behavior: in-flight tests and code account for seq_no / primary_term style concurrency rather than preserved _version across reindex in all scenarios (see PR description in #216).
  • Obsolete / legacy hooks removed or replaced (e.g. obsolete search helpers and dead serializer utilities removed as part of the migration; verify any custom subclasses).

Added

• Elasticsearch .NET (Elastic.Clients.Elasticsearch) integration across repositories, jobs, utilities, and tests, replacing the NEST-based stack carried by Foundatio.Parsers 7.x. by @niemyjski in #216 (82296c0c)
• JSONPath-style patch support restored in JsonPatcher with an internal filter evaluator (and related tests) as part of feature parity work during the upgrade. by @niemyjski in #216 (82296c0c)
• TopHits aggregation caching path: raw hit JSON serialization for round-tripping / caching scenarios, with converter updates for tophits. by @niemyjski in #216 (82296c0c)
• ElasticUtility task / snapshot helpers completed or stubbed to expected behavior (WaitForTaskAsync, WaitForSafeToSnapshotAsync, snapshot deletion helpers, etc.) for parity with operations used by repositories and jobs. by @niemyjski in #216 (82296c0c)
• Migration guide: docs/guide/upgrading-to-elastic-clients-elasticsearch.md with breaking-change checklist and doc updates across getting-started and configuration examples. by @niemyjski in #216 (82296c0c)

Changed

• Wildcard / index resolution paths updated (e.g. ResolveIndexAsync vs GetAsync) to avoid client/server mismatches where the server rejects requests with unexpected bodies. by @niemyjski in #216 (82296c0c)
• Docs and samples updated for modern connection settings, ConfigureIndex / ConfigureIndexMapping patterns, and IsValidResponse-style validation. by @niemyjski in #216 (82296c0c)
• Test infrastructure: xUnit v3-oriented fixes (e.g. cancellation token patterns), semantic JSON comparison for serialization-sensitive tests, and stability fixes for reindex/alias scenarios. by @niemyjski in #216 (82296c0c)

Fixed

• Aggregation metadata (Meta) deserialization: fixed InvalidCastException paths when metadata maps were exposed as unexpected dictionary interfaces under the new client (multiple aggregatio...

v7.18.3

What's Changed

• Fix concurrency and correctness issues in reindex and bulk operations by @niemyjski in #272

Full Changelog: v7.18.2...v7.18.3

v7.18.2

What's Changed

• chore(deps): bump elasticsearch/elasticsearch from 8.19.14 to 8.19.15 by @dependabot[bot] in #268
• chore(deps): bump kibana/kibana from 8.19.14 to 8.19.15 by @dependabot[bot] in #267
• fix: make ExistsAsync(id) real-time for soft-delete models by @niemyjski in #270
• fix: improve reindex reliability with distributed locking and observability by @niemyjski in #271

Full Changelog: v7.18.1...v7.18.2

v7.18.1

What's Changed

• Serialize concurrent ConfigureIndexesAsync with distributed lock + cache marker by @niemyjski in #244

Full Changelog: v7.18.0...v7.18.1

v7.18.0

v7.18.0

This is a major feature release for Foundatio.Repositories, adding nested field support, FieldConditions with range operators and boolean grouping, AfterQuery hooks, patch modification tracking, and comprehensive NRT safety. This release depends on Foundatio v13.0.0 and Foundatio.Parsers v7.18.0.

Documentation: repositories.foundatio.dev -- significant doc updates in this release including Elasticsearch consistency & dirty reads guide, patch notifications, default search fields & CopyTo patterns, FieldConditions range operators, and EntityChanged snippets.

Themes:

• Nested Field Support: Full query, aggregation, and serialization support for nested Elasticsearch fields
• FieldConditions: Range operators, boolean grouping, and runtime validation for field-level access control
• Patch Operations: Modification status tracking, automatic UpdatedUtc date setting, and nested property path support in scripts
• AfterQuery Hooks: Post-query result transformation and required field guarantees
• NRT Safety: Full nullable reference type audit with TreatWarningsAsErrors
• Foundatio v13 Core: .NET 10 target, xUnit v3, two-phase dispose

Breaking Changes

• Notification delivery delay is now opt-in: Previously notifications had a default delay. by @niemyjski in #197 (71fa2dac)
• Custom field helpers throw RepositoryException on unsupported types: Previously silently failed. by @niemyjski in #264 (e514e14b)
• Custom field null safety, FindOneAsync Empty, Join and Exists guards: Tighter validation. by @niemyjski in #263 (8492891e)
• Removed obsolete LogTraceRequest method. by @niemyjski in a3a86c03
• Removed obsolete SearchAsync and CountBySearchAsync methods -- deprecated since v7.15. by @niemyjski in 92809baf
• Updated to Foundatio 13.0.0 core -- see migration guide
• NRT annotations updated across entire codebase
• FieldConditions now validate field mappings at runtime: FieldEquals, FieldNotEquals, and string range operators (FieldGreaterThan, FieldGreaterThanOrEqual, FieldLessThan, FieldLessThanOrEqual) now throw QueryValidationException if the target field is an analyzed text field with no .keyword sub-field. Previously, these operators silently generated a TermQuery against tokenized data, which almost never matched correctly. If your index mappings use plain .Text() without a .keyword sub-field, you must either:
  • Add a .keyword sub-field: m.Text(p => p.Name(f => f.Property).Fields(f => f.Keyword(k => k.Name("keyword")))) or use .AddKeywordField() / .AddKeywordAndSortFields()
  • Switch to FieldContains() / FieldNotContains() for full-text token matching instead of exact matching
    Similarly, FieldContains and FieldNotContains now throw QueryValidationException if used on a non-analyzed (keyword) field. Switch to FieldEquals() / FieldNotEquals() for exact matching on keyword fields.
    Additional new validations that may throw QueryValidationException:
  • Range operators (FieldGreaterThan, etc.) with null values — use FieldHasValue() / FieldEmpty() or the *If variants instead
  • Range operators with collection values — use a single scalar value or FieldEquals() for multi-value matching
  • FieldEquals(f => f.IsDeleted, true) in ActiveOnly mode (the default for soft-delete models) — the soft-delete filter already excludes deleted documents, creating a contradictory query that returns zero results. Use .IncludeSoftDeletes() or .SoftDeleteMode(SoftDeleteQueryMode.DeletedOnly) instead
    Properties/mappings to check: Review all index configuration classes for text fields used with FieldEquals/FieldNotEquals. Any .Text() mapping that does not call .AddKeywordField(), .AddKeywordAndSortFields(), or manually define .Fields(f => f.Keyword(k => k.Name("keyword"))) will now throw at query time. by @niemyjski in #243(989503a9)

Added

• Nested field support: Full query, aggregation tests, and serialization for nested Elasticsearch fields. by @niemyjski in #150 (a19bed03)
• FieldConditions range operators, boolean grouping, and runtime validation. by @niemyjski in #243 (989503a9)
• AfterQuery event hook for post-query result transformation. by @niemyjski in #237 (0e0650cd)
• AddRequiredField for guaranteed _source field inclusion. by @niemyjski in #236 (d47aa39a)
• PatchAsync returns modification status. by @niemyjski in #235 (e8377d11)
• Auto-set UpdatedUtc for all patch operations with extensible date tracking. by @niemyjski in #227 (636cf235)
• Patch notification documentation and tests. by @niemyjski in #262 (979a1f0c)
• Default Id sort for deterministic pagination. by @niemyjski in #203 (98a6a208)
• SetCacheClient made protected to allow per-repository cache override. by @niemyjski in #228 (077e52e3)
• Log warning when terms aggregations contain doc_count_error_upper_bound. by @niemyjski in #223
• Added EntityChanged serialization round-trip tests. by @niemyjski in ed18b876

Changed

• Fix NRT code smells across repository layer. by @niemyjski in #265 (b4340a1a)
• Fix _isEnsured premature caching, aggregation data loss, and missing AnyContext calls. by @niemyjski in #222
• Fix argument validation, error handling, and exception correctness. by @niemyjski in #221
• Fix code quality: cache invalidation, resource disposal, null safety, and test assertions. by @niemyjski in #218
• Fix response validation, performance, and resource management issues. by @niemyjski in #217
• Renamed partial parameter to patch in ApplyDateTracking(PartialPatch) -- partial is a C# keyword. by @niemyjski in aa5a20b0
• Fix thread-safe double-dispose in Index, ElasticConfiguration, and ElasticReadOnlyRepositoryBase. by @niemyjski in 5323c41d
• Target framework updated to .NET 8, .NET 10
• Upgraded to xUnit v3

Fixed

• RenameFieldScript and RemoveFieldScript now support nested property paths. by @niemyjski in #232 (cd0f5243)
• Bulk error handling: Improved retry, cache, and exception consistency. by @niemyjski in #229 (95423e9a)
• FieldConditionsQueryBuilder now consults per-query QueryFieldResolver. by @niemyjski in #225 (cde847df)
• System.Text.Json deserialization for FindResults. by @niemyjski in #200
• Thread-safety in calculated field script evaluation. by @niemyjski in #185
• Get by ID(s) with cache key no longer sets the id-based cache key. by @niemyjski in [#161](https://github.com/FoundatioFx/Foundatio...

v7.18.0-beta6

What's Changed

• Fix response validation, performance, and resource management issues by @niemyjski in #217
• Fix code quality: cache invalidation, resource disposal, null safety, and test assertions by @niemyjski in #218
• Bump Jint from 4.6.0 to 4.6.1 by @dependabot[bot] in #220
• Fix argument validation, error handling, and exception correctness by @niemyjski in #221
• Fix _isEnsured premature caching, aggregation data loss, and missing AnyContext calls by @niemyjski in #222
• Log warning when terms aggregations contain doc_count_error_upper_bound by @niemyjski in #223
• Fix FieldConditionsQueryBuilder to consult per-query QueryFieldResolver by @niemyjski in #224
• Fix FieldConditionsQueryBuilder to consult per-query QueryFieldResolver by @niemyjski in #225
• Assert nested Meta property is excluded in include/exclude tests by @niemyjski in #226
• Auto-set UpdatedUtc for all patch operations with extensible date tracking by @niemyjski in #227
• Fix bulk error handling: retry, cache, and exception consistency by @niemyjski in #229
• Fix RenameFieldScript and RemoveFieldScript to support nested property paths by @niemyjski in #232
• Return modification status from PatchAsync overloads by @niemyjski in #235
• Add AddRequiredField for guaranteed _source field inclusion by @niemyjski in #236
• Add AfterQuery event hook for post-query result transformation by @niemyjski in #237
• Bump rollup from 4.57.1 to 4.59.0 in /docs by @dependabot[bot] in #213
• Bump dompurify from 3.3.1 to 3.3.2 in /docs by @dependabot[bot] in #230
• Bump minimatch from 10.1.2 to 10.2.4 in /docs by @dependabot[bot] in #214
• Enhance FieldConditions with range operators, boolean grouping, and runtime validation by @niemyjski in #243
• Fix NRT errors and enable TreatWarningsAsErrors by @niemyjski in #254
• Bump vite from 7.3.1 to 7.3.2 in /docs by @dependabot[bot] in #253
• Bump picomatch from 4.0.3 to 4.0.4 in /docs by @dependabot[bot] in #245
• Bump lodash-es from 4.17.23 to 4.18.1 in /docs by @dependabot[bot] in #258
• Bump Jint from 4.6.4 to 4.8.0 by @dependabot[bot] in #256

Full Changelog: v7.18.0-beta4...v7.18.0-beta6

v7.18.0-beta4

What's Changed

• Adds nested field support with query, aggregation tests and serialization fixes by @niemyjski in #150

Full Changelog: v7.18.0-beta3...v7.18.0-beta4

v7.18.0-beta3

Full Changelog: v7.18.0-beta2...v7.18.0-beta3

v7.18.0-beta2

What's Changed

• BREAKING CHANGE: Make notification delivery delay opt-in by @niemyjski in #197
• Fix System.Text.Json deserialization for FindResults by @niemyjski in #200
• fix: Add default Id sort for deterministic pagination by @niemyjski in #203

Full Changelog: v7.18.0-beta1...v7.18.0-beta2

v7.18.0-beta1

What's Changed

• Passes TimeProvider to date setting extensions by @niemyjski in #184
• Fixes thread-safety issue in calculated field script evaluation by @niemyjski in #185
• Bump elasticsearch/elasticsearch from 8.19.5 to 8.19.9 by @dependabot[bot] in #189
• Bump kibana/kibana from 8.19.5 to 8.19.9 by @dependabot[bot] in #188

Full Changelog: v7.17.17...v7.18.0-beta1