Adds nested field support with query, aggregation tests and serialization fixes (#150)

* Adds nested field support in tests

Introduces nested field support and related tests.

This commit addresses the need for testing nested fields.
It includes a new test file, updates configurations,
and adds utilities for generating test data with nested properties.
A nested field is also added to the default search fields.

* formatting updates

* Adds nested aggregations filtering tests

Adds tests to verify filtering of nested aggregations using include and exclude.

This ensures that aggregations can be filtered based on specific values within nested fields, providing more granular control over the aggregation results.

Also improves readability in existing tests.

* Fixes nested aggregation serialization

Addresses issues with deserializing nested aggregations in Elasticsearch queries.

This change ensures that nested aggregations, particularly those involving terms aggregations on nested fields, are correctly serialized and deserialized. It corrects the format of aggregations expressions in tests and handles the deserialization of SingleBucketAggregates correctly.

* Adds comprehensive nested field query tests

Updates Foundatio.Parsers.ElasticQueries dependency to support enhanced nested field parsing.
Introduces new test cases for various nested query scenarios in FindAsync:
- Queries on individual nested fields without group syntax.
- Nested range queries.
- Nested AND conditions without group syntax.
- Queries mixing nested and non-nested fields.
- Negated nested group queries.

Refactors existing test code for clarity and minor performance improvements.

* Add comprehensive serialization round-trip tests and upgrade parsers

- Add round-trip tests for all aggregate types (ValueAggregate, StatsAggregate, ExtendedStatsAggregate, PercentilesAggregate, ObjectValueAggregate, DateValueAggregate, BucketAggregate, SingleBucketAggregate)

- Add round-trip tests for all bucket types (KeyedBucket<string>, KeyedBucket<double>, DateHistogramBucket, RangeBucket) with inner aggregation preservation

- Add deep nesting round-trip test (sbucket -> bucket -> keyed bucket -> value)

- Upgrade Foundatio.Parsers.ElasticQueries to 7.18.0-beta4

- Simplify SetDefaultFields to use plain strings consistently

Made-with: Cursor

* Fix serializer converters to properly handle non-public setters

Both Newtonsoft and System.Text.Json converters now explicitly extract Items for PercentilesAggregate, matching the existing pattern used by BucketsNewtonsoftJsonConverter for inner aggregations. This fixes a latent bug where serializer.Populate and element.Deserialize could not set properties with internal/protected setters through the custom converters. Reverts PercentilesAggregate.Items back to internal set to preserve the original API contract.

Made-with: Cursor

* Add isolated serializer tests and deep nesting coverage

Re-adds isolated Newtonsoft and System.Text.Json tests for SingleBucketAggregate to catch serializer-specific regressions. Adds 3-level deep sbucket nesting test and mixed sub-aggregation test (stats + percentiles + bucket-with-sub-agg inside a single bucket).

Made-with: Cursor

* Refactor converters to use pattern matching and readonly types

Replaces if/switch chains with switch expressions in both Newtonsoft and STJ aggregate converters. Extracts dedicated methods for percentiles and sbucket deserialization. Deserializes to IReadOnlyList to skip constructor allocations. Removes serializer-specific tests in favor of the existing all-serializers loop test.

Made-with: Cursor

* Clean up GetProperty and document tophits limitation

Makes GetProperty and GetTokenType static directly, removing the unnecessary GetPropertyStatic/GetProperty indirection. Adds comments to both converters explaining why TopHitsAggregate cannot be round-tripped.

Made-with: Cursor

* Remove JsonConstructor attributes, handle all types explicitly in converters

SingleBucketAggregate and MultiBucketAggregate no longer need JsonConstructor attributes. The STJ converter now explicitly extracts Aggregations for sbucket (matching the Newtonsoft converter pattern), so both serializers handle non-public setters consistently through converter-level extraction rather than relying on STJ fallback behavior.

Made-with: Cursor

* updated deps

* Ensures culture-invariant string casing

Updates `ToLower()` calls to `ToLowerInvariant()` for JSON property name lookups and Elasticsearch default field configurations. This promotes consistent string comparison and conversion, mitigating potential issues with culture-specific casing rules across different environments.

Adds documentation to `AggregationQueryTests` explaining the limitations of JSON round-tripping for `TopHitsAggregate` due to its internal use of `ILazyDocument` references.

Refactors test setup in `NestedFieldTests` by renaming `utcToday` to `baseDate` for improved clarity, better reflecting its role as a fixed reference point in the tests.

* Docs: Add comprehensive nested query/aggregation guide

Adds detailed documentation explaining how the framework automatically handles Elasticsearch nested types.

Explains how filter and aggregation expressions on nested fields are automatically wrapped in appropriate nested queries and aggregations. Covers:
- Basic and advanced nested query patterns
- Nested field aggregations and result access
- Sorting, _exists_/_missing_, and default fields with nested paths
- The Top Hits aggregation and its serialization limitations

Author: niemyjski

docs/guide/elasticsearch-setup.md

@@ -380,6 +380,8 @@ This creates:
     ))
 ```
 
+Fields mapped as `nested` are automatically wrapped in Elasticsearch `nested` queries and `nested` aggregations when queried through filter or aggregation expressions. See [Nested Queries](/guide/querying#nested-queries) and [Nested Field Aggregations](/guide/querying#nested-field-aggregations) for details and examples.
+
 ### Index Settings
 
 ```csharp

docs/guide/querying.md

@@ -296,6 +296,206 @@ Console.WriteLine($"Average Salary: {avgSalary}");
 Console.WriteLine($"Latest Created: {maxDate}");
 ```
 
+### Nested Field Aggregations
+
+When aggregating on fields that are mapped as `nested` in Elasticsearch, the framework automatically wraps the aggregation in a nested aggregation context. Use the same `parentObject.childField` syntax you use for flat fields:
+
+```csharp
+// Terms aggregation on a nested field
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:peerReviews.rating"));
+
+// Multiple aggregation types on nested fields
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:peerReviews.reviewerEmployeeId min:peerReviews.rating max:peerReviews.rating"));
+```
+
+The framework detects nested fields via the index mapping and groups all nested aggregations under a single `SingleBucketAggregate` keyed by the nested path. Access results through that wrapper:
+
+```csharp
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:peerReviews.rating min:peerReviews.rating max:peerReviews.rating"));
+
+// All nested aggregations are grouped under a single-bucket aggregate
+var nestedAgg = results.Aggregations["nested_peerReviews"] as SingleBucketAggregate;
+
+var ratingTerms = nestedAgg.Aggregations.Terms<int>("terms_peerReviews.rating");
+foreach (var bucket in ratingTerms.Buckets)
+{
+    Console.WriteLine($"Rating {bucket.Key}: {bucket.Total}");
+}
+
+var minRating = nestedAgg.Aggregations.Min("min_peerReviews.rating")?.Value;
+var maxRating = nestedAgg.Aggregations.Max("max_peerReviews.rating")?.Value;
+```
+
+Include and exclude filtering works the same way as non-nested aggregations:
+
+```csharp
+// Only include specific terms
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:(peerReviews.reviewerEmployeeId @include:emp1 @include:emp2)"));
+
+// Exclude specific terms
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:(peerReviews.rating @exclude:1 @exclude:2)"));
+```
+
+### Top Hits Aggregation
+
+The `tophits` sub-aggregation returns the top matching documents within each bucket:
+
+```csharp
+var results = await repository.CountAsync(q => q
+    .AggregationsExpression("terms:(age tophits:_)"));
+
+var bucket = results.Aggregations.Terms<int>("terms_age").Buckets.First();
+var topHits = bucket.Aggregations.TopHits();
+var employees = topHits.Documents<Employee>();
+```
+
+::: warning TopHitsAggregate Cannot Be Serialized
+`TopHitsAggregate` holds `ILazyDocument` references that contain raw Elasticsearch document bytes and require an active serializer instance to materialize into typed objects. These references are lost during JSON serialization, which means:
+
+- **Caching**: `CountResult` or `FindResults` containing `TopHitsAggregate` cannot be cached and restored via JSON serialization (Newtonsoft or System.Text.Json). The top hits data will be `null` after deserialization.
+- **Workaround**: If you need to cache results that include top hits, materialize the documents into concrete types *before* caching, and cache those typed results separately.
+:::
+
+## Nested Queries
+
+When querying fields inside [nested objects](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html), the framework automatically wraps filter expressions in the required Elasticsearch `nested` query. You do not need to manually construct nested queries -- just use dotted field paths.
+
+### Prerequisites
+
+The field must be mapped as `nested` in your index configuration:
+
+```csharp
+public override TypeMappingDescriptor<Employee> ConfigureIndexMapping(
+    TypeMappingDescriptor<Employee> map)
+{
+    return map
+        .Dynamic(false)
+        .Properties(p => p
+            .SetupDefaults()
+            .Keyword(f => f.Name(e => e.Id))
+            .Text(f => f.Name(e => e.Name).AddKeywordAndSortFields())
+            .Nested<PeerReview>(f => f.Name(e => e.PeerReviews).Properties(p1 => p1
+                .Keyword(f2 => f2.Name(p2 => p2.ReviewerEmployeeId))
+                .Scalar(p3 => p3.Rating, f2 => f2.Name(p3 => p3.Rating))))
+        );
+}
+```
+
+### Basic Nested Queries
+
+Query nested fields with standard filter expressions using `parentObject.childField` syntax:
+
+```csharp
+// Exact match on a nested field
+var results = await repository.FindAsync(q => q
+    .FilterExpression("peerReviews.rating:5"));
+
+// Range query on a nested field
+var results = await repository.FindAsync(q => q
+    .FilterExpression("peerReviews.rating:[4 TO 5]"));
+
+// Match on a nested keyword field
+var results = await repository.FindAsync(q => q
+    .FilterExpression("peerReviews.reviewerEmployeeId:bob_456"));
+```
+
+### Combining Nested Conditions
+
+Multiple conditions on the same nested path are combined into a single `nested` query with a `bool` clause:
+
+```csharp
+// AND: both conditions must match the SAME nested document
+var results = await repository.FindAsync(q => q
+    .FilterExpression("peerReviews.rating:5 AND peerReviews.reviewerEmployeeId:bob_456"));
+
+// OR: either condition can match across different nested documents
+var results = await repository.FindAsync(q => q
+    .FilterExpression("peerReviews.rating:>=4 OR peerReviews.reviewerEmployeeId:bob_456"));
+```
+
+### Mixing Nested and Non-Nested Fields
+
+Nested fields and regular fields can be used together. The framework only wraps the nested portions in a `nested` query:
+
+```csharp
+// "name" is a root-level field, "peerReviews.rating" is nested
+var results = await repository.FindAsync(q => q
+    .FilterExpression("name:Alice peerReviews.rating:5"));
+```
+
+### Negating Nested Conditions
+
+```csharp
+// Exclude employees who have any peer review with rating 5
+var results = await repository.FindAsync(q => q
+    .FilterExpression("NOT peerReviews.rating:5"));
+```
+
+### Default Fields with Nested Paths
+
+Nested fields can be included in default search fields via `SetDefaultFields` in your query parser configuration. This allows unqualified search terms to match against nested fields:
+
+```csharp
+protected override void ConfigureQueryParser(ElasticQueryParserConfiguration config)
+{
+    base.ConfigureQueryParser(config);
+    config.SetDefaultFields([
+        nameof(Employee.Id).ToLowerInvariant(),
+        nameof(Employee.Name).ToLowerInvariant(),
+        "peerReviews.reviewerEmployeeId"
+    ]);
+}
+```
+
+With this configuration, a bare search term like `bob_456` will match against `id`, `name`, and `peerReviews.reviewerEmployeeId`:
+
+```csharp
+// Searches id, name, AND the nested peerReviews.reviewerEmployeeId field
+var results = await repository.FindAsync(q => q.SearchExpression("bob_456"));
+```
+
+### Sorting on Nested Fields
+
+Sort expressions on nested fields automatically include the required `nested` context:
+
+```csharp
+// Sort descending by a nested numeric field
+var results = await repository.FindAsync(q => q
+    .SortExpression("-peerReviews.rating"));
+```
+
+### Exists / Missing on Nested Fields
+
+`_exists_` and `_missing_` queries on nested fields are automatically wrapped in a `nested` query:
+
+```csharp
+// Find employees that have at least one peer review with a reviewerEmployeeId
+var results = await repository.FindAsync(q => q
+    .FilterExpression("_exists_:peerReviews.reviewerEmployeeId"));
+```
+
+### Deeply Nested Types
+
+Multi-level nesting (nested objects inside other nested objects) is supported. The framework resolves the correct nested path at each level:
+
+```csharp
+// Given a mapping: parent (nested) -> child (nested inside parent)
+// Query a deeply nested field
+var results = await repository.FindAsync(q => q
+    .FilterExpression("parent.child.field1:value"));
+```
+
+### Known Limitations
+
+| Limitation | Details |
+|---|---|
+| **TopHits round-tripping** | `TopHitsAggregate` cannot survive JSON serialization. See the [Top Hits Aggregation](#top-hits-aggregation) warning above. |
+
 ## Field Selection
 
 Field selection controls which fields are returned from Elasticsearch via `_source` filtering. This reduces network payload and deserialization cost when you only need a subset of fields from a document.

samples/Foundatio.SampleApp/Client/Foundatio.SampleApp.Client.csproj

@@ -7,8 +7,8 @@
   </PropertyGroup>
 
   <ItemGroup>
-    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="8.0.19" />
-    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.DevServer" Version="8.0.19" PrivateAssets="all" />
+    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="10.0.3" />
+    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.DevServer" Version="10.0.3" PrivateAssets="all" />
   </ItemGroup>
 
   <ItemGroup>

samples/Foundatio.SampleApp/Server/Foundatio.SampleApp.Server.csproj

@@ -8,7 +8,7 @@
 
   <ItemGroup>
     <PackageReference Include="Foundatio.Extensions.Hosting" Version="13.0.0-beta3.8" />
-    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.Server" Version="8.0.19" />
+    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.Server" Version="10.0.3" />
   </ItemGroup>
 
   <ItemGroup>

src/Foundatio.Repositories.Elasticsearch/Foundatio.Repositories.Elasticsearch.csproj

@@ -3,7 +3,7 @@
     <Compile Include="..\Foundatio.Repositories\Extensions\TaskExtensions.cs" Link="Extensions\TaskExtensions.cs" />
   </ItemGroup>
   <ItemGroup>
-    <PackageReference Include="Foundatio.Parsers.ElasticQueries" Version="7.18.0-beta3" Condition="'$(ReferenceFoundatioRepositoriesSource)' == '' OR '$(ReferenceFoundatioRepositoriesSource)' == 'false'" />
+    <PackageReference Include="Foundatio.Parsers.ElasticQueries" Version="7.18.0-beta4" Condition="'$(ReferenceFoundatioRepositoriesSource)' == '' OR '$(ReferenceFoundatioRepositoriesSource)' == 'false'" />
     <ProjectReference Include="$(FoundatioProjectsDir)Foundatio.Parsers\src\Foundatio.Parsers.ElasticQueries\Foundatio.Parsers.ElasticQueries.csproj" Condition="'$(ReferenceFoundatioRepositoriesSource)' == 'true'" />
     <ProjectReference Include="..\Foundatio.Repositories\Foundatio.Repositories.csproj" />
   </ItemGroup>

src/Foundatio.Repositories/Models/Aggregations/MultiBucketAggregate.cs

@@ -1,11 +1,12 @@
-using System.Collections.Generic;
+using System.Collections.Generic;
 
 namespace Foundatio.Repositories.Models;
 
 public class MultiBucketAggregate<TBucket> : BucketAggregateBase
     where TBucket : IBucket
 {
     public MultiBucketAggregate() { }
+
     public MultiBucketAggregate(IReadOnlyDictionary<string, IAggregate> aggregations) : base(aggregations) { }
 
     public IReadOnlyCollection<TBucket> Buckets { get; set; } = EmptyReadOnly<TBucket>.Collection;

src/Foundatio.Repositories/Models/Aggregations/PercentilesAggregate.cs

@@ -1,4 +1,4 @@
-using System.Collections.Generic;
+using System.Collections.Generic;
 using System.Diagnostics;
 
 namespace Foundatio.Repositories.Models;

src/Foundatio.Repositories/Models/Aggregations/SingleBucketAggregate.cs

@@ -1,4 +1,4 @@
-using System.Collections.Generic;
+using System.Collections.Generic;
 using System.Diagnostics;
 
 namespace Foundatio.Repositories.Models;
@@ -7,6 +7,7 @@ namespace Foundatio.Repositories.Models;
 public class SingleBucketAggregate : BucketAggregateBase
 {
     public SingleBucketAggregate() { }
+
     public SingleBucketAggregate(IReadOnlyDictionary<string, IAggregate> aggregations) : base(aggregations) { }
 
     public long Total { get; set; }

src/Foundatio.Repositories/Utility/AggregationsNewtonsoftJsonConverter.cs

@@ -1,4 +1,5 @@
-using System;
+using System;
+using System.Collections.Generic;
 using Foundatio.Repositories.Models;
 using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
@@ -16,52 +17,44 @@ public override object ReadJson(JsonReader reader, Type objectType, object exist
     {
         var item = JObject.Load(reader);
         var typeToken = item.SelectToken("Data.@type") ?? item.SelectToken("data.@type");
+        string type = typeToken?.Value<string>();
 
-        IAggregate value = null;
-        if (typeToken != null)
+        IAggregate value = type switch
         {
-            string type = typeToken.Value<string>();
-            switch (type)
-            {
-                case "bucket":
-                    value = new BucketAggregate();
-                    break;
-                case "exstats":
-                    value = new ExtendedStatsAggregate();
-                    break;
-                case "ovalue":
-                    value = new ObjectValueAggregate();
-                    break;
-                case "percentiles":
-                    value = new PercentilesAggregate();
-                    break;
-                case "sbucket":
-                    value = new SingleBucketAggregate();
-                    break;
-                case "stats":
-                    value = new StatsAggregate();
-                    break;
-                case "tophits":
-                    // TODO: Have to get all the docs as JToken and 
-                    //value = new TopHitsAggregate();
-                    break;
-                case "value":
-                    value = new ValueAggregate();
-                    break;
-                case "dvalue":
-                    value = new ValueAggregate<DateTime>();
-                    break;
-            }
-        }
+            "bucket" => new BucketAggregate(),
+            "exstats" => new ExtendedStatsAggregate(),
+            "ovalue" => new ObjectValueAggregate(),
+            "percentiles" => DeserializePercentiles(item, serializer),
+            "sbucket" => DeserializeSingleBucket(item, serializer),
+            "stats" => new StatsAggregate(),
+            // TopHitsAggregate cannot be round-tripped: it holds ILazyDocument references (raw ES doc bytes) that require a serializer instance to materialize.
+            "value" => new ValueAggregate(),
+            "dvalue" => new ValueAggregate<DateTime>(),
+            _ => null
+        };
 
-        if (value == null)
-            value = new ValueAggregate();
+        value ??= new ValueAggregate();
 
         serializer.Populate(item.CreateReader(), value);
 
         return value;
     }
 
+    private static PercentilesAggregate DeserializePercentiles(JObject item, JsonSerializer serializer)
+    {
+        if ((item.SelectToken("Items") ?? item.SelectToken("items")) is { } itemsToken)
+            return new PercentilesAggregate(itemsToken.ToObject<IReadOnlyList<PercentileItem>>(serializer));
+
+        return new PercentilesAggregate();
+    }
+
+    private static SingleBucketAggregate DeserializeSingleBucket(JObject item, JsonSerializer serializer)
+    {
+        var aggregationsToken = item.SelectToken("Aggregations") ?? item.SelectToken("aggregations");
+        var aggregations = aggregationsToken?.ToObject<IReadOnlyDictionary<string, IAggregate>>(serializer);
+        return new SingleBucketAggregate(aggregations);
+    }
+
     public override bool CanWrite => false;
 
     public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)