Observed behavior
INatsKVStore.GetKeysAsync(IEnumerable<string> filters, ...) hangs indefinitely when the filter pattern matches zero keys in the bucket, even though the bucket contains other keys.
Of note -
I was previously using a client-side filtering pattern that worked as expected - something like:
await foreach (var key in store.GetKeysAsync(cancellationToken: ct)) { if (key.StartsWith(prefix, StringComparison.Ordinal)) // process key }
I switched to using filters as I expected a performance boost from server-side key filtering and that's all I changed to see the hang:
await foreach (var key in store.GetKeysAsync([prefix + ">"], cancellationToken: ct))
Expected behavior
GetKeysAsync should return an empty enumerable promptly when no keys match the filter pattern (similar to how it handles an empty bucket).
Server and client version
- NATS Server: 2.11.8
- NATS.Net Client: 2.5.10
Host environment
Client:
- .NET 8.0 (ASP.NET Core Web API)
- Linux container (amd64)
- Running in Kubernetes
- NATS.Net 2.5.10
We have a thin internal wrapper around the NATS.Net client that handles connection management and authentication, but the GetKeysAsync call is a direct pass-through to the underlying INatsKVStore.
Server:
- NATS 2.11.8
- 3-node JetStream cluster
- Running in Kubernetes
- Linux containers (amd64)
KV Bucket Configuration:
- Replicas: 3
- History: 1
- MaxAge: 7 days
Steps to reproduce
- Create a KV bucket with some keys (e.g.,
foo.bar.1, foo.bar.2)
- Call
GetKeysAsync with a filter that matches NO keys: ["nonexistent.prefix.>"]
a. A maybe relevant note - its not that the filter matches no keys, its that I've deleted all of the keys that the filter used to match. (In case there's a meaningful difference for no vs. tombstone here)
- The call hangs indefinitely instead of returning empty
=====
I tried to write a simple app to test this locally to give good repro steps, but wasn't seeing the same behavior - I'll note my experiments for posterity.
What We Tested
┌──────────────────────────────────┬─────────────────────┐
│ Configuration │ Result │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.12.3 (latest) single node │ ✅ All tests passed │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.11.8 single node │ ✅ All tests passed │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.11.8 with 3-node cluster │ ✅ All tests passed │
└──────────────────────────────────┴─────────────────────┘
Test Scenarios
- Filter matching NO keys (nonexistent.>) - Passed in all configs
- Filter matching ONLY deleted keys/tombstones (deleted.>) - Passed in all configs
- Unfiltered GetKeysAsync (control) - Passed
- Filter matching existing keys (control) - Passed
Observed behavior
INatsKVStore.GetKeysAsync(IEnumerable<string> filters, ...)hangs indefinitely when the filter pattern matches zero keys in the bucket, even though the bucket contains other keys.Of note -
I was previously using a client-side filtering pattern that worked as expected - something like:
await foreach (var key in store.GetKeysAsync(cancellationToken: ct)) { if (key.StartsWith(prefix, StringComparison.Ordinal)) // process key }I switched to using filters as I expected a performance boost from server-side key filtering and that's all I changed to see the hang:
await foreach (var key in store.GetKeysAsync([prefix + ">"], cancellationToken: ct))Expected behavior
GetKeysAsyncshould return an empty enumerable promptly when no keys match the filter pattern (similar to how it handles an empty bucket).Server and client version
Host environment
Client:
We have a thin internal wrapper around the NATS.Net client that handles connection management and authentication, but the GetKeysAsync call is a direct pass-through to the underlying INatsKVStore.
Server:
KV Bucket Configuration:
Steps to reproduce
foo.bar.1,foo.bar.2)GetKeysAsyncwith a filter that matches NO keys:["nonexistent.prefix.>"]a. A maybe relevant note - its not that the filter matches no keys, its that I've deleted all of the keys that the filter used to match. (In case there's a meaningful difference for no vs. tombstone here)
=====
I tried to write a simple app to test this locally to give good repro steps, but wasn't seeing the same behavior - I'll note my experiments for posterity.
What We Tested
┌──────────────────────────────────┬─────────────────────┐
│ Configuration │ Result │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.12.3 (latest) single node │ ✅ All tests passed │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.11.8 single node │ ✅ All tests passed │
├──────────────────────────────────┼─────────────────────┤
│ NATS 2.11.8 with 3-node cluster │ ✅ All tests passed │
└──────────────────────────────────┴─────────────────────┘
Test Scenarios