[API Proposal]: HybridCache: span-based fetch

[mgravell]

Background and motivation

This is follow-on work related to DefaultInterpolatedStringHandler and MemoryCache; the core idea here is that HybridCache should be able to satisfy local "hit" requests synchronously without the code of a string key parameter, instead leaving the key unmaterialized. This is achieved by taking the existing 2 HybridCache.GetOrCreateAsync methods that take a string key parameter, and add an overload that has ref DefaultInterpolatedStringHandler key instead. Since Roslyn prefers this signature, existing HybridCache code of the form:

var value = hybridCache.GetOrCreateAsync($"/some/{key}/path/{id}", ...);

will automatically update (at next recompile) to the new usage, with zero code changes required. We will also ensure documentation is clear about preferring this API.

Since this is a public abstraction with 1st and 3rd party implementations, and DefaultInterpolatedStringHandler is a dangerous type, we do not blindly let implementations deal with this; the two new methods are non-virtual, with a new virtual method that takes the key as ReadOnlySpan<char> key. That way, implementations are not exposed to the fiddly DefaultInterpolatedStringHandler API and we do not risk leaks / double-pool-restore.

API compatibility:

This should probably target net10+ only, since it requires related features. If the MemoryCache API gets backported to net9 (unlikely), in theory we could use the DefaultInterpolatedStringHandler via unsafe-accessor, but: it seems preferable to consider this as net10+.

API Proposal

namespace Microsoft.Extensions.Caching.Hybrid;

public abstract partial class HybridCache
{
+    protected virtual ValueTask<T> GetOrCreateAsync<TState, T>(ReadOnlySpan<char> key, TState state, Func<TState, CancellationToken, ValueTask<T>> factory,
+        HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default)
+        => GetOrCreateAsync(key.ToString(), state, factory, options, tags, cancellationToken);

+   public ValueTask<T> GetOrCreateAsync<T>(ref DefaultInterpolatedStringHandler key, Func<CancellationToken, ValueTask<T>> factory,
+     HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default)
+ {
+     try
+     {
+         return GetOrCreateAsync(key.Text, factory, WrappedCallbackCache<T>.Instance, options, tags, cancellationToken);
+     }
+     finally
+     {
+         key.Clear();
+     }
+ }

+    public ValueTask<T> GetOrCreateAsync<TState, T>(ref DefaultInterpolatedStringHandler key, TState state, Func<TState, CancellationToken, ValueTask<T>> factory,
+        HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default)
+    {
+        try
+        {
+            return GetOrCreateAsync(key.Text, state, factory, options, tags, cancellationToken);
+        }
+        finally
+        {
+            key.Clear();
+        }
+    }
}

Notes:

• implementations shown here to illustrate that we're not breaking abstract rules
• the finally without await does not in this instance represent premature cleanup; the span cannot traverse the async/await boundary, so we know it is safe to call Clear() in all exit scenarios

The bottom two methods are the public API surface, using try/finally to ensure correct handling of the key. The first two method is the protected virtual method that specific implementations may choose to override to exploit allocation-free L1 lookups. If they do not choose to do so, we simply allocate the string as we would have originally. This ensures the default behaviour is valid on existing implementations.

A note is added to clarify the slightly unusual usage of a span on a *Async method:

<remarks>If the specific implementation cannot get the value synchronously, that implementation must copy the <paramref name="key"/> contents (using a leased buffer, string, or similar) before the asynchronous transition.</remarks>

This atypical usage is because of the high probability of local synchronous "hit" results, hence ValueTask<T> as the return value.

API Usage

No change at the caller:

var value = hybridCache.GetOrCreateAsync($"/some/{key}/path/{id}", ...);

[mgravell]

/cc @jodydonetti

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-extensions-caching
See info in area-owners.md if you want to be subscribed.

[julealgon]

@mgravell is the idea here that whenever there is a cache hit, it doesn't end up allocating the string for the key when using interpolation handlers? So, it only allocates if it is generating the actual entry (first time or after expiration, etc)?

If that's the case, wouldn't it still be a better idea to create a dedicated struct (say, a record struct, which gets equality implemented by default) to store the key "variables" instead of having them be part of a string which uses a lot more memory?

Using your example:

var value = hybridCache.GetOrCreateAsync(new MyCustomKeyStruct(key, id), ...);

Or is there a major downside to doing something like that?

Sorry if this is a dumb question, I'm just not as familiar with interpolation handlers yet and was curious.

[mgravell]

@julealgon the idea of HybridCache here is to radically simplify dealing with IMemoyCache and IDistributedCache; in both cases we need the ability to talk their native language - IMemoryCache talks object, and IDistributedCache talks string. Fortunately, most IMemoryCache is actually MemoryCache, which we can type-test against to access the alt-lookup API (new in net10 for MemoryCache - it specializes string keys internally already, for the string rehashing dictionary feature, and also for Marvin on down-level). This means with this one simple change, we get zero change to the perceived calling code, and end-to-end alloc-free L1 hits.

Now contrast MyCustomKeyStruct: this would need to be per consumer, since different call paths have different key parameters, they'd need to implement all the washing logic (or we rely on them using record struct, not a reasonable assumption), they have to pay boxing for IMemoryCache, we need a formatter for IDistributedCache, etc. We can't have a per-key TKey without RCache or similar, and that would also force either a HybridCache<TKey> or internal per-call type lookup.

Basically: the string-like API is a very practical and efficient way of avoiding a wide raft of problems; this last change allows us to avoid the actual string in the intended most common code path.

instead of having them be part of a string which uses a lot more memory?

Don't forget:

• in the miss case, we need the string anyway, for IDistributedCache, and an object for IMemoryCache which might as well be the string key that we'd need in some other cases
• in the hit case: we don't generate a string, so "a lot more" becomes zero

There is also the "no IDistributedCache" scenario; we'd still need to box the struct, which isn't free, but is probably cheaper than the string, I'll grant - but we'd also lose the dictionary rehashing feature; this is specific to TKey===string.

Also, to underline the point: HybridCache is already locked and loaded - we can't change it radically without making it something else. This is an optimization of an existing API, not a complete overhaul. HybridCache currently takes string: we want to make that cheaper.

[julealgon]

@mgravell

IMemoryCache talks object, and IDistributedCache talks string.

As someone who is currently heavily using only IMemoryCache in our codebase (we have plans for a distributed cache but its not yet in effect), I completely forgot IDistributedCache was string due to the serialization aspect.

Now contrast MyCustomKeyStruct: this would need to be per consumer, since different call paths have different key parameters, they'd need to implement all the washing logic (or we rely on them using record struct, not a reasonable assumption),

Yeah, we've been creating dedicated key record struct's for each scenario. I guess the benefits only exist now when we are still using IMemoryCache in NET472 land, but like you alluded to most of that would go away with this allocation-free path.

... they have to pay boxing for IMemoryCache

I feel dumb for not realizing this. Of course... because it's a value type and the key is not generic. 🤦‍♂️

Basically: the string-like API is a very practical and efficient way of avoiding a wide raft of problems; this last change allows us to avoid the actual string in the intended most common code path.

Noted. Thanks!

There is also the "no IDistributedCache" scenario; we'd still need to box the struct, which isn't free, but is probably cheaper than the string, I'll grant - but we'd also lose the dictionary rehashing feature; this is specific to TKey===string.

Interesting. So, at the end of the day, string-based keys will probably be the recommended way to handle cache keys of all types I assume.

I believe in our current scenario (pure IMemoryCache + NET472) it is probably quite wasteful, but with these improvements in NET9/10 most of the downsides would vanish.

Also, to underline the point: HybridCache is already locked and loaded - we can't change it radically without making it something else. This is an optimization of an existing API, not a complete overhaul. HybridCache currently takes string: we want to make that cheaper.

👍

[jodydonetti]

As someone who is currently heavily using only IMemoryCache in our codebase (we have plans for a distributed cache but its not yet in effect), I completely forgot IDistributedCache was string due to the serialization aspect.

Not really, it's string because it's the type of the key in IDistributedCache, see here.

[jodydonetti]

Hi @mgravell

Background and motivation

the core idea here is that HybridCache should be able to satisfy local "hit" requests synchronously without the code of a string key parameter, instead leaving the key unmaterialized.

Dumb question (I'm still thinking about it), but in the case of L1+L2 this will allocate a string every time L2 is needed, correct?

Not that it is worse than now, mind you, just pointing it out for me to get the full picture.