feat: remote functions cache API

[dummdidumm]

Adds a new remote functions cache API. Simple example:

/// file: src/routes/data.remote.js
import { query, command } from '$app/server';

export const getFastData = query(async () => {
	query.cache('100s');
	return { data: '...' };
});

export const updateData = command(async () => {
	// invalidates getFastData;
	// the next time someone requests it, it will be called again
	getFastData().invalidate();
});

For more info see the updated docs.

This is very WIP, and the adapter part isn't implemented yet (there are a few ways to approach it and we need to agree on the other APIs first). But it workds in dev and preview (public cache is implemented as runtime cache which very likely needs some more hardening).

TODOs/open questions:

• right now query.cache() is "last one wins" except for tags which are merged. After sitting with it for a while this is I think the most straightforward and understandeable solution, but we can also approach it differently.
  • there could be one entry for public and one for private, i.e. you can have both. Drawback is that you might accidentally cache something publicly that you want to keep private
  • other way around: as soon as something is declared private it cannot become public anymore. Drawback is that you might call a private remote function from a public remote function, being fully aware of that and you only use secure parts of the privately-cached remote function so you are limited by the framework's decision now
  • error if you have both private and public. My least favorite option because we should recover gracefully; if that's your favorite you'd rather go "use private and ignore public; maybe have a warning at dev time"
  • same options for ttl and stale
• is ttl and stale descriptive enough? Should it be maxAge and swr instead (closer to the web cache nomenclature)?
• how to best integrate this with adapters? either they provide a file with some exports which are like hooks which we call at specific points (setHeaders, invalidate etc) or we don't do anything and do this purely via headers, and adapters can check these headers and either do runtime cache based on it and/or add cdn cache headers (though maybe they have to clone the response then; not sure how much of an overhead that is and if that matters)
• this only works for remote functions right now, and it only works when you are calling them from the client. We could additionally have a SvelteKit-native runtime cache for public caching, and/or the adapter can hook into this to cache somewhere else than in memory (Vercel can use runtime cache, CF can use their cache, etc; i.e. this is related to the question above). This way we get more cache hits between client/server calls (or rather, we can get full page request cache this way, which we don't have at all right now).
• can this be enhanced in a way that this is usable for full page requests, too (e.g. inside handle hook?). Private cache doesn't make sense there at least. I'd say it's possible to implement and would be intuitive with this API (we can say "do this in handle or load", or "assuming you use remote functions only we take the lowest cache across all of them as the page cache", etc etc, many possibilities) but we should do that later and not bother with it now.

[changeset-bot]

🦋 Changeset detected

Latest commit: 85b8ae4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@sveltejs/kit	Minor
@sveltejs/adapter-node	Patch
@sveltejs/adapter-vercel	Patch
@sveltejs/adapter-netlify	Patch
@sveltejs/adapter-cloudflare	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[svelte-docs-bot]

Preview: https://svelte-dev-git-preview-kit-15678-svelte.vercel.app/

[elliott-with-the-longest-name-on-github]

I don't think the URL is the right cache key here... it should probably be the remote function key instead. Also, tags should be additive, not replace the default key.

[elliott-with-the-longest-name-on-github]

One thing I don't think is quite right here: Caching remote functions requires two coordinated layers of caching. One of them is a runtime cache at the server level, which needs to be used to cache direct-from-server remote function calls. This cache should be granular -- i.e. if I call getUser and getTeam and they have separate cache durations, that's fine -- they can be cached separately. The other is the "request-level" cache, which basically needs to take the lowest common denominator TTL and apply it to the request.

Overall, I really think SvelteKit core should not do anything with the information from the cache API -- instead, the adapters should do everything:

• Adapters should provide a get(key: string): Promise<string | undefined> function, which SvelteKit calls for every query
• Adapters should provide invalidate(key: string): Promise<void> and invalidateTag(tag: string): Promise<void> functions, which SvelteKit delegates to for
• Adapters should receive a Map<string, CacheInvocation> object that they can use to do... whatever they want. For example, the Vercel adapter would likely map over this and cache everything in the runtime cache, and, if the request is a remote request for a query endpoint, it would consolidate the runtime cache TTLs to find the soonest-to-be-invalidated entry and set the overall request cache time to that

[ottomated]

Should be a type like

`${number}d`  | `${number}h` | `${number}m` | `${number}s`  | `${number}ms`

[dummdidumm]

Ms/day don't really make sense imo but otherwise yes; on my list of todos once we agree on the API 👍

[ottomated]

Would cache('immutable') be an accepted option? i.e. no ttl, only gets invalidated manually

[dummdidumm]

Caching remote functions requires two coordinated layers of caching

Overall, I really think SvelteKit core should not do anything with the information from the cache API

Both these points are correct, and in fact right now this PR doesn't do anything in core with the public cache, this is all handled outside of the SvelteKit runtime (dev/preview for now, adapters are todo). We can discuss how exactly to do this once we agree on the user-facing API👍

[Rich-Harris]

Disorganised thoughts incoming:

• Should it be on getRequestEvent() or should it just be an import? What are the advantages/disadvantages? (One advantage of the import: there's less cost to growing the surface area, e.g. if we later add a refreshAfter('60s') API or whatever. Though a corresponding disadvantage is that imports are less discoverable than event.*)
• What does this API actually do? (I know I could read the code but you told us not to 😜) That influences API design choices like nomenclature. I assume that for private it's not setting actual HTTP cache control headers because then I don't think it would be possible to programmatically invalidate them?
• Assuming then that this populates an origin-scoped Cache object in the browser, presumably we can populate it with hydrated data?
• I would go with maxAge and staleWhileRevalidate — clearer than ttl and stale or swr. I think clarity > brevity in this case
• I don't think cache should accept string | CacheOptions — maxAge isn't an option, so it would be better to do cache(maxAge, options?)
• having both refresh() and invalidate() feels potentially confusing. I wonder if something like refresh({ force: true }) would be better
• 'last one wins' — I really don't think this is the right choice. I think that just like you can't do headers.set('cache-control', ...) multiple times today, you shouldn't be able to call cache(...) multiple times for a single remote function. The framework can't possibly divine the developer's intent; it should force them to be explicit about it
• Do we really need tags? It's something people ask for because it's what they're used to, but I think it's a bad solution to the problem. We've resisted the urge to support tag-based refreshes for good reasons (relating to type safety, bundling and so on) and I think the same thinking applies here. While we can't have an equivalent of requested, we could support cache.invalidate(myQuery) alongside myQuery(123).invalidate() and I bet that would cover all realistic use cases without sacrificing the aforementioned benefits. If I'm wrong, we can always add tags in future but my vote would be to leave them out until then
• Public and private caches are such different beasts that I'm not totally sure they should even share an API. For example what does it mean for me to do somePubliclyCachedQuery().invalidate()? I can't invalidate other people's browser storage. The public cache implies adapter-specific behaviour that could involve KV stores and semi-proprietary headers and credit cards (you often have to pay for this stuff) while the private cache is just a Cache object with the same behaviour everywhere. I feel like mixing them up in a single function is going to lead to a lot of confusion. (Aside: I think 'server' vs 'client' caching might be a better way of thinking about this, rather than 'public' vs 'private', which a) is very HTTP-specific and b) doesn't really communicate that you're dealing with entirely distinct mechanisms)
• 'can this be enhanced in a way that this is usable for full page requests' — I don't think it makes sense for the client cache, because what would it mean? That would only make sense in the context of a service worker. So the question is whether exposing the adapter-provided server caching mechanism provides enough value over current approaches (i.e. cache-control, ISR, etc) to justify the cost of making the API less focused

[Rich-Harris]

A third option, between an import and a property of RequestEvent — methods on query:

import { query, command } from '$app/server';

export const getFastData = query(async () => {
-	const { cache } = getRequestEvent();
-	cache('100s');
+	query.cache('100s');
	return { data: '...' };
});

This solves all the problems at once — we don't need to worry about cluttering RequestEvent, query.* is discoverable as we add new stuff, and it neatly explains why you can only call the method inside a query (whereas anything on RequestEvent is a weird forbidden appendage in any other context)

[Rich-Harris]

While we're thinking about a cache API, I'd love for us to think about bfcache — this is one of the very few genuine weaknesses of SPA navigation relative to The Olde Web.

Even then, traditional bfcache is a bit ropey. If I click into one of my pending tasks, mark it as complete, and then navigate back to my task list, the completed task is still visible in the pending list. You see this bug a lot with GitHub. But the back navigation sure is fast!

We can offer the best of both worlds: we can keep the list of pending tasks cached, but also invalidate it if one gets completed. That way, when I navigate back without completing a task it can happen instantly, but if I did update the list then SvelteKit knows that it has to fetch fresh data.

This might be controversial but I think this behaviour should be opt-out rather than in. Something like this, perhaps:

export const getPendingTasks = query(async () => {
  // disable bfcache altogether for this query
  query.bfcache(false);

  return await db.select().from(task).where(...);
});

export const getPendingTasks = query(async () => {
  // enable bfcache, but configure it
  query.bfcache({
    limit: 5, // evict if we're more than 5 navigations away from having used this query
    maxAge: '10m' // evict after 10 minutes whatever happens
  });

  return await db.select().from(task).where(...);
});

Maybe bfcache is the wrong name since it's not what's actually happening.

Implementation-wise, we would continue to remove queries from memory when they were offscreen, but instead of discarding their data altogether we would put them in a session-scoped Cache. If a query was refreshed in a command/form handler, and wasn't visible onscreen but was in the session-scoped cache, it would be evicted. requested would include queries that lived in the cache, in addition to ones currently onscreen.

As with the prerender Cache, we would delete any orphaned caches on startup.

This would only apply during popstate navigation — everything else would result in fresh data being fetched (modulo whatever caching API we land on here). I think this would all feel pretty nice.

[dummdidumm]

I like the query.cache(...) idea. But where does invalidate(...) live then? command/form.invalidate(...)? Or is it on query too because you can only invalidate queries? Probably that ...

Public and private caches are such different beasts that I'm not totally sure they should even share an API

I would argue they are very similar with respects to how you use them. The "only" difference is that one cache lives in the user's browser and the other lives in your CDN or server runtime. The web's caching headers also share all the same stuff and you can then declare whether this should be saved in the user's browser or the CDN. The way I think about this feature is "you get caching you know from the web's caching headers but with the possibility to invalidate them before they run out".

For example what does it mean for me to do somePubliclyCachedQuery().invalidate()? I can't invalidate other people's browser storage.

It's not in other people's browser storage, it's in your runtime storage or the CDN, and that you can easily invalidate with that.

having both refresh() and invalidate() feels potentially confusing. I wonder if something like refresh({ force: true }) would be better

Just from reading that I wouldn't know what refresh({ force: true }) means. But they're also just different things. One is sending the query along with a response, the other is telling the browser or the CDN "hey the next time someone asks about this thing do a request to me".

Do we really need tags?

They are a very good way to organize your API surface into specific categories. But yes, we could start without them, if we make it possible to invalidate all permutations of a query (e.g. query.invalidate(getTodo))

you shouldn't be able to call cache(...) multiple times for a single remote function

This is different to load/API functions IMO since those are not as easily composable (you can fetch(...) your own API endpoints but it's more indirect). Remote functions are very composable and people should be able to leverage that. My idea was therefore to have last-one-wins or "shortest time / narrowest scope". Another possibility would be to have cache only work on the outer-most query that is called, i.e. for the inner ones it's ignored (?). But all these "I error / I ignore" solutions feel a bit less "we do the right thing and have a graceful fallback" than I'd like to.

it would be better to do cache(maxAge, options?)

To quote you: I think clarity > brevity in this case which I would apply to this, too, then, to say "always pass an object please".

While we're thinking about a cache API, I'd love for us to think about bfcache

Caching/when to refresh in general is a topic we need to discuss further. It's sensible to e.g. want to refresh the data every X seconds proactively if it's used, or reload on each navigation (back or forth), and so on (also see #15039). So I agree with much of what you laid out there.

[Rich-Harris]

It's not in other people's browser storage, it's in your runtime storage or the CDN

Ah so it's not like cache-control: public, max-age=n, which would cause a response to be cached by CDN but also in the browser's HTTP cache? You get one or the other?

That in itself might cause some confusion — if I have a response that I know is good to cache for 10 minutes, and will never be invalidated, then I probably want it cached by both CDNs and browsers. I don't want the browser to have to go back to the CDN even for a 304, I already said it was good for 10 minutes. I would guess that the majority of cases are like that (i.e. can be expressed with bog-standard cache-control), and the cases where I need to invalidate responses, using a mechanism like ISR or Cloudflare's cache purging API, are comparatively rare. (I am glossing over the additional complexity of runtime caches/KV stores.)

For this reason I still think we need clearer separation at an API level. It's probably fine for a single API to cover both public and private caches if we're just mirroring HTTP semantics, but for stuff that needs to be invalidated (and therefore needs adapter buy-in, and involves often-paid-for additional platform features) it has to be something different.

Perhaps this is where it does make sense to use tags after all, precisely because that maps to how platforms think about cache purging, and having tags be something meaningful (rather than generated from remote function keys) is the feature — I want to be able to go into my CDN dashboard and clear stuff, or POST to a webhook, and in both cases I need to be dealing with predictable keys.

Something like this perhaps:

export const getPost = query(v.string(), async (slug) => {
	query.cacheByTag(['post', `post:${slug}`]);
	return await db.select().from(post).where(...);
});

export const invalidatePost = command(v.string(), (slug) => {
	if (!(await isAdmin())) return;
	query.invalidateTag(`post:${slug}`);
});

export const invalidateAllPosts = command(v.string(), (slug) => {
	if (!(await isAdmin())) return;
	query.invalidateTag('post');
});

(Note that I didn't bother to include a maxAge here, because the assumption is that once generated the content is good to go until invalidated. Though if you did need to specify an expiration then that could be done with a second argument.)

cache and cacheByTag (bikesheddable) would be mutually exclusive.

This approach also eliminates the refresh/invalidate confusion I mentioned above — queries themselves would not have an invalidate method.

But all these "I error / I ignore" solutions feel a bit less "we do the right thing and have a graceful fallback" than I'd like to.

We absolutely cannot be in the business of guessing what the developer intended when it comes to this stuff; there are meaningful consequences to getting it wrong. Even the most conservative option (ignore public if both public and private, choose the shortest maxAge) is bad, if it means that the origin server gets hammered for a resource that the developer thought was cached immutably.

it would be better to do cache(maxAge, options?)

maxAge isn't an option, it's... maybe 'operand' is the wrong technical term but you get the idea. It's appropriate for an operand to occupy an argument slot by itself. If the choice is between cache(options: string | CacheOptions): void or cache(options: CacheOptions): void then the latter is the clear winner, but for the probably-fairly-common case where you only want to specify maxAge then doesn't this seem nicer?

query.cache('60s');
query.cache('60s', { scope: 'public' });

query.cache({ maxAge: '60s' });
query.cache({ maxAge: '60s', scope: 'public' });

[Rich-Harris]

(If we agree on keeping the APIs separate, then we can punt on cacheByTag and invalidateTag for now and just worry about the basic HTTP-mirroring API)