Caching layer

[uNetworkingAB]

Another solution to excessive JS calls could be to simply add a caching layer that stores whatever the JS callback wrote, for a duration of X seconds, seding it back from cache until next update.

This could be entirely separate, as a wrapper of SSLApp / App like CachingApp, CachingSSLApp. Like a proxy but built-in.

It could be good for benchmarks while not having to change existing code. It also makes it possible to build real world examples like "what's the price of gold" being updated every 30 seconds, hitting the cache all other times.

Alternatively the API could be that you just add an integer after the handler, for how many seconds the cache is valid, defaulting to 0. Then you could mix cached endpoints with non-cached ones seamlessly.

[uasan]

This is interesting.
What will be the cache key, will all variables from the router participate in the cache key?

[uNetworkingAB]

The entire URL would be key

[uasan]

Clearly, this is ideal for static endpoints that have immutable response.

But, if you add to the interface it is possible to pass a custom asynchronous cache revalidation function, then this will already be a cool smart cache layer, the revalidation function should receive all the request parameters and return true or false, if it returns true then the client can give the old response if false, then you need to call the router handler which will return a new response.

Thank you, this is a good idea

[uNetworkingAB]

All public HTTP web APIs like those you access when browsing the stock market would be ideal candidates for having at least 30 second cache. It's not just static endpoints, it's literally all endpoints that deal with public data. Every single financial app that lets you browse some market would be ideal use.

custom asynchronous cache revalidation function

This kills the point, though. Calling into JS to decide if you don't need to call into JS will not fix anything. The idea is to entirely avoid calling any JS in the hot path, only doing so at sparse intervals to update the cache.

[uNetworkingAB]

Arguments against the cache would be that:

• This is too app specific and should be implemented by the user as needed, not built in
• If needed, caching is already done in the proxy and most people use Cloudflare anyways
• It's bloat, and too limited anyways

The arguments for it would be:

• JS is too slow for implementing a cache, as calling into it kills performance defeating the entire point (for smallish simple responses)
• Since the library is about supporting stand-alone best-case performance, having a built-in solution that can boost benchmarking scores by a lot (for JS use, not C++) without changing any code, and which has a valid, motivatable use for mentioned cases, it can be worth having in the library (again, everything performance related should be in the lib)
• The PR itself, from scoring higher in benchmarks is valuable and even just 1 second cache will still boost performance by a lot in those cases
• It's pretty simple to implement and can be entirely separate
• It would entirely kill the need for uWS.DeclarativeResponse, as the cache is better in every way - simplifying the project

[uasan]

Yes, you are probably right, but in our case you cannot give away the cache if it is no longer valid, the cache lifespan is not constant and depends on external factors, so in this case a revalidation function is needed.

This kills the point, though. Calling into JS to decide if you don't need to call into JS will not fix anything.

I understand, but I see greater potential for optimizing the validation function, inside this function there is no need to include the onAbort handler, there is no need to call cork(), i.e. minus 2 calls from C++ to JS.

It’s also very cool if you, on your part, make requests cork, i.e. If you called a function revalidation once and its promise has not yet been resolved, but exactly the same requests came from clients (match url), then you do not call the function revalidated, but continue to wait for the resolve of the promise from the first call of this function, this greatly optimizes the count.

Such optimization cannot be done with endpoint handlers, but it can be done with revalidation function, that's why I suggested it, it makes sense if you, on your part, optimize its call (without onAbort() it, without cork(), without repeated calls if promised pending)

[uNetworkingAB]

The cache absolutely cannot call any JS whatsoever. That kills the entire purpose. What you can do is:

1. Specify a timeout in seconds per-URL
2. Prematurely invalidate the cache per URL
3. Potentially specify what headers (such as "Authorization") should be part of the cache key

The cache must operate entirely separate from any JS, so that optimal performance can be delivered.

[uNetworkingAB]

Reference for perf. difference in question:

[uasan]

I don’t argue, without calling the JS this is an ideal case, it gives +50%, but if the revalidation function gives +20% compared to calling the endpoint handler, it also makes sense.

[miscellaneo]

So, the API would be something like this?

require('uWebSockets.js')
  .App()
  .get(
    '/some-route',
    (res, req) => {
      // Implementation here.
    },
    {
      // Cache on the "Authorization" header in addition to the URL.
      cacheHeaders: ['Authorization'],
      // Cache for thirty seconds.
      cacheTtl: 1000 * 30,
    },
  );

Would there be other options to control cache size, by number of entries or by memory used?

[uNetworkingAB]

Something like that yes

[uasan]

I propose to auto-assign caching HTTP headers, if the cacheTtl option is specified, send header
Cache-Control: max-age=${cacheTtl / 1000}

[uasan]

It is also possible to automatically generate a response hash and transmit it in the Etag header, then you can automatically send clients a 304 status if the client cache has the same hash, this is better than always send clients a 200 status and a response that they already have in the client cache

[uasan]

I came up with a better option: when creating caching routes, you need to return the controller object of this router so that you can change the contents of the response body and possibly the response headers.

Then we will be able to update the cache without calling JS functions on every request, this is an excellent performance and makes it possible to always send actual cache

const route = CachingApp().get('/some', (res, req) => body, { ...cacheOptions });

// Later in the application life cycle
// when an event occurs that requires the cache to be updated
// we simply update the cache of the desired route

route.setHeader(newHeder);
route.setBody(newBody);

[e3dio]

app.get() returns app for chaining more routes app.get().get() so your current proposal would not work, can't make it return route