A new API for work during unload

[domenic]

What problem are you trying to solve?

It's a known issue that many sites want to perform some work during document unloading. This usually includes writing to storage, or sending information to servers. (Previous discussion: #963.)

Some of the simpler use cases are solved by APIs like fetch(..., { keepalive: true }), or by using synchronous storage APIs like localStorage.

But the more complex cases, such as writing to async storage APIs like IndexedDB, or performing some sort of async operation before the HTTP request, require more work. (Examples of such pre-request async operations include using WebCrypto to hash/encrypt data, or using CompressionStream to compress it.)

What solutions exist today?

The current best solution the platform offers for this is service workers. By sending messages to the service worker,, the service worker can then use its ability to run in the background to perform the appropriate actions. The message can carry along any data necessary to perform those actions, e.g., the not-yet-encrypted-or-compressed payload, or the user's unsaved changes that need to be written to IndexedDB.

However, requiring service workers is pretty heavyweight for this use case. Even if there are no fetch handlers, the disk space consumption of the service worker registration database means this is hard to deploy at scale. And, it uses up a process's worth of memory---not only during unload time, when the work is being performed, but the entire time any document from that origin is open.

Additionally, the developer experience of service worker registration, installation, upgrading, client claiming, etc. is a lot to manage, when the goal is to just run some background code after unload. For example, they require a separate same-origin service worker file being hosted, instead of allowing creation from blob: URLs, which means that libraries for this functionality need to consist of two files, not just one. (And it gets worse if the library needs to be able to integrate with a site's existing service worker script!)

How would you solve it?

We (myself, @fergald, @pmeenan) think there are possible new APIs which could allow sites to perform this kind of processing, but at a lower cost. Here are the ideas we've come up with:

1. A PageHideWorklet. This would be a special type of worklet which a document would register early in its lifetime. Then, during unload, right after firing the pagehide event, it gets spun up and runs its code. The document would need to synchronously set the data that the worklet plans to consume, either continuously throughout the document's lifetime, or in the pagehide handler (or both). But the worklet could run asynchronously for some amount of time (see below).

   On the implementation level, this could be implemented either with a separate process for the worklet, which gets spun up at unload time, or it could be implemented with an in-process worklet plus some code that keeps the unloading document's process alive, even while stopping its event loop and freeing up most of its resources.

2. A minor extension to SharedWorker. SharedWorkers are already reasonable tools for this: they don't have heavyweight registrations and persistent state like service workers, and they are allowed per spec to stay alive after document unloading for some time.

   In theory, this could involve no spec changes, just implementation changes to allow shared workers to stay alive for longer. In practice, it would probably be better to include a hint at construction time that this SharedWorker is intended to perform post-unload processing, and so the browser should keep it alive for a longer time. Something like new SharedWorker(url, { usageHint: "after-pagehide-processing" }).

We're currently leaning toward (2), as it seems like a simple extension of what exists today.

With regards to privacy/security, the intent here would be to be exactly as powerful as service workers are today. Today there are implementation-defined limits on how long service workers stay alive after all of the documents from their origin are closed, and different browsers have made different choices for them. (And I believe some have discussed changing these limits over time, or in reaction to other signals.) We would allow these mechanisms to operate for those same implementation-defined amounts of time.

Anything else?

We also considered extendable `pagehide` event handlers, but we don't like them very much. Expand if you want to read more

The idea: inside a pagehide event, event.waitUntil(promise) would allow you to extend the document's lifetime and continue running JavaScript, while the document unloads. This would be up to some implementation-defined limit (per above).

In parallel, the user agent would be loading the new document. This would not block loading the new document in any way: even if the new document completely finishes before the promise from the old document settles, we could visibly swap in the new document, while the old document continues running. It is sort of like keeping the old document in bfcache, except JavaScript continues to run. Chrome already has this sort of document state for ~3 seconds for some cross-process navigations, and I believe other implementations might as well.

This is probably the most convenient option for web developers, as they can colocate all their code into the pagehide handler. But keeping unloaded documents alive in that way, even with opt in, is scary. And probably the memory consumed by the document, with all the DOM and JS objects its built up throughout its lifetime, is quite high.

[smaug----]

It is a bit unclear why a worklet would be used and not a (temporary) dedicated worker.
IDB and fetch etc are after all defined to work in workers.

Do we need SharedWorker?

[domenic]

A dedicated worker is pretty tightly tied to its owner document's lifetime. Are you thinking we could somehow loosen that, instead of using shared workers?

I thought using shared workers would be easier, both from a spec and implementation point of view, since they are already separate from any single document's lifetime. But maybe that's not necessarily the case?

[annevk]

Colleagues and I are rather wary of the longish timeout service workers have today and as I understand a big reason service workers are even kept alive is to reduce the cost of the ongoing navigation. This was discussed quite a bit as part of the fetchLater() work (which is still ongoing). I guess this is not meant to replace that?

[asutherland]

SharedWorker seems least bad for this use-case[1], especially since MessagePorts can now generate close events which provides symmetry to the SharedWorkerGlobalScope "connect" event and this is conceivably something sites could already be doing.

1: In particular, I agree that a pagehide event with a waitUntil for a document would have terrifying lifetime implications. And a PageHideWorklet would be at odds with worklets currently not using tasks.

[annevk]

Hmm, that close event still has a pretty severe unfixed bug: #10201.

[domenic]

Colleagues and I are rather wary of the longish timeout service workers have today and as I understand a big reason service workers are even kept alive is to reduce the cost of the ongoing navigation.

Understood. I tried to address this in the OP by suggesting that this new mode for shared workers would be subject to whatever implementation-defined limits a browser places on service workers today.

Basically, we should not make this new mode any worse than service workers, as otherwise web sites will need to continue to use service workers for this use case.

I guess this is not meant to replace that?

Correct. They apply to related but separate use cases. fetchLater() is an upgrade over fetch()-in-unload because it can be made more reliable by setting up the fetch ahead of time, and putting more in the browser's hands.

But, there are cases where neither fetchLater() nor fetch()-in-unload can work today:

• Non-fetch use cases, like async storage (e.g., writing to IndexedDB)
• Cases where async steps (like encryption or compression) are required before fetching.
  • In some cases you can try to do these async steps before calling fetchLater(), but there's a chance the page will be unloaded during your async steps, and then you'll lose the data.

[domenic]

Let's tentatively call this API new SharedWorker(url, { extendedLifetime: true }).

What do we do if multiple clients have mismatched values of extendedLifetime? Some possibilities:

1. If >=1 new SharedWorker() invocation includes { extendedLifetime: true }, then we treat the shared worker as having an extended lifetime. Basically, any client can extend the lifetime at any time.

2. A shared worker's extended lifetime is specified as being relative to the clients that requested that extended lifetime. If page A requests extended lifetime, and page B doesn't, and then page A disappears, and then 10 minutes later page B disappears, the worker shuts down immediately, since page B did not care about extended lifetime.

3. Only the first new SharedWorker() invocation controls the lifetime. The 2nd onward have extendedLifetime ignored. (And maybe we should log a console warning explaining that it was ignored.)

4. Only the first new SharedWorker() invocation controls the lifetime. All others have to match, and if they don't match, we throw an exception or fail the worker creation.

For our use case, any of these will work. We expect people to be using a specific shared worker for unloading purposes, and always calling with { extendedLifetime: true } in their pagehide handler.

(1), (3), and (4) are pretty easy to implement. (2) adds a bit more complexity, but is kind of theoretically nice in some ways.

(4) might be simplest to start with since it can evolve into any of the others later.

Edit: @yoshisatoyanagisawa reminded me that we error the worker if type or credentials options mismatch. https://html.spec.whatwg.org/#dom-sharedworker step 11.4. So going with (4) initially seems like an especially good idea now.

[yoshisatoyanagisawa]

Thanks @domenic for listing the possible behavior on the option mismatches. I also come up with the other corner case while I am checking the Chromium code.

extendedLifetime aims to extend its lifetime after all clients have been unloaded. If a new client gets created after all clients unloaded and SharedWorker is running due to extendedLifetime is true. What is an expected behavior for this?

1. connect to the existing SharedWorker and extend its lifetime until a newly added client unload + extendedLifetime duration.
2. create a new SharedWorker instance, and leaving the existing SharedWorker to be destructed after the duration.

Focusing on the use case mentioned in #10997 (comment), Option 2 might be enough. However, considering the case using the extendedLifetime option to avoid SharedWorker creation after the navigation, Option 1 can be preferred by web developers.

[domenic]

Good catch. I agree (1) seems nicer and less wasteful. If there are implementation/architecture reasons why it is especially hard, then we could consider (2), since we don't have strong use cases for (1) behavior. But I would default to (1) if possible.

[fergald]

@domenic 4 above (exceptions if the lifetimes don't match) means that changing from non-extended to extended or vice versa is tricky and maybe impossible for sites that people keep open constantly (e.g. gmail, calendar, facebook).