Pointer Compression and Isolate Groups

[jasnell]

For a while now we've been trying to work towards the ability to enable v8 pointer compression (https://v8.dev/blog/pointer-compression) by default in Node.js (https://github.com/search?q=repo%3Anodejs%2Fnode+%22Pointer+compression%22&type=issues). The key limitation that we have had up to this point is that enabling pointer compression has historically forced a process-wide maximum v8 heap limit of 4 GB, which obviously would be a significant breaking change in the runtime, despite the fact that enabling pointer compression has been benchmarked to provide a 40% performance improvement overall.

The key reason for the 4 GB limitation is that enabling pointer compression introduces a concept of a "pointer cage" within which the compressed pointers are held. This pointer cage is limited in size at compile time (I believe but could be misremembering but it might be tunable at compile time but it still ends up placing a fairly strict limit on heap size that would be a breaking change). But this size limit is inherent in the concept of pointer compression and cannot be avoided in order for pointer compression to even function.

For a while now, Cloudflare Workers (which also uses v8) has been running in a generally unsupported configuration of v8 that enables pointer compression but disables the shared pointer cage. As v8 development has progressed, the need to solve limitations of the shared pointer cage have become critical to prevent v8 from having to continue maintaining this unsupported configuration mode that workers has relied on.

So earlier this year Cloudflare partnered with Igalia to implement a new feature in v8 that will be fully landing soon called an "Isolate Group". Most of the work has already landed in v8.

An Isolate Group represents a whole pointer cage with the 4 GB limit. The key difference is that we can now create multiple Isolate Groups within a single process, essentially allowing us to create multiple groupings of isolates with each group having a maximum of 4 GB but removing the process-wide maximum heap restriction.

Any number of Isolates can be created within an Isolate Group, all of which would sit within the same pointer cage.

So whereas today in Node.js, with pointer compression enabled, the process would look something like...

  +----------------------------------------------------------------------------+
  |                           Pointer Cage   (4GB)                             |
  |   +-----------------+     +-----------------+    +-----------------+       |
  |   |     Isolate     |     |    Isolate      |    |     Isolate     |       |
  |   |  (main thread)  |     |   (worker 1)    |    |    (worker 2)   |       |
  |   +-----------------+     +-----------------+    +-----------------+       |
  +----------------------------------------------------------------------------+

With Isolate Groups it could look more along the lines of ...

  +------------------------+------------------------+--------------------------+
  |   Pointer Cage   (4GB) |   Pointer Cage   (4GB) |  Pointer Cage   (4GB)    |
  |   +-----------------+  |   +-----------------+  |  +-----------------+     |
  |   |     Isolate     |  |   |    Isolate      |  |  |     Isolate     |     |
  |   |  (main thread)  |  |   |   (worker 1)    |  |  |    (worker 2)   |     |
  |   +-----------------+  |   +-----------------+  |  +-----------------+     |
  +------------------------+------------------------+--------------------------+

In other words, rather than creating all isolates (main thread + worker thread isolates) in a single shared process-wide pointer cage, each isolate can be created in a separate pointer cage ("isolate group"), of which we can now have any number, meaning the entire process is no longer limited to just a single 4 GB heap.

Obviously ,this still becomes a breaking change because individual isolates will have the imposed 4 GB limit but with pointer compression this is far less of a limitation than it actually may first appear. The vast majority of Node.js applications that exceed 4 GB heaps without pointer compression will run just fine in a 4 GB heap of compressed pointers.

When Node.js finally updates to a version of v8 that has the Isolate Groups work included, I plan to open a PR that adds an experimental compile-time flag to enable automatic use of Isolate Groups. The code change for this is simple:

With this new compile flag enabled, rather than:

auto isolate = v8::Isolate::New(...);

We would have...

auto group = v8::IsolateGroup::New();
auto isolate = v8::Isolate::New(group, ...);

And compile the Node.js process with pointer compression enabled.

My hope is that we would soon thereafter be able to make this the default build mode for Node.js in a later phase 2.

In that later phase, if no issues are encountered, we would flip the default so that the main release builds would compile with pointer compression and isolate groups enabled, with a compile flag option to disable these.

This ought to result in a significant memory and performance improvement across the board in Node.js and ALSO provides the benefit of allowing every Node.js worker thread to run in its own v8 sandbox (https://v8.dev/blog/sandbox) also boosting overall security.

Cloudflare and Igalia soon plan to publish a blog post covering all of this in far more detail along with some additional work we are collaborating on. There are a number of additional technical details that should be covered in extensive detail in that blog post. That blog post would likely serve as the official public announcement of this work but for now, I wanted to open this issue as a way of giving folks here a heads up and to serve as a tracking issue for the work as it progresses. It was important to us that whatever work was done here did not just benefit Cloudflare but could also be used by any runtime built on v8 including Node.js, Deno, and even browsers.

The actual changes to Node.js are expected to be minimal and the vast majority of applications will see no difference other than improved performance and memory usage. Applications that do require massive heap sizes would likely see an impact so we will need to work out some solutions for those.

I will be sharing more details as things progress so watch this thread. And if you have any questions, ask away and I'll answer whatever I can!

/cc @mcollina @joyeecheung @targos @anonrig @nodejs/v8 @nodejs/workers @nodejs/tsc

[ronag]

Super cool. Would this isolate group possibly enable more efficient ways of transferring/sharing resources between isolates in the future?

On a separate note. It would be nice if we had a somewhat official pointer compression build to let people try it out, with the very restrictive memory limit. We used to have a build hidden somewhere but it is no longer built with more recent versions of Node.

[devsnek]

I've been following the isolate group work and I'm very excited about it, but I've been unable to get clear answers about whether adopting this model will lock a runtime out of using the upcoming shared structs proposal. I don't know how much cloudflare workers cares about this, but I think it will need to be supported in node and deno.

[joyeecheung]

We used to have a build hidden somewhere but it is no longer built with more recent versions of Node.

I think that is because the CI broke e.g. https://ci.nodejs.org/job/node-test-commit-linux-pointer-compression/602/nodes=rhel8-x64/console

[richardlau]

On a separate note. It would be nice if we had a somewhat official pointer compression build to let people try it out, with the very restrictive memory limit. We used to have a build hidden somewhere but it is no longer built with more recent versions of Node.

It's part of https://unofficial-builds.nodejs.org/. The builds broke from Node.js 22 and were disabled. Feel free to submit pull requests to the recipe to get them back.

nodejs/unofficial-builds#158 tried to reenable for Node.js 22 but the logs for the 22.11.0 pointer compressed build show that it's still broken.

[jasnell]

@ronag:

Would this isolate group possibly enable more efficient ways of transferring/sharing resources between isolates in the future?

Not really, and in fact if we end up enabling v8 sandbox where each isolate group is a separate sandbox this will get even trickier, but there are options. But yeah, isolate groups won't impact this.

@devsnek:

... but I've been unable to get clear answers about whether adopting this model will lock a runtime out of using the upcoming shared structs proposal.

Isolates running within the same isolate group ought to be able to use shared structs. If we have a model where every worker thread is running a separate isolate group, then there will likely be restrictions in that case using shared structs between them. However, if we allow multiple worker threads to be created within a single isolate group, those ought to be able to take advantage of that with the tradeoff being that those collectively will be limited to the heap 4 GB limit.

[mcollina]

If I read this correctly, we need an API to create a new isolate group, and that pass that as an option to the Worker constructor. By default, it would use whatever main uses.

[bnoordhuis]

Related: #55325 (fairly sure that one is isolate groups not being fully baked yet)

[jasnell]

If I read this correctly, we need an API to create a new isolate group, and that pass that as an option to the Worker constructor. By default, it would use whatever main uses.

For worker threads, what I imagine is, by default all worker threads would run in their own isolate groups, with an option to allow spawning a new worker thread in the same isolate group as the current.

So... for example,

// main thread is in its own isolate group...

// worker is created in a separate isolate group...
const worker = new Worker('....');

// main thread is in its own isolate group...

// worker is created within this threads isolate group...
const worker = new Worker('...', { group: 'parent' });

But that's just what I'm thinking right now. We'll need to evaluate the options.

[jasnell]

@bnoordhuis :

fairly sure that one is isolate groups not being fully baked yet ...

Yep. There are still a couple of todos remaining. The Igalia folks are on it but please let me know if any more pop up. I don't intend to rush this in. Want to make sure things are solid.

[jasnell]

Another possible approach for the JS API here would be to expose Isolate Groups as an object under the worker_threads module such as...

import { WorkerGroup } from 'node:worker_threads';

const group = new WorkerGroup();  // Wraps an IsolateGroup
const worker = group.newWorker('...', { ... });  // Creates a new worker thread within the group

The advantage of this API is that it leaves the existing Worker constructor options alone and it becomes explicitly discoverable by checking for the existence of WorkerGroup as opposed to knowing whether or not a group option is supported (as it would be silently ignored in older Node.js versions).

We've got a number of weeks before everything lands in v8 so there's lots of time to bikeshed alternatives here.

[mcollina]

@jasnell that's more in line to what I was thinking.

[joyeecheung]

Obviously ,this still becomes a breaking change because individual isolates will have the imposed 4 GB limit but with pointer compression this is far less of a limitation than it actually may first appear. The vast majority of Node.js applications that exceed 4 GB heaps without pointer compression will run just fine in a 4 GB heap of compressed pointers.

I wonder if this is something we should put into the next survey or just poll it a bit, because it seems a surprising amount of applications do need more than 4GB. Previously I thought it was mostly needed for giant servers (I have heard from some enterprise users about this), but then I learned running tsc type checking on large code bases like Babel can already take about 12GB, and there are probably a lot of large TypeScript monorepos out there? I have also extended it to ~10GB when running c8 on big enough coverage reports. Arguably the tools can make use of workers more to spread out the loads to multiple cages, but that will take time and in the mean time for end users, updating CLI dev dependencies to do it can be a lot harder than just updating your own application code to use workers.

Not to say that this is a blocker for releasing pointer compression as the benefit is too tempting to ignore for apps that don't need 4GB, but if the survey/poll shows that it can regress too many people that do need >4GB then I think we need to prioritize it a bit to have a good story about how to make the transition less painful. That might need better support from the version managers (e.g. imagine instead of using 26 in the setup-node github action, it's possible to use 26/unlimited, which I believe just needs an extra release channel + support in nvm to work)

[gengjiawen]

Since memory became cheaper as the AI hype. I still think Node.js without memory limit would be a perfect default. Provide an official slim version (point compressions on) is a good idea from my point of view.

[laurisvan]

@jasnell Do you have a schedule for the change? I wonder that if Node TSC ends up disabling this feature by default, perhaps I might be able to help getting the unofficial builds back up & running once the internals work again.

For our company, pointer compression brought so many benefits that we rather keep on running legacy versions of Node until there is a working alternative. Memory is cheap only as long as you don't buy it in form of virtual machines :)