Support load balancing HTTP requests across worker threads

[mcollina]

I think we can support load balancing HTTP requests across worker threads by leveraging the same mechanisms that allows Cluster to do it. If a libuv handle can cross processes, it definitely can cross threads.

I've tried to do it outside of core in https://github.com/mcollina/tbal, but unfortunately it does not work because it requires some mangling of libuv handles that is only possible from within core.

The idea is to allow worker threads to have an libuv IPC channel similar to child processes, so that they can exchange libuv handles through it.

This should be opt-in.

[juanarbol]

Interesting. Could you please be a bit more specific with the implementation? Currently, moving sockets between workers is not possible.

[theanarkh]

Maybe we can achieve this by sharing fd 🤔(Or listen on the port in the main thread and distribute fd to child threads?).

server.js

const { Worker, isMainThread, parentPort, threadId } = require('worker_threads');
const os = require('os');
const net = require('net');
const http = require('http');

if (isMainThread) {
    const handle = net._createServerHandle('127.0.0.1', '9999')
    if (typeof handle === 'number') {
        console.error('exit', handle);
        process.exit();
    }
    const workers = [];
    for (let i = 0; i < os.cpus().length; i++) {
        new Worker(__filename).postMessage(handle.fd)
    }
} else {
    parentPort.on('message', (fd) => {
        console.log(`worker ${threadId}`)
        http.createServer((_, res) => {
            res.end(`${threadId}`)
        }).listen({fd});
    });
}

client.js

const http = require('http')
for (let i = 0; i < 10; i++) {
    http.get("http://127.0.0.1:9999", res => {
        let chunk
        res.on('data', (data) => {
            chunk = chunk ? Buffer.concat([chunk, data]) : data;
        });
        res.on('end', () => {
            console.log(chunk.toString())
        })
    })
}

[juanarbol]

The thing is. That handle you create on the main thread, is a socket (under the hood), and that socket/handle will be managed on your main event loop; you can't transfer that handle to another event loop.

[theanarkh]

All threads share the fd among process, so main thread just need to send the fd to other worker threads, then the worker threads create a socket based on fd.

[mcollina]

Sending the fd is possible, but we must tell the server on the main process that it should not respond to that fd, or doing anything with it.

Plus the tricky part is managing the "shutdown" of the server.

[flakey5]

If this is something y'all would be interested in pursuing I'd be willing to take a swing at it 👀

[mcollina]

cc @nodejs/workers

I think this would be great.

[gireeshpunathil]

IIRC, events that libuv manages have extended / unused attributes. one idea can be to use one of those attribute to attach the owner thread info. loops of all the threads will poll for the event, but only the one whose thread id matches with the thread info field will process the callback, the rest (including main thread) ignore the event.

[juanarbol]

I think I'm not considering something in the upfront.

https://github.com/libuv/libuv/blob/d2d92b74a8327daf9652a9732454937f3529bd30/src/unix/stream.c#L539

Every time uv_accept is called (at least in unix) the first operation made is check that the client's loop is associated with the server's loop.

Is this idea more like a reversed proxy with spawned workers in a IP range? And those workers will write into the reverse proxy socket? I'm very confused with this one.

Are the workers just responsable of creating the buffers and writing into the "main" process client sockets? If that is the case, I would consider this as a passable to achieve. IMHO.

[juanarbol]

it may be possible to pass like an option for "http handlers" to the workers, they will compute your request, but the streams will remain on front. Not quite sure if that is what you would like to have

[flakey5]

Apologizes for the delay, got a bit busy. Here's what I'm thinking based off of y'alls comments and my (beginner) understanding of libuv,

For opting in, we add some workerThreads property to the options for createServer (for http 1 and 2). This will tell how many worker threads to make. I'd like for there to be a way to just use all of the threads available, but I feel like workerThreads: true might be a little weird syntax-wise.

Two options that work around the issue that @juanarbol pointed out here with the loops needing to match up:

1. Share the server stream across the different threads
   • I have no idea how possible this actually is, but if it is then it should have the intended result
   • This might be sketchy or slow depending on what needs to be done for it to be thread safe
2. What @juanarbol suggested where the main thread handled the streams while the threads compute the request
   • I'd imagine the stream would hang until timeout or an event from the worker thread (data or end)
   • Also not sure if this would provide the intended result. Although this should still take a lot of load off the main thread ofc

I think the first is worth a shot, but also the second is most doable.

Wdyt? Also, what criteria to we want to consider for deciding which thread to use?