Concurrent processing

[keithlayne]

I'm sure we could have fun running Promise.all(<7000 things>).

I don't have experience with node and doing this kinda thing. Obviously when shelling out there are lots of processes, and that's one thing, but I imagine something like #-of-processors concurrent jobs would probably be a good starting point.

Maybe there's already something that's a perfect drop-in for node for this. I don't know if something like workers makes sense, or just run async off a work queue.

We could probably wrangle sqlite into a acting as a queue, looks like it does data change callbacks.

[cspotcode]

I've used throat or p-queue with promises to get a simple max-concurrency worker pool. Combined with zx we can call out to external processes.

The SQLite hack I threw together does everything in-memory. To use data change callbacks and other cross-process stuff, we'll need to switch from sql.js to a proper native binding. I think native bindings exist so it should be fine to switch.

[keithlayne]

Okay, yeah, I had looked at that lib in the past but forgot how it worked. Feels like it would be more reliable and flexible to run it off of disk, but maybe that's overkill.

I also haven't used zx before. I'll have to check out the docs. I tend to write scripts either in shell or some other scripting language, and usually don't do exec-type stuff from other languages, which I'll admit is silly and I have no good reason why.

[keithlayne]

Also, just confirming: for now the goal just to run typedoc against the top-level targets? As in, not against the deps at all?

I was imagining pushing the deps into the queue while working on the parent at some point. Seems potentially racy.

[cspotcode]

I have only been attempting to run against the top-level, not against deps. But if anyone wants to push deps into the queue, I'm all for it.

…

On Wed, Jul 14, 2021, 5:44 PM Keith Layne ***@***.***> wrote: Also, just confirming: for now the goal just to run typedoc against the top-level targets? As in, not against the deps at all? I was imagining pushing the deps into the queue while working on the parent at some point. Seems potentially racy. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OF6DVU6TVZZFFATB6TTXYALXANCNFSM5AMHTRXA> .

[cspotcode]

This is my first time using zx, but I'm surprised at how easy it is. I think it calls out to bash under the hood anyway.

…

On Wed, Jul 14, 2021, 5:51 PM Andrew Bradley ***@***.***> wrote: I have only been attempting to run against the top-level, not against deps. But if anyone wants to push deps into the queue, I'm all for it. On Wed, Jul 14, 2021, 5:44 PM Keith Layne ***@***.***> wrote: > Also, just confirming: for now the goal just to run typedoc against the > top-level targets? As in, not against the deps at all? > > I was imagining pushing the deps into the queue while working on the > parent at some point. Seems potentially racy. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#11 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAC35OF6DVU6TVZZFFATB6TTXYALXANCNFSM5AMHTRXA> > . >