Multithreaded Compilation

[brockelmore]

Requested Feature

For sufficiently large repos, solc compilation can be the slowest part of testing. In those cases we should multithread compilation as much as possible. Here is a rough sketch of a potential way to achieve this?

Suggested solution

1. Determine a threshold for multithreading compilation (# of sol files in repo?) that offsets any overhead
2. Spin up 1 thread per file, determining dependency graph
3. Hash trees/dep graph, and remove duplicate trees
4. Compile the needed contracts using the existing threads, in parallel

There could be issues here that I am unaware of but documenting the thoughts I had anyway

[maxsam4]

I feel like the overhead will not be worth it in most cases. Individual contracts are usually so small that invoking solc is going to consume more time than solc compiling them.

This feature would make more sense in solc itself but I won't put it on the priority list even there.

[lucas-manuel]

I tend to agree that it would be cool but could be more trouble than it is worth, especially if we have compilation caching etc.

[mattsse]

this is actually already in use for workspaces that require different solc versions
gakonst/ethers-rs#652

[gakonst]

@mattsse bumping this, seems like next up in our priority list along with #769?

[onbjerg]

Marking this and #769 as high prio if I understand your comment correctly