Design: Concurrent compilation

[anderseknert]

Screenshot below shows trace output of the first 100 milliseconds of running regal lint (compiling Rego) vs. the next 100 milliseconds (evaluating it). While evaluation is also performed in a single goroutine by default, the Rego API allows the user to introduce concurrency in a safe way. The compile API however provides no such provisions, and so I'm left with 15 cores doing nothing but to watch a single core do all the work for the first 100 milliseconds of the program's execution.

While there are many compiler tasks that need to be run sequentially, there are also those who don't, as well as places that may benefit from concurrency within a single task. But even if we can't perfectly divide our tasks among goroutines, even doing half of them in parallel would likely have quite a positive impact on compilation time anywhere concurrency is possible.

I suggest we start with identifying and documenting the various dependencies that exist in the compiler, and then introduce some "light" concurrency where it feels most obvious/safe to do so. This issue aims to provide space for that discussion, and more issues should be created later for actual implementation items.

[srenatus]

See #8102. There, if we have more than one module, we'll fan out one coroutine doing the stage for each module, wait for them to be done. It's blunt and simple, but we can use it as a base to experiment with other work distribution schemes.