Move parsing logic to a warmup job

[ZIJ]

Currently Digger orchestrator checks out the repo first in order to parse digger.yml and determine the impacted projects. This approach is problematic for 2 reasons:

1. The orchestrator has to be granted higher level of privilege than it otherwise would need. And if encrypted sensitive data stored in the repo (e.g. sops) then the orchestrator would need to have decryption privileges, which would effectively make it "ring zero" in terms of trust level.
2. Filesystem operations on the orchestrator (git pull) can lead to performance issues, and make it more difficult to achieve HA setup compared to a service that does not rely on FS. Also in case of heavy repos this will increase the memory footprint.

We could solve for both by doing the initial parsing within the CI instead of the orchestrator:

1. Orchestrator receives a request via webhook, but does not do any parsing
2. Orchestrator triggers a "warmup job" that does parsing (with decryption if needed) and determines which projects are affected
3. The warmup job reports its findings back to the orchestrator, e.g. "projects A, C, D and Z need to be planned"
4. The orchestrator triggers 3 additional jobs to plan projects C, D and Z
5. The warmup job continues with project A

The continuation of the first job is actually optional - it'll only save some init time. We could just as well trigger 4 jobs afresh for each project. It can be argued that this approach is actually cleaner because it'll make it easy to understand and debug in case of failures; each job only does one thing.

[ZIJ]

Related to

• Support sops secret encryption #1106