FTR:

• We could add a standalone Spanner emulator binary to our syz-env image like described here: https://github.com/GoogleCloudPlatform/cloud-spanner-emulator/blob/master/README.md#via-pre-built-linux-binaries
• parallelism: 1 and failFast: true could be configured per workflow template: https://argo-workflows.readthedocs.io/en/latest/fields/#workflowspec. Thus the iterateItems step could be made its own template, but there's still a question on how to distinguish between "try the next kernel revision" and "I've managed to build this one. Stop iterating.".
• Lifecycle hooks seem to be the recommended way of implementing onExit callbacks: https://argo-workflows.readthedocs.io/en/latest/lifecyclehook/
• MinIO seems be unavoidable for local argo workflow runs, there are no simpler artifact storage options.

We could add a standalone Spanner emulator binary

Please give more context. Do you need it for the local development or testing?
It should be Ok to use it as a dev environment but I'm hesitating to add it as a CI dependency.
Staging environment may be more reliable way for check your system. And spanner mocks looks like a better (more stable, predictable, faster) alternative for CI testing.

We could add a standalone Spanner emulator binary

Please give more context. Do you need it for the local development or testing? It should be Ok to use it as a dev environment but I'm hesitating to add it as a CI dependency. Staging environment may be more reliable way for check your system. And spanner mocks looks like a better (more stable, predictable, faster) alternative for CI testing.

For dev environment, it already exists - there's make run-go-tests in syz-cluster that runs the tests in the k8s environment. I meant running those tests on the GitHub CI as well (ideally by getting rid of make run-go-tests as well).

I've just added a syz-cluster/pkg/db: add support for running under syz-env commit that runs the Spanner emulator binary for our Go tests. Seems to be quite simple, actually.

For some other external dependencies (e.g. Argo workflows or the blob storage), there are already mocks in the code, but for the DB I fear that there will just be too much boilerplate code. If we can just run a db emulator for the unit tests, why not do it?

Doing so lets us test not just the higher level logic, but also all SQL commands that are used in the implementations.

Unit testing allows to test some component in the isolated environment where you can simulate any kind of the external service behavior (inclulding failure). Mocks give you the required flexibility and are the faster way to emulate external dependencies.
Using the emulator for unit testing we make UT slower and test some interaction... It is not the unit testing by definition... I think it worth discussion...

Emulator may be used to test interaction. In our case we can just use real service instead of the emulator that will always differ from it.

Let's add the testing and error processing sections to the design doc?