How do we run hydros continously? Serverlessly?

[jlewi]

One of the key goals of hydros is to be declarative. That means we should be continuously running it. Right now we don't have a good story for that. There's a bunch of different issues related to this

#62
#61
#5

There's a couple different patterns we could follow. The current proposal was to

1. Turn the resources into K8s controllers
2. Use existing GitOps toolks to sync to the cluster

This seems unnecessarily heavyweight given all we really need is to run a cronjob. Basically all we need is to periodically run

hydros apply /path/to/repo_config.yaml

That will trigger hydros to check out the repository at the specified commit and then apply any resources we find.

I think we should just try to run this on Cloud Run.

[jlewi]

One big problem here is observability. When a repo contains N resources that need to be applied how do we understand the status of all of them? This is especially important if we rely on reconcile being called multiple times in order to do with any ordering. For example, we have Image resources which build images and ReplicatedImage which replicates them.

[jlewi]

For observability really all we need is a key value store. Using k8s to get ETCD to get a key value store is rather expensive. We could potentially use https://github.com/cockroachdb/pebble. Can we use pebble with GCS is the backend? Is there a diffferent serverless key value store we could use?