Consolidate Binaries and Configuration Sprawl

[jonstacks]

We now have 3 different k8s deployments (api,tunnel, and bindings) each running their own manager and each in a separate binary. Each of these also takes their own cobra command line argument options. Some of these options should be shared across each manager and some are unique to each manager.

There are 2 things I think we should tackle:

1. Our image size is now much larger. Since we build 3 statically linked binaries, much of the each binary size is probably our libs + stdlib. We can probably re-shrink our image size and reduce compilation time by creating a single binary that takes an argument for which manager to run. Ex: ./manager tunnel or ./manager bindings
2. Configuration sprawl. When we add a new CLI option to a manger, we have to add custom serialization to the helm chart depending on the field type and then we also have to de-serialize it from the command line argument in our go program. Some of these options are duplicated across the managers. You have to go look at each k8s deployment to figure out the configuration for the system. It would be nice if we had a single config file that contained the global options as well as the options for each manager.

[jonstacks]

@masonj5n and I talked about this synchronously and decided to just consolidate the binaries for now and hold off on the configuration sprawl.

Some goals we'd like to fix with the configuration sprawl IMO:

1. Not passing config as individual arguments to the binary. This makes us have to template out each new config value in helm and makes the arguments section a bit unwieldy, especially when it comes to defaulting options. We also have defaults specified in the command flags which makes it difficult to maintain a single source of truth.
2. Having a single source of truth for defaults. I believe this source of truth should be the go code(probably constants). This would allow us to not have to plumb a bunch of values from the values.yaml file(with logic) through to a proposed ConfigMap. Instead, when users want to override the default config, they simply supply it as K/V pairs.
3. A single place to see config. This likely looks like a ConfigMap with just the overrides specified. Whether or not we want to pass values from the configmap as environment variables or we want to mount the configmap as a volume seems like an implementation detail. Having a single configmap that shows all of the overrides of default values seems like it could be useful for troubleshooting customer issues.

An example of some clean configuration I've seen is Argo's argo-cmd-params-cm.yaml. It has reasonable defaults and I can override only the things I want to by setting those properties in a ConfigMap like so:

apiVersion: v1
data:
  applicationsetcontroller.debug: "true"
  applicationsetcontroller.log.level: debug
  server.log.level: debug
kind: ConfigMap