Allow for environment variable substitution in redpanda.yaml

[patrickangeles]

Allow for templating of redpanda.yaml such that environment variables could be applied when the config file is read.

It can make multi-node deployments easier/simpler such that they can be driven off the same configuration file.

JIRA Link: CORE-856

[dotnwat]

would it be sufficient to make all configuration options defined in the configuration file to be overridable by defining something like REDPANDA_SOME_OPTION_NAME=value? Precedence rules can be defined like, cli, then env, then config file.

[michaelandrepearce]

How would that work for complex structures in the config such as list objects for multi listeners.

Having env substitute imo is much simpler, as well it should be as simple as envsubst file before processing the file on read.
Im not a c++ person but in java or .net i would simply add this logic simply detokenising the file prior to file load. Its then a one liner and doesnt change complexity or alter existing config loading/parsing.

Also env substitute allows for just partial value being tokenism, e.g. you may want to have all the nodes having same config except parts where the node id maybe only changed, e.g. the external api listener you may have a full qualified name which is same for all nodes apart from a suffix part on the first segment, as such, e.g. broker-1.redpanda.mydomain.com, broker-2.redpanda.mydomain.com

You could just template the redpanda.yml the value to:
broker-${NODE_ID}.redpanda.mydomain.com

And just then set env value for NODE_ID to 1,2 etc.