[clickhouse] Make it possible for oximeter to read/write to a clickhouse cluster and/or single node

[karencfv]

As part of the work to roll out a replicated ClickHouse cluster, Oximeter needs to have the ability to swap between reading and writing to a single node clickhouse server, a replicated cluster, or both. This will aid in stage one of RFD 468

• [oximeter] Modify to write to both single node and replicated cluster #7419
• [reconfigurator] Ability to switch reads from one clickhouse installation to another #7420

[karencfv]

This functionality is not meant to be long lived, as it will only be used during stage 1 and 2 of RFD 468. This means the solution should be designed accordingly. Nothing overly complex.

A few options come to mind:

Option 1

Have oximeter grow the ability to write to both single node/replicated cluster at the same time and read from a designated one. The idea would be to have a way to tell oximeter to swap between reading from a single node or the replicated cluster. This way we can have our customer racks continue reading from the single node, and have the dogfood rack reading from the replicated cluster. We'll also be able to swap where the reads are happening on dog food. This will allow us to run long running stress tests and compare results between single node and the replicated cluster.

Questions

• Which would be a non-intrusive way to implement a mechanism to swap where reads are happening from. It'd be best to implement something simple that can easily be removed for when we no longer run a single node and a replicated cluster side by side.
• How would reconfigurator handle this change?
• How complex would it be to implement this solution?

Option 2

Create an additional oximeter SMF service in a different zone that has an oximeter agent that connects to the replicated ClickHouse cluster.

Questions

• How would the oximeter producers and collector behave if there is a second oximeter in a different zone?
• Would it make sense/be possible to include this as part of the clickhouse policy? Or would we have to have a separate oximeter policy?
• I'm not sure how we'd handle reads from the API for stress testing.
• How complex would it be to implement this solution?

There may be other options that I haven't thought of here. Would be good to explore a bit more.

[andrewjstone]

Which would be a non-intrusive way to implement a mechanism to swap where reads are happening from. It'd be best to implement something simple that can easily be removed for when we no longer run a single node and a replicated cluster side by side.

We could almost certainly use internal DNS for this. I believe we already update internal DNS via reconfigurator, and even if we don't it should be easy enough.

[karencfv]

After a live chat we've decided to follow with option number 1. Option number 2 is much more complex to implement and doesn't really provide much additional value.