receive: add --tsdb.flush-blocks-on-shutdown flag

[prymitive]

By default Receive will flush (write to disk and/or upload to S3) on shutdown. This takes ages and we don't need it at all because we run Thanos on bare metal. Allow this to be configurable since this feature seems to be aimed mostly at environments like k8s, where instances ephemeral.

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Verification

[GiedriusS]

How to do maintenance on physical servers without dumping blocks on shut down? Is there any API you use or how do you do it? Please elaborate.

[prymitive]

How to do maintenance on physical servers without dumping blocks on shut down? Is there any API you use or how do you do it? Please elaborate.

I don’t think I follow.
Is this question assuming that there is no replication and everything is down while you take one instance out?
Prometheus doesn’t write blocks on shutdown and yet people run it on many physical servers.

[prymitive]

I tried to run make docs but that made a ton of changes to a lot of files.
Either my local clone did something wrong or all docs are very stale.

[MichaHoffmann]

Lets ignore the docs check, ill add a followup PR to sync the docs.

[prymitive]

that was my laptop, had older thanos binary in my PATH and make build puts binaries in go/bin for some reason

[GiedriusS]

How to do maintenance on physical servers without dumping blocks on shut down? Is there any API you use or how do you do it? Please elaborate.

I don’t think I follow. Is this question assuming that there is no replication and everything is down while you take one instance out? Prometheus doesn’t write blocks on shutdown and yet people run it on many physical servers.

With replication factor 1 or 2 (or any other RF really) and with this flag enabled, how to wipe/destroy a server without losing any recently written data?

[prymitive]

With replication factor 1 or 2 (or any other RF really) and with this flag enabled, how to wipe/destroy a server without losing any recently written data?

I imagine that you would:

• stop writing to that server
• let it sit there until it writes a block and uploads it to object storage
• remove that instance

[GiedriusS]

With replication factor 1 or 2 (or any other RF really) and with this flag enabled, how to wipe/destroy a server without losing any recently written data?

I imagine that you would:

* stop writing to that server

* let it sit there until it writes a block and uploads it to object storage

* remove that instance

So, at that point, you only have to do very few checks before shutting down because all uploads have already succeeded, no?

[prymitive]

So, at that point, you only have to do very few checks before shutting down because all uploads have already succeeded, no?

In a theoretical scenario that I haven't actually tried, yes.
But there's a lot of ifs here, for example there's an assumption that I'm using object storage - but I don't have to, I can have a bare metal setup that's simply used for sharding
that doesn't upload anything anywhere.