Skip to content

Commit a23f386

Browse files
aknuds1aknuds1
committed
Bucket store: Add GCS rate limiting
Signed-off-by: Arve Knudsen <[email protected]>
1 parent 54bfe91 commit a23f386

File tree

11 files changed

+1672
-12
lines changed

11 files changed

+1672
-12
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
* [FEATURE] Memberlist: Add experimental support for zone-aware routing in order to reduce memberlist cross-AZ data transfer. #13129 #13651 #13664
3535
* [FEATURE] Query-frontend and querier: Add experimental support for performing query planning in query-frontends and distributing portions of the plan to queriers for execution. #13058
3636
* [FEATURE] Querier: Add `querier.mimir-query-engine.enable-reduce-matchers` flag that enables a new MQE AST optimization pass that eliminates duplicate or redundant matchers that are part of selector expressions. #13178
37+
* [ENHANCEMENT] Bucket storage: Add support for GCS rate limiting with exponential ramping following Google Cloud Storage best practices. Enable upload rate limiting with `-gcs.upload-rate-limit-enabled` and configure with `-gcs.upload-max-qps` and `-gcs.upload-ramp-period`. Enable read rate limiting with `-gcs.read-rate-limit-enabled` and configure with `-gcs.read-max-qps` and `-gcs.read-ramp-period`. #13703
3738
* [ENHANCEMENT] Compactor, Store-gateway: Change default value of `-compactor.upload-sparse-index-headers` to `true`. This improves lazy loading performance in the store-gateway. #13089
3839
* [ENHANCEMENT] Store-gateway: Verify CRC32 checksums for 1 out of every 128 chunks read from object storage and the chunks cache to detect corruption. #13151
3940
* [ENHANCEMENT] Ingester: the per-tenant postings for matchers cache is now stable. Use the following configuration options: #13101

‎cmd/mimir/config-descriptor.json‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9095,6 +9095,72 @@
90959095
"fieldType": "int",
90969096
"fieldCategory": "advanced"
90979097
},
9098+
{
9099+
"kind": "field",
9100+
"name": "upload_rate_limit_enabled",
9101+
"required": false,
9102+
"desc": "Enable rate limiting for GCS uploads. When enabled, uploads will gradually ramp up following Google Cloud Storage best practices.",
9103+
"fieldValue": null,
9104+
"fieldDefaultValue": false,
9105+
"fieldFlag": "blocks-storage.gcs.upload-rate-limit-enabled",
9106+
"fieldType": "boolean",
9107+
"fieldCategory": "advanced"
9108+
},
9109+
{
9110+
"kind": "field",
9111+
"name": "upload_max_qps",
9112+
"required": false,
9113+
"desc": "Maximum queries per second limit for GCS uploads.",
9114+
"fieldValue": null,
9115+
"fieldDefaultValue": 3200,
9116+
"fieldFlag": "blocks-storage.gcs.upload-max-qps",
9117+
"fieldType": "int",
9118+
"fieldCategory": "advanced"
9119+
},
9120+
{
9121+
"kind": "field",
9122+
"name": "upload_ramp_period",
9123+
"required": false,
9124+
"desc": "Time period over which the upload rate doubles, following Google's recommendation.",
9125+
"fieldValue": null,
9126+
"fieldDefaultValue": 1200000000000,
9127+
"fieldFlag": "blocks-storage.gcs.upload-ramp-period",
9128+
"fieldType": "duration",
9129+
"fieldCategory": "advanced"
9130+
},
9131+
{
9132+
"kind": "field",
9133+
"name": "read_rate_limit_enabled",
9134+
"required": false,
9135+
"desc": "Enable rate limiting for GCS reads. When enabled, reads will gradually ramp up following Google Cloud Storage best practices.",
9136+
"fieldValue": null,
9137+
"fieldDefaultValue": false,
9138+
"fieldFlag": "blocks-storage.gcs.read-rate-limit-enabled",
9139+
"fieldType": "boolean",
9140+
"fieldCategory": "advanced"
9141+
},
9142+
{
9143+
"kind": "field",
9144+
"name": "read_max_qps",
9145+
"required": false,
9146+
"desc": "Maximum queries per second limit for GCS reads.",
9147+
"fieldValue": null,
9148+
"fieldDefaultValue": 16000,
9149+
"fieldFlag": "blocks-storage.gcs.read-max-qps",
9150+
"fieldType": "int",
9151+
"fieldCategory": "advanced"
9152+
},
9153+
{
9154+
"kind": "field",
9155+
"name": "read_ramp_period",
9156+
"required": false,
9157+
"desc": "Time period over which the read rate doubles, following Google's recommendation.",
9158+
"fieldValue": null,
9159+
"fieldDefaultValue": 1200000000000,
9160+
"fieldFlag": "blocks-storage.gcs.read-ramp-period",
9161+
"fieldType": "duration",
9162+
"fieldCategory": "advanced"
9163+
},
90989164
{
90999165
"kind": "block",
91009166
"name": "http",
@@ -14949,6 +15015,72 @@
1494915015
"fieldType": "int",
1495015016
"fieldCategory": "advanced"
1495115017
},
15018+
{
15019+
"kind": "field",
15020+
"name": "upload_rate_limit_enabled",
15021+
"required": false,
15022+
"desc": "Enable rate limiting for GCS uploads. When enabled, uploads will gradually ramp up following Google Cloud Storage best practices.",
15023+
"fieldValue": null,
15024+
"fieldDefaultValue": false,
15025+
"fieldFlag": "ruler-storage.gcs.upload-rate-limit-enabled",
15026+
"fieldType": "boolean",
15027+
"fieldCategory": "advanced"
15028+
},
15029+
{
15030+
"kind": "field",
15031+
"name": "upload_max_qps",
15032+
"required": false,
15033+
"desc": "Maximum queries per second limit for GCS uploads.",
15034+
"fieldValue": null,
15035+
"fieldDefaultValue": 3200,
15036+
"fieldFlag": "ruler-storage.gcs.upload-max-qps",
15037+
"fieldType": "int",
15038+
"fieldCategory": "advanced"
15039+
},
15040+
{
15041+
"kind": "field",
15042+
"name": "upload_ramp_period",
15043+
"required": false,
15044+
"desc": "Time period over which the upload rate doubles, following Google's recommendation.",
15045+
"fieldValue": null,
15046+
"fieldDefaultValue": 1200000000000,
15047+
"fieldFlag": "ruler-storage.gcs.upload-ramp-period",
15048+
"fieldType": "duration",
15049+
"fieldCategory": "advanced"
15050+
},
15051+
{
15052+
"kind": "field",
15053+
"name": "read_rate_limit_enabled",
15054+
"required": false,
15055+
"desc": "Enable rate limiting for GCS reads. When enabled, reads will gradually ramp up following Google Cloud Storage best practices.",
15056+
"fieldValue": null,
15057+
"fieldDefaultValue": false,
15058+
"fieldFlag": "ruler-storage.gcs.read-rate-limit-enabled",
15059+
"fieldType": "boolean",
15060+
"fieldCategory": "advanced"
15061+
},
15062+
{
15063+
"kind": "field",
15064+
"name": "read_max_qps",
15065+
"required": false,
15066+
"desc": "Maximum queries per second limit for GCS reads.",
15067+
"fieldValue": null,
15068+
"fieldDefaultValue": 16000,
15069+
"fieldFlag": "ruler-storage.gcs.read-max-qps",
15070+
"fieldType": "int",
15071+
"fieldCategory": "advanced"
15072+
},
15073+
{
15074+
"kind": "field",
15075+
"name": "read_ramp_period",
15076+
"required": false,
15077+
"desc": "Time period over which the read rate doubles, following Google's recommendation.",
15078+
"fieldValue": null,
15079+
"fieldDefaultValue": 1200000000000,
15080+
"fieldFlag": "ruler-storage.gcs.read-ramp-period",
15081+
"fieldType": "duration",
15082+
"fieldCategory": "advanced"
15083+
},
1495215084
{
1495315085
"kind": "block",
1495415086
"name": "http",
@@ -17240,6 +17372,72 @@
1724017372
"fieldType": "int",
1724117373
"fieldCategory": "advanced"
1724217374
},
17375+
{
17376+
"kind": "field",
17377+
"name": "upload_rate_limit_enabled",
17378+
"required": false,
17379+
"desc": "Enable rate limiting for GCS uploads. When enabled, uploads will gradually ramp up following Google Cloud Storage best practices.",
17380+
"fieldValue": null,
17381+
"fieldDefaultValue": false,
17382+
"fieldFlag": "alertmanager-storage.gcs.upload-rate-limit-enabled",
17383+
"fieldType": "boolean",
17384+
"fieldCategory": "advanced"
17385+
},
17386+
{
17387+
"kind": "field",
17388+
"name": "upload_max_qps",
17389+
"required": false,
17390+
"desc": "Maximum queries per second limit for GCS uploads.",
17391+
"fieldValue": null,
17392+
"fieldDefaultValue": 3200,
17393+
"fieldFlag": "alertmanager-storage.gcs.upload-max-qps",
17394+
"fieldType": "int",
17395+
"fieldCategory": "advanced"
17396+
},
17397+
{
17398+
"kind": "field",
17399+
"name": "upload_ramp_period",
17400+
"required": false,
17401+
"desc": "Time period over which the upload rate doubles, following Google's recommendation.",
17402+
"fieldValue": null,
17403+
"fieldDefaultValue": 1200000000000,
17404+
"fieldFlag": "alertmanager-storage.gcs.upload-ramp-period",
17405+
"fieldType": "duration",
17406+
"fieldCategory": "advanced"
17407+
},
17408+
{
17409+
"kind": "field",
17410+
"name": "read_rate_limit_enabled",
17411+
"required": false,
17412+
"desc": "Enable rate limiting for GCS reads. When enabled, reads will gradually ramp up following Google Cloud Storage best practices.",
17413+
"fieldValue": null,
17414+
"fieldDefaultValue": false,
17415+
"fieldFlag": "alertmanager-storage.gcs.read-rate-limit-enabled",
17416+
"fieldType": "boolean",
17417+
"fieldCategory": "advanced"
17418+
},
17419+
{
17420+
"kind": "field",
17421+
"name": "read_max_qps",
17422+
"required": false,
17423+
"desc": "Maximum queries per second limit for GCS reads.",
17424+
"fieldValue": null,
17425+
"fieldDefaultValue": 16000,
17426+
"fieldFlag": "alertmanager-storage.gcs.read-max-qps",
17427+
"fieldType": "int",
17428+
"fieldCategory": "advanced"
17429+
},
17430+
{
17431+
"kind": "field",
17432+
"name": "read_ramp_period",
17433+
"required": false,
17434+
"desc": "Time period over which the read rate doubles, following Google's recommendation.",
17435+
"fieldValue": null,
17436+
"fieldDefaultValue": 1200000000000,
17437+
"fieldFlag": "alertmanager-storage.gcs.read-ramp-period",
17438+
"fieldType": "duration",
17439+
"fieldCategory": "advanced"
17440+
},
1724317441
{
1724417442
"kind": "block",
1724517443
"name": "http",
@@ -20132,6 +20330,72 @@
2013220330
"fieldType": "int",
2013320331
"fieldCategory": "advanced"
2013420332
},
20333+
{
20334+
"kind": "field",
20335+
"name": "upload_rate_limit_enabled",
20336+
"required": false,
20337+
"desc": "Enable rate limiting for GCS uploads. When enabled, uploads will gradually ramp up following Google Cloud Storage best practices.",
20338+
"fieldValue": null,
20339+
"fieldDefaultValue": false,
20340+
"fieldFlag": "common.storage.gcs.upload-rate-limit-enabled",
20341+
"fieldType": "boolean",
20342+
"fieldCategory": "advanced"
20343+
},
20344+
{
20345+
"kind": "field",
20346+
"name": "upload_max_qps",
20347+
"required": false,
20348+
"desc": "Maximum queries per second limit for GCS uploads.",
20349+
"fieldValue": null,
20350+
"fieldDefaultValue": 3200,
20351+
"fieldFlag": "common.storage.gcs.upload-max-qps",
20352+
"fieldType": "int",
20353+
"fieldCategory": "advanced"
20354+
},
20355+
{
20356+
"kind": "field",
20357+
"name": "upload_ramp_period",
20358+
"required": false,
20359+
"desc": "Time period over which the upload rate doubles, following Google's recommendation.",
20360+
"fieldValue": null,
20361+
"fieldDefaultValue": 1200000000000,
20362+
"fieldFlag": "common.storage.gcs.upload-ramp-period",
20363+
"fieldType": "duration",
20364+
"fieldCategory": "advanced"
20365+
},
20366+
{
20367+
"kind": "field",
20368+
"name": "read_rate_limit_enabled",
20369+
"required": false,
20370+
"desc": "Enable rate limiting for GCS reads. When enabled, reads will gradually ramp up following Google Cloud Storage best practices.",
20371+
"fieldValue": null,
20372+
"fieldDefaultValue": false,
20373+
"fieldFlag": "common.storage.gcs.read-rate-limit-enabled",
20374+
"fieldType": "boolean",
20375+
"fieldCategory": "advanced"
20376+
},
20377+
{
20378+
"kind": "field",
20379+
"name": "read_max_qps",
20380+
"required": false,
20381+
"desc": "Maximum queries per second limit for GCS reads.",
20382+
"fieldValue": null,
20383+
"fieldDefaultValue": 16000,
20384+
"fieldFlag": "common.storage.gcs.read-max-qps",
20385+
"fieldType": "int",
20386+
"fieldCategory": "advanced"
20387+
},
20388+
{
20389+
"kind": "field",
20390+
"name": "read_ramp_period",
20391+
"required": false,
20392+
"desc": "Time period over which the read rate doubles, following Google's recommendation.",
20393+
"fieldValue": null,
20394+
"fieldDefaultValue": 1200000000000,
20395+
"fieldFlag": "common.storage.gcs.read-ramp-period",
20396+
"fieldType": "duration",
20397+
"fieldCategory": "advanced"
20398+
},
2013520399
{
2013620400
"kind": "block",
2013720401
"name": "http",

0 commit comments

Comments
 (0)