Update the default configuration to enable ingest storage concurrency (#15072)

#### What this PR does

Today, when talking to a community member, I realised ingest storage
concurrency is still disabled in the default config. At Grafana Labs,
we're running it in production since a long time, and we should just
make it the default.

#### Which issue(s) this PR fixes or relates to

N/A

#### Checklist

- [ ] Tests updated.
- [ ] Documentation added.
- [x] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.

---------

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Author: pracucci

CHANGELOG.md

@@ -186,6 +186,12 @@
 * [ENHANCEMENT] Distributor: OTLP endpoint now returns partial success (HTTP 200) instead of HTTP 429 when the usage tracker rejects some series due to the active series limit but other series are successfully ingested. The `RejectedDataPoints` field reports the count of distributor-side rejections (usage tracker filtering). #14789
 * [ENHANCEMENT] MQE: Account for memory consumption of labels returned by binary operations in query memory consumption estimate earlier. #15033
 * [ENHANCEMENT] Query-frontend: Log the number of series and samples returned for queries in `query stats` log lines. #15044
+* [ENHANCEMENT] Ingest storage: Update the default configuration to enable ingest storage concurrency: #15072
+  * `-ingest-storage.kafka.fetch-concurrency-max` from `0` to `12`
+  * `-ingest-storage.kafka.ingestion-concurrency-max` from `0` to `8`
+  * `-ingest-storage.kafka.ingestion-concurrency-queue-capacity` from `5` to `3`
+  * `-ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard` from `80` to `40`
+  * `-ingest-storage.kafka.max-buffered-bytes` from `100MB` to `1GB`
 * [BUGFIX] Ingester: enforce a minimum 10s delay between TSDB head compaction iterations when an iteration approaches or exceeds the configured `-blocks-storage.tsdb.head-compaction-interval`, so ingestion is not starved by back-to-back compactions. #15061
 * [BUGFIX] Update to Go v1.25.9. #15030
 * [BUGFIX] Distributor: OTLP partial success responses now correctly populate `RejectedDataPoints` with the actual count of rejected samples, instead of always reporting 0. In classical architecture, this includes rejected samples propagated from the ingester. #14789

cmd/mimir/config-descriptor.json

@@ -9236,7 +9236,7 @@
               "required": false,
               "desc": "The maximum number of concurrent fetch requests that the ingester makes when reading data from Kafka during startup. Concurrent fetch requests are issued only when there is sufficient backlog of records to consume. Set to 0 to disable.",
               "fieldValue": null,
-              "fieldDefaultValue": 0,
+              "fieldDefaultValue": 12,
               "fieldFlag": "ingest-storage.kafka.fetch-concurrency-max",
               "fieldType": "int"
             },
@@ -9256,7 +9256,7 @@
               "required": false,
               "desc": "The maximum number of buffered records ready to be processed. This limit applies to the sum of all inflight requests. Set to 0 to disable the limit.",
               "fieldValue": null,
-              "fieldDefaultValue": 100000000,
+              "fieldDefaultValue": 1000000000,
               "fieldFlag": "ingest-storage.kafka.max-buffered-bytes",
               "fieldType": "int"
             },
@@ -9266,7 +9266,7 @@
               "required": false,
               "desc": "The maximum number of concurrent ingestion streams to the TSDB head. Every tenant has their own set of streams. 0 to disable.",
               "fieldValue": null,
-              "fieldDefaultValue": 0,
+              "fieldDefaultValue": 8,
               "fieldFlag": "ingest-storage.kafka.ingestion-concurrency-max",
               "fieldType": "int"
             },
@@ -9286,7 +9286,7 @@
               "required": false,
               "desc": "The number of batches to prepare and queue to ingest to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0.",
               "fieldValue": null,
-              "fieldDefaultValue": 5,
+              "fieldDefaultValue": 3,
               "fieldFlag": "ingest-storage.kafka.ingestion-concurrency-queue-capacity",
               "fieldType": "int"
             },
@@ -9296,7 +9296,7 @@
               "required": false,
               "desc": "The expected number of times to ingest timeseries to the TSDB head after batching. With fewer flushes, the overhead of splitting up the work is higher than the benefit of parallelization. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0.",
               "fieldValue": null,
-              "fieldDefaultValue": 80,
+              "fieldDefaultValue": 40,
               "fieldFlag": "ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard",
               "fieldType": "int"
             },

cmd/mimir/help-all.txt.tmpl

@@ -1480,25 +1480,25 @@ Usage of ./cmd/mimir/mimir:
   -ingest-storage.kafka.dial-timeout duration
     	The maximum time allowed to open a connection to a Kafka broker. (default 2s)
   -ingest-storage.kafka.fetch-concurrency-max int
-    	The maximum number of concurrent fetch requests that the ingester makes when reading data from Kafka during startup. Concurrent fetch requests are issued only when there is sufficient backlog of records to consume. Set to 0 to disable.
+    	The maximum number of concurrent fetch requests that the ingester makes when reading data from Kafka during startup. Concurrent fetch requests are issued only when there is sufficient backlog of records to consume. Set to 0 to disable. (default 12)
   -ingest-storage.kafka.fetch-max-wait duration
     	The maximum amount of time a Kafka broker waits for some records before a Fetch response is returned. (default 5s)
   -ingest-storage.kafka.ingestion-concurrency-batch-size int
     	The number of timeseries to batch together before ingesting to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 150)
   -ingest-storage.kafka.ingestion-concurrency-estimated-bytes-per-sample int
     	The estimated number of bytes a sample has at time of ingestion. This value is used to estimate the timeseries without decompressing them. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 500)
   -ingest-storage.kafka.ingestion-concurrency-max int
-    	The maximum number of concurrent ingestion streams to the TSDB head. Every tenant has their own set of streams. 0 to disable.
+    	The maximum number of concurrent ingestion streams to the TSDB head. Every tenant has their own set of streams. 0 to disable. (default 8)
   -ingest-storage.kafka.ingestion-concurrency-queue-capacity int
-    	The number of batches to prepare and queue to ingest to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 5)
+    	The number of batches to prepare and queue to ingest to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 3)
   -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard int
-    	The expected number of times to ingest timeseries to the TSDB head after batching. With fewer flushes, the overhead of splitting up the work is higher than the benefit of parallelization. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 80)
+    	The expected number of times to ingest timeseries to the TSDB head after batching. With fewer flushes, the overhead of splitting up the work is higher than the benefit of parallelization. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 40)
   -ingest-storage.kafka.last-produced-offset-poll-interval duration
     	How frequently to poll the last produced offset, used to enforce strong read consistency. (default 1s)
   -ingest-storage.kafka.last-produced-offset-retry-timeout duration
     	How long to retry a failed request to get the last produced offset. (default 10s)
   -ingest-storage.kafka.max-buffered-bytes int
-    	The maximum number of buffered records ready to be processed. This limit applies to the sum of all inflight requests. Set to 0 to disable the limit. (default 100000000)
+    	The maximum number of buffered records ready to be processed. This limit applies to the sum of all inflight requests. Set to 0 to disable the limit. (default 1000000000)
   -ingest-storage.kafka.max-consumer-lag-at-startup duration
     	The guaranteed maximum lag before a consumer is considered to have caught up reading from a partition at startup, becomes ACTIVE in the hash ring and passes the readiness check. Set both -ingest-storage.kafka.target-consumer-lag-at-startup and -ingest-storage.kafka.max-consumer-lag-at-startup to 0 to disable waiting for maximum consumer lag being honored at startup. (default 15s)
   -ingest-storage.kafka.producer-max-buffered-bytes int

cmd/mimir/help.txt.tmpl

@@ -416,25 +416,25 @@ Usage of ./cmd/mimir/mimir:
   -ingest-storage.kafka.dial-timeout duration
     	The maximum time allowed to open a connection to a Kafka broker. (default 2s)
   -ingest-storage.kafka.fetch-concurrency-max int
-    	The maximum number of concurrent fetch requests that the ingester makes when reading data from Kafka during startup. Concurrent fetch requests are issued only when there is sufficient backlog of records to consume. Set to 0 to disable.
+    	The maximum number of concurrent fetch requests that the ingester makes when reading data from Kafka during startup. Concurrent fetch requests are issued only when there is sufficient backlog of records to consume. Set to 0 to disable. (default 12)
   -ingest-storage.kafka.fetch-max-wait duration
     	The maximum amount of time a Kafka broker waits for some records before a Fetch response is returned. (default 5s)
   -ingest-storage.kafka.ingestion-concurrency-batch-size int
     	The number of timeseries to batch together before ingesting to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 150)
   -ingest-storage.kafka.ingestion-concurrency-estimated-bytes-per-sample int
     	The estimated number of bytes a sample has at time of ingestion. This value is used to estimate the timeseries without decompressing them. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 500)
   -ingest-storage.kafka.ingestion-concurrency-max int
-    	The maximum number of concurrent ingestion streams to the TSDB head. Every tenant has their own set of streams. 0 to disable.
+    	The maximum number of concurrent ingestion streams to the TSDB head. Every tenant has their own set of streams. 0 to disable. (default 8)
   -ingest-storage.kafka.ingestion-concurrency-queue-capacity int
-    	The number of batches to prepare and queue to ingest to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 5)
+    	The number of batches to prepare and queue to ingest to the TSDB head. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 3)
   -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard int
-    	The expected number of times to ingest timeseries to the TSDB head after batching. With fewer flushes, the overhead of splitting up the work is higher than the benefit of parallelization. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 80)
+    	The expected number of times to ingest timeseries to the TSDB head after batching. With fewer flushes, the overhead of splitting up the work is higher than the benefit of parallelization. Only use this setting when -ingest-storage.kafka.ingestion-concurrency-max is greater than 0. (default 40)
   -ingest-storage.kafka.last-produced-offset-poll-interval duration
     	How frequently to poll the last produced offset, used to enforce strong read consistency. (default 1s)
   -ingest-storage.kafka.last-produced-offset-retry-timeout duration
     	How long to retry a failed request to get the last produced offset. (default 10s)
   -ingest-storage.kafka.max-buffered-bytes int
-    	The maximum number of buffered records ready to be processed. This limit applies to the sum of all inflight requests. Set to 0 to disable the limit. (default 100000000)
+    	The maximum number of buffered records ready to be processed. This limit applies to the sum of all inflight requests. Set to 0 to disable the limit. (default 1000000000)
   -ingest-storage.kafka.max-consumer-lag-at-startup duration
     	The guaranteed maximum lag before a consumer is considered to have caught up reading from a partition at startup, becomes ACTIVE in the hash ring and passes the readiness check. Set both -ingest-storage.kafka.target-consumer-lag-at-startup and -ingest-storage.kafka.max-consumer-lag-at-startup to 0 to disable waiting for maximum consumer lag being honored at startup. (default 15s)
   -ingest-storage.kafka.producer-max-buffered-bytes int

docs/sources/mimir/configure/configuration-parameters/index.md

@@ -5450,7 +5450,7 @@ kafka:
   # only when there is sufficient backlog of records to consume. Set to 0 to
   # disable.
   # CLI flag: -ingest-storage.kafka.fetch-concurrency-max
-  [fetch_concurrency_max: <int> | default = 0]
+  [fetch_concurrency_max: <int> | default = 12]
 
   # When enabled, the fetch request MaxBytes field is computed using the
   # compressed size of previous records. When disabled, MaxBytes is computed
@@ -5462,12 +5462,12 @@ kafka:
   # The maximum number of buffered records ready to be processed. This limit
   # applies to the sum of all inflight requests. Set to 0 to disable the limit.
   # CLI flag: -ingest-storage.kafka.max-buffered-bytes
-  [max_buffered_bytes: <int> | default = 100000000]
+  [max_buffered_bytes: <int> | default = 1000000000]
 
   # The maximum number of concurrent ingestion streams to the TSDB head. Every
   # tenant has their own set of streams. 0 to disable.
   # CLI flag: -ingest-storage.kafka.ingestion-concurrency-max
-  [ingestion_concurrency_max: <int> | default = 0]
+  [ingestion_concurrency_max: <int> | default = 8]
 
   # The number of timeseries to batch together before ingesting to the TSDB
   # head. Only use this setting when
@@ -5479,14 +5479,14 @@ kafka:
   # use this setting when -ingest-storage.kafka.ingestion-concurrency-max is
   # greater than 0.
   # CLI flag: -ingest-storage.kafka.ingestion-concurrency-queue-capacity
-  [ingestion_concurrency_queue_capacity: <int> | default = 5]
+  [ingestion_concurrency_queue_capacity: <int> | default = 3]
 
   # The expected number of times to ingest timeseries to the TSDB head after
   # batching. With fewer flushes, the overhead of splitting up the work is
   # higher than the benefit of parallelization. Only use this setting when
   # -ingest-storage.kafka.ingestion-concurrency-max is greater than 0.
   # CLI flag: -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard
-  [ingestion_concurrency_target_flushes_per_shard: <int> | default = 80]
+  [ingestion_concurrency_target_flushes_per_shard: <int> | default = 40]
 
   # The estimated number of bytes a sample has at time of ingestion. This value
   # is used to estimate the timeseries without decompressing them. Only use this

operations/mimir-tests/test-ingest-storage-auto-client-rack-generated.yaml

@@ -1949,14 +1949,8 @@ spec:
         - -ingest-storage.kafka.auto-create-topic-default-partitions=1000
         - -ingest-storage.kafka.client-rack=zone-a
         - -ingest-storage.kafka.consumer-group-offset-commit-interval=5s
-        - -ingest-storage.kafka.fetch-concurrency-max=12
-        - -ingest-storage.kafka.ingestion-concurrency-batch-size=150
         - -ingest-storage.kafka.ingestion-concurrency-estimated-bytes-per-sample=500
-        - -ingest-storage.kafka.ingestion-concurrency-max=8
-        - -ingest-storage.kafka.ingestion-concurrency-queue-capacity=3
-        - -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard=40
         - -ingest-storage.kafka.last-produced-offset-poll-interval=500ms
-        - -ingest-storage.kafka.max-buffered-bytes=1000000000
         - -ingest-storage.kafka.topic=ingest
         - -ingester.max-global-metadata-per-metric=10
         - -ingester.max-global-metadata-per-user=30000
@@ -2096,14 +2090,8 @@ spec:
         - -ingest-storage.kafka.auto-create-topic-default-partitions=1000
         - -ingest-storage.kafka.client-rack=zone-b
         - -ingest-storage.kafka.consumer-group-offset-commit-interval=5s
-        - -ingest-storage.kafka.fetch-concurrency-max=12
-        - -ingest-storage.kafka.ingestion-concurrency-batch-size=150
         - -ingest-storage.kafka.ingestion-concurrency-estimated-bytes-per-sample=500
-        - -ingest-storage.kafka.ingestion-concurrency-max=8
-        - -ingest-storage.kafka.ingestion-concurrency-queue-capacity=3
-        - -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard=40
         - -ingest-storage.kafka.last-produced-offset-poll-interval=500ms
-        - -ingest-storage.kafka.max-buffered-bytes=1000000000
         - -ingest-storage.kafka.topic=ingest
         - -ingester.max-global-metadata-per-metric=10
         - -ingester.max-global-metadata-per-user=30000
@@ -2237,14 +2225,8 @@ spec:
         - -ingest-storage.kafka.auto-create-topic-default-partitions=1000
         - -ingest-storage.kafka.client-rack=zone-c
         - -ingest-storage.kafka.consumer-group-offset-commit-interval=5s
-        - -ingest-storage.kafka.fetch-concurrency-max=12
-        - -ingest-storage.kafka.ingestion-concurrency-batch-size=150
         - -ingest-storage.kafka.ingestion-concurrency-estimated-bytes-per-sample=500
-        - -ingest-storage.kafka.ingestion-concurrency-max=8
-        - -ingest-storage.kafka.ingestion-concurrency-queue-capacity=3
-        - -ingest-storage.kafka.ingestion-concurrency-target-flushes-per-shard=40
         - -ingest-storage.kafka.last-produced-offset-poll-interval=500ms
-        - -ingest-storage.kafka.max-buffered-bytes=1000000000
         - -ingest-storage.kafka.topic=ingest
         - -ingester.max-global-metadata-per-metric=10
         - -ingester.max-global-metadata-per-user=30000