|
| 1 | +// Module is included in the following assemblies: |
| 2 | +// * logging/cluster-logging-loki.adoc |
| 3 | +// * observability/logging/log_collection_forwarding/log-forwarding.adoc |
| 4 | +// * observability/logging/troubleshooting/log-forwarding-troubleshooting.adoc |
| 5 | + |
| 6 | +:_mod-docs-content-type: PROCEDURE |
| 7 | +[id="log6x-loki-rate-limit-errors_{context}"] |
| 8 | += Troubleshooting Loki rate limit errors |
| 9 | + |
| 10 | +If the Log Forwarder API forwards a large block of messages that exceeds the rate limit to Loki, Loki generates rate limit (`429`) errors. |
| 11 | + |
| 12 | +These errors can occur during normal operation. For example, when adding the {logging} to a cluster that already has some logs, rate limit errors might occur while the {logging} tries to ingest all of the existing log entries. In this case, if the rate of addition of new logs is less than the total rate limit, the historical data is eventually ingested, and the rate limit errors are resolved without requiring user intervention. |
| 13 | + |
| 14 | +In cases where the rate limit errors continue to occur, you can fix the issue by modifying the `LokiStack` custom resource (CR). |
| 15 | + |
| 16 | +[IMPORTANT] |
| 17 | +==== |
| 18 | +The `LokiStack` CR is not available on Grafana-hosted Loki. This topic does not apply to Grafana-hosted Loki servers. |
| 19 | +==== |
| 20 | + |
| 21 | +.Conditions |
| 22 | + |
| 23 | +* The Log Forwarder API is configured to forward logs to Loki. |
| 24 | +
|
| 25 | +* Your system sends a block of messages that is larger than 2 MB to Loki. For example: |
| 26 | ++ |
| 27 | +[source,text] |
| 28 | +---- |
| 29 | +"values":[["1630410392689800468","{\"kind\":\"Event\",\"apiVersion\":\ |
| 30 | +....... |
| 31 | +...... |
| 32 | +...... |
| 33 | +...... |
| 34 | +\"received_at\":\"2021-08-31T11:46:32.800278+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-31T11:46:32.799692+00:00\",\"viaq_index_name\":\"audit-write\",\"viaq_msg_id\":\"MzFjYjJkZjItNjY0MC00YWU4LWIwMTEtNGNmM2E5ZmViMGU4\",\"log_type\":\"audit\"}"]]}]} |
| 35 | +---- |
| 36 | +
|
| 37 | +* After you enter `oc logs -n openshift-logging -l component=collector`, the collector logs in your cluster show a line containing one of the following error messages: |
| 38 | ++ |
| 39 | +[source,text] |
| 40 | +---- |
| 41 | +429 Too Many Requests Ingestion rate limit exceeded |
| 42 | +---- |
| 43 | ++ |
| 44 | +.Example Vector error message |
| 45 | +[source,text] |
| 46 | +---- |
| 47 | +2023-08-25T16:08:49.301780Z WARN sink{component_kind="sink" component_id=default_loki_infra component_type=loki component_name=default_loki_infra}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true |
| 48 | +---- |
| 49 | ++ |
| 50 | +.Example Loki ingester error message |
| 51 | +[source,text] |
| 52 | +---- |
| 53 | +level=warn ts=2023-08-30T14:57:34.155592243Z caller=grpc_logging.go:43 duration=1.434942ms method=/logproto.Pusher/Push err="rpc error: code = Code(429) desc = entry with timestamp 2023-08-30 14:57:32.012778399 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream |
| 54 | +---- |
| 55 | +
|
| 56 | +.Procedure |
| 57 | +
|
| 58 | +* Update the `ingestionBurstSize` and `ingestionRate` fields in the `LokiStack` CR: |
| 59 | ++ |
| 60 | +[source,yaml] |
| 61 | +---- |
| 62 | +apiVersion: loki.grafana.com/v1 |
| 63 | +kind: LokiStack |
| 64 | +metadata: |
| 65 | + name: logging-loki |
| 66 | + namespace: openshift-logging |
| 67 | +spec: |
| 68 | + limits: |
| 69 | + global: |
| 70 | + ingestion: |
| 71 | + ingestionBurstSize: 16 # <1> |
| 72 | + ingestionRate: 8 # <2> |
| 73 | +# ... |
| 74 | +---- |
| 75 | +<1> The `ingestionBurstSize` field defines the maximum local rate-limited sample size per distributor replica in MB. This value is a hard limit. Set this value to at least the maximum logs size expected in a single push request. Single requests that are larger than the `ingestionBurstSize` value are not permitted. |
| 76 | +<2> The `ingestionRate` field is a soft limit on the maximum amount of ingested samples per second in MB. Rate limit errors occur if the rate of logs exceeds the limit, but the collector retries sending the logs. As long as the total average is lower than the limit, the system recovers and errors are resolved without user intervention. |
0 commit comments