High load average after upgrading from v1.22.2 to v1.39.0

[mrpk1906]

Is your question request related to a specific component?

VictoriaLogs

Describe the question in detail

I'm running VictoriaLogs in Docker with a single node.

services:
  vlogs:
    image: docker.io/victoriametrics/victoria-logs:v1.39.0
    volumes:
      - /data/vlogs:/vlogs
    ports:
      - '9428:9428'
    command:
      - -storageDataPath=/vlogs
      - -retentionPeriod=90d
      - -retention.maxDiskSpaceUsageBytes=1970GiB
      - -search.maxQueryDuration=600s
      - -insert.maxLineSizeBytes=1MB
      - -defaultMsgValue=-

After upgrading the VictoriaLogs to v1.39.0, the load average is much higher (screenshot below)

The CPU utilization:

My question is, what changes between the 2 versions affected the load average?

Key concepts docs

• VictoriaLogs key conepts - https://docs.victoriametrics.com/victorialogs/keyconcepts/
• LogsQL - https://docs.victoriametrics.com/victorialogs/logsql/

[func25]

Hi, there have been many changes across these versions. I deployed both and wasn't able to reproduce the issue. Could you share your ingestion rate and which queries or alerts were running during the period you observed the load?

• Upgrading to v1.40.0 is recommended. If the load remains high afterward, please share a CPU profile following these instructions.

[mrpk1906]

Hi @func25

I've updated VictoriaLogs to v1.40.0.

Below is the ingestion rate and request rate

This VictoriaLogs instance is write-only, queries are rarely used.

Below are the metrics, I upgraded to v1.39.0 since 12/04.

Here is the load avg in the last 35d

I also attached the pprof below

cpu.pprof.zip

Thank you!

[func25]

Oh I mistook CPU load for CPU usage. There was a change in VictoriaMetrics/VictoriaMetrics@65251c8 to improve concurrent I/O. Could you compare the number of threads between the two versions? There is a panel in the dashboard that shows thread counts.

When the load jumps, it means many processes or threads are waiting for resources (usually I/O or locks), so the kernel counts them as "load." They do not consume additional CPU, so it may not be a big problem.

[mrpk1906]

I see,

The Goroutines and Thread metrics:

I've read the changelog and tried setting -fs.maxConcurrency=150, but the CPU load is still high. Let me try to reduce it to 64. Or do you have any suggestions to tune for a better IO schedule?

[func25]

I don't have a specific number in mind. If you rely on slow storage, keep the value high. Otherwise, feel free to dial it down.