VLInsert v1.40.0 memory usage increased by a lot

[clavinjune]

Is your question request related to a specific component?

VLInsert

Describe the question in detail

Just deployed VLInsert with newer version v1.35.0 -> v1.40.0, it seems the CPU is reduced, but the memory usage is significantly increased, we deployed VLInsert in cluster mode in Kubernetes. Is this the expected tradeoff?

N.B. we're still using v1.35.0 for vlselect and vlstorage

Key concepts docs

• VictoriaLogs key conepts - https://docs.victoriametrics.com/victorialogs/keyconcepts/
• LogsQL - https://docs.victoriametrics.com/victorialogs/logsql/

[func25]

Hey @clavinjune. Memory growth is expected, but it shouldn't get too high. What resources does the replica have, and what ingestion format are you using?

[clavinjune]

we're using

        resources:
          limits:
            cpu: "5"
            memory: 15Gi
          requests:
            cpu: "3"
            memory: 3Gi

we're both using the otel ingestion format, jsonline, and loki (faro) @func25

[clavinjune]

[vadimalekseev]

Hi @clavinjune, could you please collect the memory profiles for v1.35.0 and v1.40.0 using this documentation?
https://docs.victoriametrics.com/victorialogs/#profiling

This will allow us to more precisely identify the root cause of the issue.

[clavinjune]

sadly we have bumped the version to v1.40.0 so we don't have v1.35.0 anymore, I can get the memory profiles for v1.40.0, should I share it here?

[clavinjune]

it seems the most notable memory usage is in this function

[vadimalekseev]

should I share it here?

Yes, please attach the memory profile in your comment

[clavinjune]

mem.pprof.txt v1.40.0 I couldn't upload mem.pprof.txt so I renamed it from mem.pprof to mem.pprof.txt

[vadimalekseev]

@clavinjune, I couldn't reproduce the issue locally. Could you please check if the number of requests has increased, if there have been changes to resources (usage percentage might shift if you lowered memory limits), if the request size has changed, or if there are any other anomalies in graphs?

It would be great if you could provide a dashboard snapshot so that I can review it on my end.

[clement-gautier-cko]

I don't know if it's related but I also encountered "invalid memory address or nil pointer dereference" panic errors when switching to v1.40.0, v1.39.0 is fine. (I also noticed a big memory usage increase, at least at startup)

trace:

Found 5 pods, using pod/vl-victoria-logs-cluster-vlinsert-cfdd746-l5bxk
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:12","msg":"build version: victoria-logs-20251205-181412-tags-v1.40.0-0-ga9e8c173f4"}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:13","msg":"command-line flags"}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:20","msg":"  -envflag.enable=\"true\""}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:20","msg":"  -http.shutdownDelay=\"15s\""}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:20","msg":"  -httpListenAddr=\":9481\""}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:20","msg":"  -loggerFormat=\"json\""}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/logger/flag.go:20","msg":"  -storageNode=\"vl-victoria-logs-cluster-vlstorage-0.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local
.:9491,vl-victoria-logs-cluster-vlstorage-1.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491,vl-victoria-logs-cluster-vlstorage-2.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491,vl-
victoria-logs-cluster-vlstorage-3.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491,vl-victoria-logs-cluster-vlstorage-4.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491,vl-victoria-l
ogs-cluster-vlstorage-5.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491\""}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaLogs/app/victoria-logs/main.go:43","msg":"starting VictoriaLogs at \"[:9481]\"..."}
{"ts":"2025-12-08T14:07:45.979Z","level":"info","caller":"VictoriaLogs/app/vlstorage/main.go:172","msg":"starting insert service for nodes [vl-victoria-logs-cluster-vlstorage-0.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.clu
ster.local.:9491 vl-victoria-logs-cluster-vlstorage-1.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491 vl-victoria-logs-cluster-vlstorage-2.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local
.:9491 vl-victoria-logs-cluster-vlstorage-3.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491 vl-victoria-logs-cluster-vlstorage-4.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491 vl-
victoria-logs-cluster-vlstorage-5.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491]"}
{"ts":"2025-12-08T14:07:45.980Z","level":"info","caller":"VictoriaLogs/app/vlstorage/main.go:175","msg":"initializing select service for nodes [vl-victoria-logs-cluster-vlstorage-0.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc
.cluster.local.:9491 vl-victoria-logs-cluster-vlstorage-1.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491 vl-victoria-logs-cluster-vlstorage-2.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.l
ocal.:9491 vl-victoria-logs-cluster-vlstorage-3.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491 vl-victoria-logs-cluster-vlstorage-4.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491
 vl-victoria-logs-cluster-vlstorage-5.vl-victoria-logs-cluster-vlstorage.victoria-metrics.svc.cluster.local.:9491]"}
{"ts":"2025-12-08T14:07:45.980Z","level":"info","caller":"VictoriaLogs/app/vlstorage/main.go:178","msg":"initialized all the network services"}
{"ts":"2025-12-08T14:07:45.980Z","level":"info","caller":"VictoriaLogs/app/victoria-logs/main.go:55","msg":"started VictoriaLogs in 0.000 seconds; see https://docs.victoriametrics.com/victorialogs/"}
{"ts":"2025-12-08T14:07:45.980Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:145","msg":"started server at http://0.0.0.0:9481/"}
{"ts":"2025-12-08T14:07:45.980Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:147","msg":"pprof handlers are exposed at http://0.0.0.0:9481/debug/pprof/"}
{"ts":"2025-12-08T14:07:46.357Z","level":"info","caller":"VictoriaMetrics@v1.131.0/lib/memory/memory.go:45","msg":"limiting caches to 1288490188 bytes, leaving 858993460 bytes to the OS according to -memory.allowedPercent=60, system mem
ory limit 2147483648 bytes"}
panic: runtime error: invalid memory address or nil pointer dereference

goroutine 101 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:309 +0x5b
panic({0xb9ee80?, 0x12ba940?})
        runtime/panic.go:783 +0x132
github.com/valyala/fastjson.(*Value).MarshalTo(0x419f54?, {0xc000311500?, 0xc4eb00?, 0x0?})
        github.com/valyala/fastjson@v1.6.4/parser.go:577 +0x1a
github.com/valyala/fastjson.(*Value).MarshalTo(0xc000347860, {0xc000311500?, 0xc, 0xbe78c0?})
        github.com/valyala/fastjson@v1.6.4/parser.go:588 +0x34a
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.(*fmtBuffer).encodeJSONValue(0xc0000bc258, 0x2?)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/fmt_buffer.go:76 +0x32
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeAnyValue({0xc0002ee2bf?, 0x14?, 0x97?}, 0xc0000bc270, 0xc0000bc258, {0xc0002ee2c5, 0x5})
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:371 +0x899
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeKeyValue({0xc0002ee2bd, 0xd, 0x1843}, 0xc0000bc270, 0xc0000bc258, {0x0, 0x0})
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:304 +0x34a
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeLogRecord({0xc0002ee1fd?, 0xc0002ee1fa?, 0x220?}, 0xc0000bc270, 0xc0000bc258)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:235 +0x2ad
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeScopeLogs({0xc0002ee006, 0x414, 0x1afa}, 0xc0000bc270, 0xc0001e1308)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:147 +0x1e5
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeResourceLogs({0xc0002ee003, 0x582, 0x1afd}, 0xc0001e1308)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:87 +0x3d5
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.decodeLogsData({0xc0002ee000?, 0x5?, 0x0?}, 0xc0001e1308)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/pb.go:39 +0xa7
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.pushProtobufRequest({0xc0002ee000?, 0x585, 0x16?}, {0xe84448?, 0xc00036ad80?}, {0x0?, 0xc0000e2b08?, 0x1b00?}, 0x65?)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/opentelemetry.go:89 +0x97
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.handleProtobuf.func1({0xc0002ee000, 0x585, 0x1b00})
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/opentelemetry.go:55 +0x8d
github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/protoparserutil.ReadUncompressedData({0x7f157c142e48?, 0xc00037e280?}, {0xc000015fbc, 0x4}, 0xc0001a09c0, 0xc0001e1548)
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/protoparser/protoparserutil/compress_reader.go:70 +0x3c2
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.handleProtobuf(0xc00037ca00, {0xe85ee0, 0xc0000bd470})
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/opentelemetry.go:52 +0x1d5
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry.RequestHandler({0xc0000a7355?, 0xc00038e12b?}, {0xe85ee0, 0xc0000bd470}, 0xc00037ca00)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/opentelemetry/opentelemetry.go:30 +0xfc
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert.insertHandler({0xe85ee0, 0xc0000bd470}, 0xc00037ca00, {0xc0000a7355, 0x1d})
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/main.go:81 +0x349
github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert.RequestHandler({0xe85ee0, 0xc0000bd470}, 0xc00037ca00)
        github.com/VictoriaMetrics/VictoriaLogs/app/vlinsert/main.go:46 +0xca
main.requestHandler({0xe85ee0, 0xc0000bd470}, 0xc00037ca00)
        github.com/VictoriaMetrics/VictoriaLogs/app/victoria-logs/main.go:93 +0x17d
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.builtinRoutesHandler(0xc0000b52a0, 0xc00037ca00, {0xe85ee0, 0xc0000bd470}, 0xdc0d68)
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:450 +0x7bc
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.serveWithListener.func2({0xe85ee0?, 0xc0000bd470?}, 0xc6cd77?)
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:184 +0x31
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper({0xe86120, 0xc0000acaa0}, 0xc00037ca00, 0xc000012be8)
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:357 +0x79e
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.serveWithListener.func3({0xe86120?, 0xc0000acaa0?}, 0xc0000acaa0?)
        github.com/VictoriaMetrics/VictoriaMetrics@v1.131.0/lib/httpserver/httpserver.go:188 +0x25
net/http.HandlerFunc.ServeHTTP(0x12be960?, {0xe86120?, 0xc0000acaa0?}, 0x4?)
        net/http/server.go:2322 +0x29
github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0xe85e20, 0xc00038c5a0}, 0xc00037ca00)
        github.com/klauspost/compress@v1.18.2/gzhttp/compress.go:514 +0x5df
net/http.HandlerFunc.ServeHTTP(0x41a985?, {0xe85e20?, 0xc00038c5a0?}, 0xc00038c501?)
        net/http/server.go:2322 +0x29
net/http.serverHandler.ServeHTTP({0xe83ac8?}, {0xe85e20?, 0xc00038c5a0?}, 0x1?)
        net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc00035aab0, {0xe875b0, 0xc000378b70})
        net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 45
        net/http/server.go:3493 +0x485

[clavinjune]

@vadimalekseev nope, the number of request stays

#869 (comment) as in this metrics I tried to downgrade the version and bump it again, it seems the memory constantly high, even currently (we're using v1.40.0 in vlselect, vlinsert, and vlstorage) the number still high

another question, is it safe to downgrade vlstorage from v1.40.0 to v1.39.0?