high memory usage by ekuiper v2.2.1

[ankit-4129]

Environment:

• eKuiper version (e.g. 1.3.0): 2.2.1
• Hardware configuration (e.g. lscpu):

Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             39 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      4
  On-line CPU(s) list:       0-3
Vendor ID:                   GenuineIntel
  Model name:                Intel Atom(R) x6425E Processor @ 2.00GHz
    CPU family:              6
    Model:                   150
    Thread(s) per core:      1
    Core(s) per socket:      4
    Socket(s):               1
    Stepping:                1
    CPU(s) scaling MHz:      90%
    CPU max MHz:             3000.0000
    CPU min MHz:             800.0000

• OS (e.g. cat /etc/os-release):

NAME="Oracle Linux Server"
VERSION="9.6"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Oracle Linux Server 9.6"

• Others:

What happened and what you expected to happen:
High memory usage by ekuiper with about 6000 rules.

2025-10-30T07:08:38.425199590+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="register rest endpoint for component prometheus" file="server/rest.go:235"
2025-10-30T07:08:38.425297557+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="start service pprof" file="server/server.go:278"
2025-10-30T07:08:38.425351159+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="start service prometheus" file="server/server.go:278"
2025-10-30T07:08:38.425398513+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="using ListenMode 'http'" file="server/server.go:76"
2025-10-30T07:08:38.425454337+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="Run pprof in 127.0.0.1:6060" file="server/pprof_init.go:45"
2025-10-30T07:08:38.425616008+00:00 stdout F time="2025-10-30T07:08:38Z" level=info msg="Serving kuiper (version - ) on port 20498, and restful api on http://0.0.0.0:59720." file="server/server.go:291"
2025-10-30T07:08:38.425616008+00:00 stdout F Serving kuiper (version - ) on port 20498, and restful api on http://0.0.0.0:59720.
2025-10-30T07:08:38.428911074+00:00 stderr F panic: listen tcp 0.0.0.0:59720: bind: address already in use
2025-10-30T07:08:38.428911074+00:00 stderr F 
2025-10-30T07:08:38.428911074+00:00 stderr F goroutine 163308 [running]:
2025-10-30T07:08:38.428911074+00:00 stderr F github.com/lf-edge/ekuiper/v2/internal/server.StartUp.func1()
2025-10-30T07:08:38.429024986+00:00 stderr F 	github.com/lf-edge/ekuiper/v2/internal/server/server.go:264 +0x12a
2025-10-30T07:08:38.429024986+00:00 stderr F created by github.com/lf-edge/ekuiper/v2/internal/server.StartUp in goroutine 1
2025-10-30T07:08:38.429024986+00:00 stderr F 	github.com/lf-edge/ekuiper/v2/internal/server/server.go:260 +0x143e

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

[ngjaying]

That's a lot of rules. What is the memory and your expectation? Could you run curl -o heap.pprof "http://localhost:6060/debug/pprof/heap" to fetch the memory heap profile for analyzing. Thanks.

[gaurangomar]

Our containers have a limit of ~800MB, we were expecting the memory to be under that limit.

pprof overview:

Here is the file attached:
curl_pprof_output_31_oct.zip

[ngjaying]

Each rule requires a specific memory allocation for its internal structures and state. The current consumption is a direct result of the high rule count. Maybe you can try to reduce the rule count.

[Yisaer]

The most memory usage cost by for each rule may be caused by bufferLength according to your heap profile.

You can lower each rule's bufferLength, eg: 256. The default value is 1024. @gaurangomar

[gaurangomar]

@ngjaying We have increased the container limit to 1.6 GB, for the same number for rules and streams, still the memory is increasing, Currently it's consuming 1.55 GB.

Can you help me understand if we can make some changes, or how much memory it should consume for 5800 rules.

[ngjaying]

Hi @gaurangomar,

Thanks for providing the heap profile. Based on the analysis, the memory usage is expected given the number of rules.

Root Cause

Each rule creates multiple operator nodes (source, decode, filter/project, sink, etc.), and each node allocates a buffered channel with the default bufferLength of 1024:

~5800 rules × ~4 nodes per rule × 1024 slots = ~23 million buffer slots

This explains why memory usage scales linearly with the number of rules.

Solution

You can significantly reduce memory by setting a lower bufferLength in each rule's options. For example:

{
  "id": "myRule",
  "sql": "SELECT * FROM stream",
  "options": {
    "bufferLength": 256
  }
}

Setting bufferLength: 256 (or even lower like 64 or 128 if your rules don't need large buffers) should reduce memory consumption by ~4-16x for channel allocations.

You can also set this globally via the etc/kuiper.yaml configuration:

basic:
  bufferLength: 256

Recommendation

For 5800+ rules on a resource-constrained environment:

• Start with bufferLength: 128 or 256
• Monitor for any "buffer full, drop message" warnings in logs
• Increase only if you see buffer overflow issues

Let us know if you have any questions or need further assistance!

[echils]

I also encountered the same problem. In my system, there are approximately 2803 rules, and each rule has a buffer length of 256. After running for less than one month, the actual memory usage reached as high as 11G.

heap-pprof.zip