Out of Memory Issue

[Jake-Chae]

Hello,
I am using a single Jigasi instance to provide STT (speech-to-text) service for audio streams coming from a total of 6 different servers.
The deployment is Docker-based, and both the Jigasi container and the STT server container are currently configured with a memory limit of 16 GiB.
However, during usage we are seeing the following error logs, after which the Jigasi server stops:

2025-11-11 08:57:07.070 INFO: [308] [ctx=1762818438257966354320 room=kdt-design-7th_bfbhcp@muc.liveroom.likelion.net] Participant.createStreamingSessionWithRetry#649: STT connection successfully created for participant 5d6236cb after 1 attempt(s)2025-11-11 08:57:10.197 INFO: [273] net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl$MemberListener.joined: kdt-backendj-19th_qhwzp8@muc.liveroom.likelion.net/67af1f35 has joined the kdt-backendj-19th_qhwzp8@muc.liveroom.likelion.net chat room.[1066.285s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 256k, guardsize: 0k, detached.[1066.286s][warning][os,thread] Failed to start the native thread for java.lang.Thread "WebSocket@1015685120-8400"[1066.286s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 256k, guardsize: 0k, detached.[1066.287s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-18-thread-2"2025-11-11 08:57:10.910 SEVERE: [520] net.java.sip.communicator.util.UtilActivator.uncaughtException: An uncaught exception occurred in thread=Thread[pool-18-thread-1,5,main], and message was: unable to create native thread: possibly out of memory or process/resource limits reachedjava.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Thread.java:809) at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1013) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)[1066.783s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 256k, guardsize: 0k, detached.[1066.783s][warning][os,thread] Failed to start the native thread for java.lang.Thread "WebSocket@1828296879-8403"[1066.783s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 256k, guardsize: 0k, detached.
From what I understand, this looks like OutOfMemoryError: unable to create native thread, where the JVM is no longer able to create new native threads (either due to memory exhaustion or process/thread limits).
Below is an excerpt from my current docker-compose.yml for Jigasi:

      - GC_CLIENT_EMAIL
      - GC_CLIENT_ID
      - GC_CLIENT_CERT_URL
      - SHUTDOWN_REST_ENABLED
      - SENTRY_DSN="${JIGASI_SENTRY_DSN:-0}"
      - SENTRY_ENVIRONMENT
      - SENTRY_RELEASE
      - TZ
      - JIGASI_MAX_MEMORY=1024m
      - JAVA_TOOL_OPTIONS=-Xss256k

    pids_limit: 8192
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
    mem_limit: 2g

    networks:
      meet.jitsi:

networks:
  meet.jitsi:
    external: true

The following part:

- JIGASI_MAX_MEMORY=1024m
- JAVA_TOOL_OPTIONS=-Xss256k

    pids_limit: 8192
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
    mem_limit: 2g

was added based on advice I received from GPT when I asked about the error logs.
I would greatly appreciate it if you could share any guidance or best practices related to:
Whether these settings (JIGASI_MAX_MEMORY, JAVA_TOOL_OPTIONS, pids_limit, ulimits, mem_limit) look reasonable for a single Jigasi instance handling STT for 6 servers.
Whether this unable to create native thread error is a known issue or pattern with Jigasi in high-load STT scenarios.
Any recommended configuration or tuning (JVM options, Docker limits, or Jigasi-specific settings) to avoid this type of failure.
Thank you very much for your help.

[damencho]

Not sure how you start jigasi, but the default scripts set max memory usage to 3Gb

jigasi/jigasi.sh

Line 23 in eac168d

if [ -z "$JIGASI_MAX_MEMORY" ]; then JIGASI_MAX_MEMORY=3072m; fi

so it is not good idea to limit the container to 2, you should set it to at least 3.5 to accommodate and the overhead of anything running around the jigasi process l.

[damencho]

Do you see in the process list that your setting for JIGASI_MAX_MEMORY is taken into account?

[damencho]

I think limiting it to 1Gb is too low. For how many concurrent users is this?

[Jake-Chae]

Thank you for your response.
I have confirmed that the value I configured (2) was correctly applied, and I have now updated the container memory limit to 3.5 GB.
Regarding your question about the number of concurrent users:
does this refer to the total number of participants in all conferences, or only to the number of participants who are actively speaking?
In my case, the total number of participants is about 230, and the number of actively speaking users is about 7.
The number of users is expected to increase, so any guidance on appropriate memory settings depending on the number of users would be very helpful.
Additionally, I would appreciate it if you could let me know whether the following docker-compose.yml settings are valid or reasonable for Jigasi:

• JIGASI_MAX_MEMORY=1024m
• JAVA_TOOL_OPTIONS=-Xss256k

pids_limit: 8192
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
mem_limit: 2g

Thank you again for your help.

[damencho]

I would keep the defaults of 3GB. This is how we use it in production and I have no experience with different values.

[Jake-Chae]

thank you for your help.
I’ll keep adjusting and testing the limit values on my side to see how the system behaves.

[rgumirov00]

Hello, I have the same problem. We use Jigasi to transcribe text for my service via WebSocket. MY SERVICE SEND 403 ERROR!!!. Now I'm getting an OOM (Out of Memory) exception. Heap dumps look good, with no memory leaks - the heap is about 170 MB. But we have 25k threads... All the memory is being allocated for them.

This is a development Jigasi instance, not production, of course. We waited 24 hours, but the number of threads remains the same. Memory same

In our case, one Jitsi call creates about 120 threads, and they are not being freed afterward.

[damencho]

How much is the memory on the machine running jigasi?

[rgumirov00]

1024MB, but its dev, not production
btw, 25k threads are not being freed afterward.

[damencho]

Can you send me the heapdump to damencho at jitsit dor org.

1024MB, but its dev, not production

By default jigasi is using up to 3GB of RAM, you ether need to adjust that so it matches the physical ram or have 3GB of free ram that can be used by jigasi. Second one is prefferred as having less will cause more garbage collection and more CPU usage and quality my suffer.

[rgumirov00]

Heap dump is ok, how can i send you heap and thread dumps?
Heap about 200MB, but jigasi use 3-4GB

[damencho]

Use some google drive or dropbox and send me a link via email.

[rgumirov00]

give me y email, please

[damencho]

damencho at jitsi dot org

[rgumirov00]

How much is the memory on the machine running jigasi?

sorry, i am wrong, machine has 8GB memory, but JIGASI_MAX_MEMOR = 1024MB

when i create 100-200 jitsi calls, jigasi use 3-4GB