Merge pull request #445 from Red5/perf/rtmp-thread-oversubscription

Perf/rtmp thread oversubscription

Author: mondain

RTMP_2CORE_TUNING_SUMMARY.md

@@ -0,0 +1,100 @@
+# Red5 Low-Latency Tuning on Constrained (2-Core) Deployments — Executive Summary
+
+## Situation
+
+A prior optimization pass (`perf/rtmp-ingest-cpu`, merged as #444) reduced RTMP relay **average CPU ~12%** and **allocation ~39%**. QA/DevOps reported the build felt **no smoother than 2.0.34** on **2-core, CPU-limited container** deployments, where the metric that matters is **latency / smoothness (jitter, glitches)**.
+
+## Key Finding
+
+The earlier work optimized the wrong axis for this symptom. **Average CPU and garbage are not what cause streaming jitter on a CPU-capped container.** Jitter there is driven by *tail* events — CPU-quota throttling, thread over-subscription, and GC scheduling — none of which changed between 2.0.34 and the new build. The improvements were real but moved the median (p50), not the p99 that QA observes.
+
+Three drivers were identified and measured (Docker, cgroup v2, 2 cores, 15 publishers + 30 subscribers):
+
+| Driver | Evidence | Status |
+|---|---|---|
+| **Thread over-subscription** | One dedicated platform thread per connection (60 connections → 132 OS threads) plus a soft thread leak | **Fixed** — moved to virtual threads (132 → 69 OS threads, leak gone) |
+| **Wrong GC for few cores** | Legacy ZGC's always-on concurrent threads steal from the 2 app cores | **Fixed** — GC now selected by core count |
+| **CPU-quota (CFS) throttling** | A hard fractional CPU quota freezes all threads when the per-period budget is spent | **Mitigated + deployment guidance** |
+| **JVM core mis-detection** | CPU *requests*/shares without a hard *limit* make the JVM size every pool for host cores | **Deployment guidance** |
+
+## What Changed (branch `perf/rtmp-thread-oversubscription`)
+
+1. **Connection receive threads → virtual threads.** Eliminated one platform thread per connection and a never-shutdown executor leak. Ordering preserved. *Result: ~48% fewer OS threads under load.*
+2. **Resource-aware GC selection** in `red5.sh` (≤4 cores → ParallelGC, else G1). *Result: ~2.3× less quota throttling than ZGC on 2 cores, ~7 ms worst-case pause.*
+3. **Packaging fixes** so the distribution boots from scratch: restored `jcodec` + `commons-io` (dropped by a wildcard dependency exclusion) and added `json-smart` for the chat demo webapp.
+
+### Measured impact (2-core container, identical 60-connection load)
+
+| Metric | Before (2.0.34) | After |
+|---|---|---|
+| OS threads | 132 | **69** |
+| Per-connection platform threads | 60 | **0** |
+| GC quota-throttled time / 30 s | 617 ms (ZGC) | **266 ms** (Parallel) |
+| Worst GC pause | n/a (concurrent) | **6.8 ms** |
+| CPU-quota throttling with whole-core pinning | n/a | **0 ms** |
+
+> Honest caveat: the thread fix reduces memory and scheduling overhead but **does not by itself remove CPU-quota throttling** — throttling is bound by total CPU demand. The largest jitter reductions on a capped container come from the **GC change** and, above all, **how CPU is allocated** (see below).
+
+## Recommended Startup Settings
+
+The startup script now **auto-selects the GC by detected core count**, so most deployments need no GC flags. The settings below cover the cases that still matter.
+
+### Without Docker (bare metal / VM)
+
+`red5.sh` detects cores via `nproc` and configures itself:
+
+- **2-core VM:** automatically uses `ParallelGC` — no action needed.
+- **Larger server:** automatically uses `G1` with a 200 ms pause goal.
+
+Adjust only the heap for the box, via the `JVM_OPTS` environment variable (overrides the script default):
+
+```bash
+# Example: 2-core VM, give the relay a 2 GB heap, keep the auto-selected ParallelGC
+export JVM_OPTS="-XX:+UseParallelGC -Xms512m -Xmx2g -XX:ReservedCodeCacheSize=32m"
+./red5.sh
+```
+
+If the host is large but Red5 is meant to use only some cores, pin it and tell the JVM:
+
+```bash
+taskset -c 0,1 env JVM_OPTS="-XX:+UseParallelGC -XX:ActiveProcessorCount=2 -Xms512m -Xmx2g" ./red5.sh
+```
+
+### With Docker
+
+**Prefer whole-core pinning over a fractional quota** — it eliminates CPU-quota throttling entirely (measured 0 ms vs ~300 ms / 30 s):
+
+```bash
+# Recommended: pin to whole cores (no CFS quota → no throttling stalls)
+docker run -d --cpuset-cpus="0,1" --memory=2g \
+  -p 1935:1935 -p 5080:5080 \
+  mondain/red5:latest
+# red5.sh sees 2 cores and selects ParallelGC automatically.
+```
+
+If you must use a fractional quota (`--cpus`), make the JVM size its pools to the limit and expect some throttling:
+
+```bash
+docker run -d --cpus=2 --memory=2g \
+  -e JVM_OPTS="-XX:+UseParallelGC -XX:ActiveProcessorCount=2 -Xms512m -Xmx2g -XX:ReservedCodeCacheSize=32m" \
+  -p 1935:1935 -p 5080:5080 \
+  mondain/red5:latest
+```
+
+### With Kubernetes
+
+- Use **integer CPU limits** (e.g. `limits.cpu: "2"`) and enable the **static CPU Manager policy** (`--cpu-manager-policy=static`) so pods get exclusive whole cores — this is the k8s equivalent of `--cpuset-cpus` and avoids quota throttling.
+- Avoid setting only `requests.cpu` with no `limits.cpu`: the JVM then reads the **host** core count and over-sizes GC, JIT, and thread pools.
+- If fractional limits are unavoidable, add `-XX:ActiveProcessorCount=<limit>` via `JVM_OPTS`.
+
+```yaml
+resources:
+  requests: { cpu: "2", memory: "2Gi" }
+  limits:   { cpu: "2", memory: "2Gi" }   # integer + static CPU manager → exclusive cores
+```
+
+## Bottom Line
+
+- The relay code and packaging are now correct and lighter on threads.
+- For **smoothness on capped deployments, the highest-leverage change is how CPU is allocated**: pin whole cores (cpuset / k8s static CPU manager) instead of a fractional quota, and ensure a hard limit so the JVM sizes itself correctly.
+- Validate with **p99 frame jitter and cgroup `cpu.stat` (`nr_throttled` / `throttled_usec`)** — not average CPU — comparing the same load against 2.0.34.

client/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>org.red5</groupId>
         <artifactId>red5-parent</artifactId>
-        <version>2.0.37</version>
+        <version>2.0.38</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>red5-client</artifactId>

client/src/main/java/org/red5/client/Red5Client.java

@@ -18,7 +18,7 @@ public final class Red5Client {
     /**
      * Current server version with revision
      */
-    public static final String VERSION = "Red5 Client 2.0.37";
+    public static final String VERSION = "Red5 Client 2.0.38";
 
     /**
      * Create a new Red5Client object using the connection local to the current thread A bit of magic that lets you access the red5 scope

common/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>org.red5</groupId>
         <artifactId>red5-parent</artifactId>
-        <version>2.0.37</version>
+        <version>2.0.38</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>red5-server-common</artifactId>

common/src/main/java/org/red5/server/api/Red5.java

@@ -57,12 +57,12 @@ public final class Red5 {
     /**
      * Server version with revision
      */
-    public static final String VERSION = "Red5 Server 2.0.37";
+    public static final String VERSION = "Red5 Server 2.0.38";
 
     /**
      * Server version for fmsVer requests
      */
-    public static final String FMS_VERSION = "RED5/2,0,37,0";
+    public static final String FMS_VERSION = "RED5/2,0,38,0";
 
     /**
      * Server capabilities

common/src/main/java/org/red5/server/net/rtmp/RTMPConnection.java

@@ -379,9 +379,16 @@ public abstract class RTMPConnection extends BaseConnection implements IStreamCa
     protected ScheduledFuture<?> keepAliveTask;
 
     /**
-     * Executor for received RTMP messages.
-     */
-    protected transient ExecutorService receivedPacketExecutor = Executors.newSingleThreadExecutor();
+     * Executor for received RTMP messages. Uses a virtual-thread-per-task executor: exactly one
+     * long-lived loop task is submitted per connection (guarded by {@link #receivedPacketFuture}),
+     * so the previous {@code newSingleThreadExecutor()} pinned one platform thread per connection
+     * for the life of the connection and was never shut down (a soft platform-thread leak relying
+     * on GC finalization). The loop blocks in {@code receivedPacketQueue.poll(timeout)}, which is
+     * virtual-thread-aware, so an idle connection unmounts from its carrier and holds no platform
+     * thread; the virtual thread terminates when the loop exits on close. Per-connection ordering
+     * is preserved because there is still a single serial loop per connection.
+     */
+    protected transient ExecutorService receivedPacketExecutor = Executors.newVirtualThreadPerTaskExecutor();
 
     /**
      * Future which takes packets from the queue and passes them to the handler.

io/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>org.red5</groupId>
         <artifactId>red5-parent</artifactId>
-        <version>2.0.37</version>
+        <version>2.0.38</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>red5-io</artifactId>

pom.xml

@@ -24,7 +24,7 @@
     <name>Red5</name>
     <description>The Red5 server</description>
     <groupId>org.red5</groupId>
-    <version>2.0.37</version>
+    <version>2.0.38</version>
     <url>https://github.com/Red5/red5-server</url>
     <inceptionYear>2005</inceptionYear>
     <organization>
@@ -119,6 +119,7 @@
         <cglib.version>3.2.7</cglib.version>
         <xerces.version>2.12.1</xerces.version>
         <gson.version>2.13.2</gson.version>
+        <json-smart.version>2.5.2</json-smart.version>
     </properties>
     <modules>
         <module>io</module>
@@ -386,12 +387,15 @@
                 <groupId>org.red5</groupId>
                 <artifactId>red5-io</artifactId>
                 <version>${red5-io.version}</version>
-                <exclusions>
-                    <exclusion>
-                        <artifactId>*</artifactId>
-                        <groupId>*</groupId>
-                    </exclusion>
-                </exclusions>
+                <!--
+                    No wildcard exclusion here. A previous *:* exclusion stripped ALL of red5-io's
+                    transitive dependencies; most are shared with red5-server-common (and are version
+                    managed below, so they dedup harmlessly), but two are unique to red5-io and are
+                    required at runtime: org.jcodec:jcodec (MP4Reader / HVC1Box) and commons-io. The
+                    wildcard silently dropped both from the assembled lib/, so the server failed to
+                    boot with NoClassDefFoundError: org/jcodec/common/io/SeekableByteChannel once
+                    MP4Reader began touching jcodec classes.
+                -->
             </dependency>
             <dependency>
                 <groupId>org.red5</groupId>
@@ -518,6 +522,18 @@
                 <artifactId>gson</artifactId>
                 <version>${gson.version}</version>
             </dependency>
+            <!--
+                Required at runtime by the bundled chat demo webapp (org.red5.demos.chat.*, compiled
+                against net.minidev.json). It was previously pulled via an undefined ${json-smart.version}
+                property and silently dropped, leaving the chat webapp failing to deploy with
+                NoClassDefFoundError: net/minidev/json/parser/ParseException. Managed here and bundled
+                into lib/ via the server module so webapps see it on the shared classpath.
+            -->
+            <dependency>
+                <groupId>net.minidev</groupId>
+                <artifactId>json-smart</artifactId>
+                <version>${json-smart.version}</version>
+            </dependency>
             <dependency>
                 <groupId>org.hamcrest</groupId>
                 <artifactId>hamcrest</artifactId>

server/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>org.red5</groupId>
         <artifactId>red5-parent</artifactId>
-        <version>2.0.37</version>
+        <version>2.0.38</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>red5-server</artifactId>
@@ -140,6 +140,11 @@
             <groupId>com.google.code.gson</groupId>
             <artifactId>gson</artifactId>
         </dependency>
+        <!-- runtime dependency of the bundled chat demo webapp (net.minidev.json); see parent dependencyManagement -->
+        <dependency>
+            <groupId>net.minidev</groupId>
+            <artifactId>json-smart</artifactId>
+        </dependency>
         <dependency>
             <groupId>org.red5</groupId>
             <artifactId>red5-moq-pkgr</artifactId>

server/src/main/server/red5.sh

@@ -45,10 +45,38 @@ esac
 echo "Running on " $OS
 
 # JAVA options
-# ZGC collector https://wiki.openjdk.java.net/display/zgc
-# You can set JVM additional options here if you want
-if [ -z "$JVM_OPTS" ]; then 
-    JVM_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms256m -Xmx1g -XX:InitialCodeCacheSize=8m -XX:MaxGCPauseMillis=500 -XX:ReservedCodeCacheSize=32m"
+# You can set JVM additional options here if you want; defining JVM_OPTS in the environment
+# overrides everything below.
+#
+# Container/CPU-limited note: thread pools (RTMP io threads, scheduler, GC workers, virtual-thread
+# carriers) are sized from Runtime.availableProcessors(). The JVM only derives a reduced core count
+# from a hard CPU *limit* (cgroup quota / cpuset); CPU *requests*/shares alone leave it reading the
+# host core count and oversizing every pool. On a CPU-capped deployment either set a hard limit or
+# pin the count explicitly, e.g. add: -XX:ActiveProcessorCount=2
+#
+# CFS throttling note: a hard CPU *quota* (docker --cpus / k8s fractional limit) freezes ALL threads
+# when the per-period quota is exhausted, which shows up as streaming jitter. Pinning to whole cores
+# instead (docker --cpuset-cpus / k8s static CPU manager with integer limits) removes quota
+# throttling entirely. Measured here: 15 publishers + 30 subscribers on ~2 cores produced ~300ms of
+# throttled (frozen) time per 30s under --cpus=2, and 0ms under --cpuset-cpus=0,1.
+#
+# GC selection (resource-aware). Measured under the same 2-core load (1g heap), CFS throttled time
+# per 30s and worst stop-the-world pause:
+#   ZGC (old default) 617ms / 0ms STW   G1 657ms / 9.8ms   Parallel 266ms / 6.8ms   Serial 156ms / 45ms
+# On few cores the always-on concurrent collectors (ZGC, G1) steal from the app and throttle most;
+# ParallelGC gives the best balance (low throttling, ~7ms max pause). On larger multi-core hosts with
+# bigger heaps G1 is the better general-purpose choice (Parallel/Serial full-GC pauses scale with the
+# heap). So pick by detected core count; nproc honors a hard cpu limit / cpuset (but not bare shares).
+if [ -z "$JVM_OPTS" ]; then
+    CORES=$(nproc 2>/dev/null || echo 4)
+    if [ "$CORES" -le 4 ]; then
+        # CPU-limited / small container: low-overhead throughput collector, least CFS throttling
+        GC_OPTS="-XX:+UseParallelGC"
+    else
+        # larger multi-core host: general-purpose low-pause collector
+        GC_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200"
+    fi
+    JVM_OPTS="$GC_OPTS -Xms256m -Xmx1g -XX:InitialCodeCacheSize=8m -XX:ReservedCodeCacheSize=32m"
 fi
 # Set up security options
 SECURITY_OPTS="-Djava.security.debug=failure"
@@ -71,4 +99,4 @@ export RED5_CLASSPATH="${RED5_HOME}/red5-service.jar${P}${RED5_HOME}/conf${P}${C
 
 # start Red5
 echo "Starting Red5"
-exec "$JAVA" -Dred5.root="${RED5_HOME}" $JAVA_OPTS -cp "${RED5_CLASSPATH}" -noverify "$RED5_MAINCLASS" $RED5_OPTS
+exec "$JAVA" -Dred5.root="${RED5_HOME}" $JAVA_OPTS -cp "${RED5_CLASSPATH}" "$RED5_MAINCLASS" $RED5_OPTS