MINOR Capture heap dump after OOM on CI #19031

mumrah · 2025-02-25T23:13:47Z

We have seen a few OOM errors on trunk lately. This patch adds the ability to capture a heap dump when this happens so we can better determine if the error was due to something in Gradle or within our own tests (like a memory leak).

mumrah · 2025-02-26T16:34:48Z

It looks like the UserQuotaTest is the likely culprit.

From https://github.com/apache/kafka/actions/runs/13523225366/job/37791507296

Gradle Test Run :core:test > Gradle Test Executor 38 > UserQuotaTest > testQuotaOverrideDelete(String, String) > testQuotaOverrideDelete(String, String).quorum=kraft.groupProtocol=consumer STARTED

> Task :storage:compileTestJava
Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not read message from '/127.0.0.1:50402'.

	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:99)
	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

and from https://github.com/apache/kafka/actions/runs/13534291834/job/37823471305

Gradle Test Run :core:test > Gradle Test Executor 37 > UserQuotaTest > testQuotaOverrideDelete(String, String) > testQuotaOverrideDelete(String, String).quorum=kraft.groupProtocol=consumer STARTED

> Task :storage:checkstyleMain
> Task :shell:checkstyleMain
Unexpected exception thrown.

org.gradle.internal.remote.internal.MessageIOException: Could not read message from '/127.0.0.1:38838'.
	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:99)
	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

It appears that the Gradle worker is trying to send results to the main process which causes a long GC pause which triggers this "GC overhead limit exceeded" error.

mumrah · 2025-02-26T18:13:09Z

build.gradle

@@ -54,7 +54,7 @@ ext {
  buildVersionFileName = "kafka-version.properties"

  defaultMaxHeapSize = "2g"
-  defaultJvmArgs = ["-Xss4m", "-XX:+UseParallelGC"]
+  defaultJvmArgs = ["-Xss4m", "-XX:+UseParallelGC", "-XX:-UseGCOverheadLimit"]


@ijuma WDYT about disabling this feature? From what I can tell, this will prevent a long GC pause from triggering an OOM. Instead, the build would likely just timeout (which it's doing anyways with the OOM happing in the Gradle worker).

As you said, the build is unlikely to succeed in either case. The GC overhead thing at least gives a hint that there is a memory leak or the heap is too small. Isn't that better than a timeout with no information?

mumrah · 2025-02-26T19:55:34Z

Seems to have reproduced here: https://github.com/apache/kafka/actions/runs/13550598471/job/37873138268?pr=19031

No activity for a while after

Wed, 26 Feb 2025 18:35:41 GMT > Task :streams:test-utils:copyDependantLibs
Wed, 26 Feb 2025 18:52:51 GMT > Task :streams:test-utils:jar
Wed, 26 Feb 2025 18:55:06 GMT > Task :connect:runtime:compileJava

This suggests that -XX:-UseGCOverheadLimit is working as expected. However, it also suggests that there is a real memory leak or something. This run included a larger heap of 3gb.

mumrah · 2025-03-03T23:02:37Z

I've not been able to reproduce the Gradle OOM that we're seeing on trunk, however I saw a different OOM over on my fork.

https://github.com/mumrah/kafka/actions/runs/13639578283/job/38126853058

2025-03-03T21:58:41.1436187Z Gradle Test Run :connect:runtime:test > Gradle Test Executor 71 > ConnectWorkerIntegrationTest > testPauseStopResume() STARTED
2025-03-03T21:58:43.8433931Z 
2025-03-03T21:58:43.8434407Z java.lang.OutOfMemoryError: Java heap space
2025-03-03T21:58:43.8434975Z Dumping heap to /home/runner/work/kafka/kafka/heap-dumps/java_pid471589.hprof ...
2025-03-03T21:58:46.0433855Z 
2025-03-03T21:58:46.0434629Z > Task :connect:runtime:test
2025-03-03T21:58:46.0435151Z

(Heap dump was uploaded here https://github.com/mumrah/kafka/actions/runs/13639578283/artifacts/2685005476)

This at least shows that the OOM arguments and heap dump archiving are working.

mumrah · 2025-03-03T23:17:04Z

.github/actions/run-gradle/action.yml

+        mkdir -p heap-dumps
+        HEAP_DUMP_DIR=$(readlink -f heap-dumps)
+        timeout ${TIMEOUT_MINUTES}m ./gradlew --continue --no-scan \
+        -Dorg.gradle.jvmargs="-Xmx4g -Xss4m -XX:+UseParallelGC -XX:+UseGCOverheadLimit -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$HEAP_DUMP_DIR" \


We need to pass the heap dump path directly to Gradle as well as to JUnit (inside build.gradle). That's why we have this apparent duplication.

mumrah · 2025-03-13T13:35:41Z

@chia7712 Even with the recent JUnit heap increase, we are seeing OOM on trunk https://github.com/apache/kafka/actions/runs/13807500150/job/38622104924#step:6:616

Can you take a look at this one?

chia7712 · 2025-03-13T15:44:47Z

Can you take a look at this one?

roger that

chia7712 · 2025-03-14T20:24:30Z

UserQuotaTest consumes over 1GB of memory on my local machine. I have opened https://issues.apache.org/jira/browse/KAFKA-18989 and https://issues.apache.org/jira/browse/KAFKA-18990 for memory usage optimization. Based on my local testing, merging the changes from those tickets reduces memory consumption from 1GB to ~500MB

However, the memory offered by Github action is 16GB. Our CI executes four workers, each running four tests in parallel. Consequently, if a PR happens to encounter a set of 'expensive' tests simultaneously, it may exceed the memory limit.

mumrah · 2025-03-17T20:37:24Z

@chia7712 thanks for investigating. The metric creation inside a loop sounds like the likely culprit.

mumrah · 2025-03-18T14:20:06Z

WDYT about merging this PR @chia7712? I think it would help us diagnose OOM much faster if we had a heap dump (which of course includes the thread dump in it)

chia7712

I agree collecting the log of OOM is a good idea. two small comments are left. PTAL

chia7712 · 2025-03-19T21:15:23Z

.github/actions/run-gradle/action.yml

+        mkdir -p heap-dumps
+        HEAP_DUMP_DIR=$(readlink -f heap-dumps)
+        timeout ${TIMEOUT_MINUTES}m ./gradlew --continue --no-scan \
+        -Dorg.gradle.jvmargs="-Xmx4g -Xss4m -XX:+UseParallelGC -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$HEAP_DUMP_DIR" \


maybe we can add comment to gradle.properties to remind us to keep the memory configs consistent?

chia7712 · 2025-03-19T21:15:59Z

.github/actions/run-gradle/action.yml

        -PcommitId=xxxxxxxxxxxxxxxx \
        $TEST_TASK
        exitcode="$?"
+        find heap-dumps


Is it used to debug? if so, maybe using ls can offer more useful output?

ignore the GC pause thing for now

4baadb0

github-actions bot added build Gradle build or GitHub Actions small Small PRs labels Feb 25, 2025

mumrah added 6 commits February 25, 2025 19:50

set the jvm arg in the right place

d1e4af0

increase worker heap size

641c0df

reduce heap to try to get a GC pause

76147b9

speed up testing a bit for now

95240c1

disable caching

a22eb2a

disable cache

f4121f1

mumrah added 3 commits February 26, 2025 11:35

increase JUnit heap

d6e1e56

empty

fe16ad0

only run flaky 17

75dfd65

mumrah commented Feb 26, 2025

View reviewed changes

mumrah added 2 commits February 26, 2025 13:14

really disable the build cache

ce1d399

try getting a heap dump

59f5244

mumrah added 4 commits February 26, 2025 17:47

empty

3c3bbdb

empty

7a056c9

empty

3d7b910

force an oom

ac2357b

github-actions bot added the kraft label Feb 27, 2025

mumrah added 8 commits February 26, 2025 21:55

checkstyle

5b5669b

checkstyle

b1d6a6e

always archive heap dump

9eddadd

dont retain heap dumps very long

3812530

find

dec43a7

wip

8f977fc

remove forced OOM

fc49738

Merge remote-tracking branch 'origin/trunk' into tmp-oom-in-flaky-test

98131ff

mumrah added 6 commits February 27, 2025 11:00

empty

d44ea8c

back to 2g heap

32b6815

add a bunch of runs

46a8dad

fix conflict

5d0dceb

keep trying

dfe417d

still trying

ef377e5

github-actions bot added the core Kafka Broker label Feb 28, 2025

mumrah added 7 commits February 28, 2025 14:56

lower count

dfa1359

remove ls

b8d1c46

empty

3d1bf90

empty

8f6c75c

empty

6829676

revert some things

9dcbe73

empty

603d6a0

Merge remote-tracking branch 'origin/trunk' into tmp-oom-in-flaky-test

b8f66a4

mumrah changed the title ~~WIP Investigate OOM~~ MINOR Capture heap dump after OOM on CI Mar 3, 2025

increase retention

ef070f3

mumrah requested review from ijuma and chia7712 March 3, 2025 23:07

mumrah commented Mar 3, 2025

View reviewed changes

mumrah added 3 commits March 7, 2025 10:29

remove redundant jvm arg

6d64a9b

Merge branch 'trunk-cached' into tmp-oom-in-flaky-test

a3e0227

Merge branch 'trunk-cached' into tmp-oom-in-flaky-test

70fdd70

chia7712 reviewed Mar 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR Capture heap dump after OOM on CI #19031

MINOR Capture heap dump after OOM on CI #19031

mumrah commented Feb 25, 2025 •

edited

Loading

mumrah commented Feb 26, 2025 •

edited

Loading

mumrah Feb 26, 2025

ijuma Feb 27, 2025

mumrah commented Feb 26, 2025

mumrah commented Mar 3, 2025

mumrah Mar 3, 2025

mumrah commented Mar 13, 2025 •

edited

Loading

chia7712 commented Mar 13, 2025

chia7712 commented Mar 14, 2025

mumrah commented Mar 17, 2025

mumrah commented Mar 18, 2025

chia7712 left a comment

chia7712 Mar 19, 2025

chia7712 Mar 19, 2025

MINOR Capture heap dump after OOM on CI #19031

Are you sure you want to change the base?

MINOR Capture heap dump after OOM on CI #19031

Conversation

mumrah commented Feb 25, 2025 • edited Loading

mumrah commented Feb 26, 2025 • edited Loading

mumrah Feb 26, 2025

Choose a reason for hiding this comment

ijuma Feb 27, 2025

Choose a reason for hiding this comment

mumrah commented Feb 26, 2025

mumrah commented Mar 3, 2025

mumrah Mar 3, 2025

Choose a reason for hiding this comment

mumrah commented Mar 13, 2025 • edited Loading

chia7712 commented Mar 13, 2025

chia7712 commented Mar 14, 2025

mumrah commented Mar 17, 2025

mumrah commented Mar 18, 2025

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 Mar 19, 2025

Choose a reason for hiding this comment

chia7712 Mar 19, 2025

Choose a reason for hiding this comment

mumrah commented Feb 25, 2025 •

edited

Loading

mumrah commented Feb 26, 2025 •

edited

Loading

mumrah commented Mar 13, 2025 •

edited

Loading