CI: the whole-repository build hangs after its last task, holding a runner until the 6 hour kill

[codeconsole]

The job named Neo4j Functional Tests (Java 25, indy=false) stops producing output after its last task and never finishes, holding a runner until GitHub kills it at the six hour ceiling. Seen twice. All tests in the job have passed by the time it stops.

Despite the name, nothing about this is Neo4j. That job runs

./gradlew bootJar check --continue --rerun-tasks -PgrailsIndy=... -PonlyNeo4jTests -PskipCodeStyle

which is a whole-repository build; -PonlyNeo4jTests gates only the functional example applications, so the module tests of the whole repository run in it as well.

Occurrences

Date (UTC)	Branch	Job	Outcome
2026-08-22 18:34	fix/embedded-mongo-replica-set	attempt 1	ran 18:34:00 → 00:34:16, exactly 6h00m, cancelled by the platform
2026-08-22 (retry)	same	attempt 2	hung again, cancelled manually after 2h18m
2026-08-24 22:49	ci/release-drafter-pr-concurrency (#16213)	job	hung 4h44m, cancelled manually

Where it stops

Identical in both, to the line. The last thing in the log is the AOT example application starting and shutting down, then its check task, and then nothing:

INFO 50196 --- [main] o.s.boot.tomcat.TomcatWebServer : Tomcat started on port 46097 (http)
INFO 50196 --- [main] aot.Application                 : Started Application in 7.911 seconds
Grails application running at http://localhost:46097 in environment: production
INFO 50196 --- [main] o.s.boot.tomcat.GracefulShutdown: Commencing graceful shutdown
INFO 50196 --- [tomcat-shutdown] o.s.boot.tomcat.GracefulShutdown: Graceful shutdown complete

> Task :grails-test-examples-aot:check
   ← 219 minutes of silence →
##[error]The operation was canceled.

In a run that succeeds, :grails-test-examples-aot:check is also the last task, and the next lines follow immediately:

> Task :grails-test-examples-aot:check
gradle/actions: Writing build results to /home/runner/work/_temp/.gradle-actions/build-results/...json
[Incubating] Problems report is available at: ...
BUILD SUCCESSFUL in 1h 1m 30s

So every task has completed and the build stops before Gradle reports its result. No test is running at that moment; the AOT application, which logs its own pid, has already exited.

Processes alive when the runner cleaned up

Run	orphan processes	of which mongod
2026-08-22	22	3
2026-08-24	18	0

The three mongods in the first were embedded MongoDB servers left running by specifications that relied on the JVM shutdown hook to reap them. That was a genuine leak and it has since been fixed (8dca5f2f, in #16192) - the second run shows none. It did not fix this hang, which is the reason for this issue: the leak and the hang are separate problems, and only the leak is addressed.

None of the java pids in either orphan list appears anywhere else in the log, so they are the Gradle daemon and its test workers rather than application JVMs. Gradle keeps idle workers for reuse during a build and stops them when the build ends, so their presence at the kill is consistent with a build that never ended and does not by itself say they caused it.

What is not known

What the build is doing during the silence. Two data points, no thread dump, and no reproduction: the same job passes on the same branches at other times, taking 40-65 minutes, which is its normal range on 8.0.x.

One thing worth noting rather than concluding from: develocity.apache.org returns 403 Forbidden on cache stores throughout both hung runs

java.lang.RuntimeException: Storing entry at 'https://develocity.apache.org/cache/...' response status 403: Forbidden
    at com.gradle.develocity.agent.gradle.internal.a.c.store(SourceFile:69)

and on 2026-08-24 a different job failed outright because the Develocity plugin could not be resolved at all. The build's post-task phase talks to that endpoint. That makes it a candidate, not an explanation.

Suggested next step

A thread dump would end the guessing rather than add to it. A step on this job that waits N minutes and then runs jstack against the Gradle daemon and its workers, printing the result, would name the blocked thread on the next occurrence. Something along the lines of:

- name: "Dump threads if the build is still running"
  if: always()
  run: |
    for pid in $(jps -q); do
      echo "=== $pid"
      jstack "$pid" || true
    done

run from a background watchdog started before the build, firing once the build passes its normal ceiling.

Other CI flakes tracked separately

• Flaky: AsyncPromiseSpec times out waiting for a response the endpoint cannot have taken that long to produce #16219 - AsyncPromiseSpec times out waiting for a response. It surfaces in the same kind of whole-repository build as this hang, and the runs of both carry the same develocity.apache.org … 403: Forbidden noise. Whether that is a common cause or a common environment is not established.
• Flaky: UserControllerSpec > User list in grails-test-examples/scaffolding submits the login form without a session cookie #16217 - the scaffolding application's login form is submitted without a session cookie. No known relationship to this one.