gc: finish in-flight major before nested minor in MINO_GC_FULL

Root cause of the CI hang at transient-survives-gc-yield across all
four matrix entries: mino_gc_collect(MINO_GC_FULL) ran the nested
minor BEFORE finishing the in-flight major. The minor freed a YOUNG
header that was still on major's mark stack from a prior incremental
slice; the subsequent gc_force_finish_major chased the freed pointer
through gc_mark_child_push.

gc_tick_during_major (driver.c:141-147) documents the correct
ordering -- "finish-then-minor rather than nest" -- and honours it
for the auto-tick path. The public-API path simply had the calls
reversed. Auto-tick is the common case, so the bug stayed unnoticed
locally; CI's test ordering puts transient-survives-gc-yield (which
explicitly calls (gc!)) inside an in-flight major on every push,
making the hang deterministic.

Symptom shape varied with environment:

* GHA matrix: 8-min timeout in the Test step at the identical post-
  io_test point on macos-14, ubuntu-24.04, ubuntu-24.04-arm, and
  windows-2022. The v0.255.8 trace artifact pinned it to
  transient-survives-gc-yield.
* Local sans sanitizer: intermittent SIGSEGV in tiny_free_no_lock
  (sweep double-free) or error[MTY001] "inc expects a number"
  (corrupted value via the reused freed header).
* Local under ASan: heap-use-after-free at gc_mark_child_push
  driver.c:414, freer site gc_minor_collect minor.c:472, both
  inside the same mino_gc_collect call.

Fix: reorder MINO_GC_FULL to finish the major BEFORE the minor.
MINO_GC_MINOR and MINO_GC_MAJOR unchanged.

Regression: tests/gc_test.clj gains gc-bang-during-incremental-major.
Warms up enough OLDs to start an incremental major, then calls (gc!)
in the transient shape that surfaced the bug. Fails under the old
ordering, passes under the fix.

Verified: tests/run.clj 1274/4557 green; mino_asan 1274/4557 green;
release-gate 17/17 green; 5x consecutive reproducer runs all green
with no flakiness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: leifericf

CHANGELOG.md

@@ -1,5 +1,55 @@
 # Changelog
 
+## v0.255.9 — Fix: `(gc!)` During In-Flight Major Mark Use-After-Free
+
+Root cause of the v0.255.6 / .7 / .8 CI hang: `mino_gc_collect(MINO_GC_FULL)`
+(the `(gc!)` primitive) ran the nested minor BEFORE finishing the
+in-flight major. The minor freed a YOUNG header that was still on
+major's mark stack from a prior incremental slice; the subsequent
+`gc_force_finish_major` chased the freed pointer through
+`gc_mark_child_push`.
+
+`gc_tick_during_major` (driver.c:141-147) already documents the
+correct ordering -- "finish-then-minor rather than nest" -- and
+honours it for the auto-tick path; the public-API path simply had
+the calls reversed. Auto-tick is the common case so the bug went
+unnoticed locally; CI's test ordering puts `transient-survives-gc-
+yield` (which explicitly calls `(gc!)`) inside an in-flight major
+on every push, surfacing the bug deterministically.
+
+Symptom shape under each environment:
+
+* GitHub Actions matrix (macos-14 / ubuntu-24.04{,-arm} /
+  windows-2022): 8-min timeout in the Test step, identical hang
+  point across all four OSes -- the v0.255.8 diagnostic trace
+  pinned the hanging deftest to `transient-survives-gc-yield`.
+* Local without sanitizer: intermittent SIGSEGV in
+  `tiny_free_no_lock` (sweep double-free) or
+  `error[MTY001] inc expects a number` (corrupted value via the
+  reused freed header).
+* Local under ASan: clean repro --
+  `heap-use-after-free at gc_mark_child_push driver.c:414`
+  with the freer site at `gc_minor_collect minor.c:472` inside the
+  same `mino_gc_collect` call.
+
+Fix: reorder `MINO_GC_FULL` to finish the major BEFORE the minor,
+mirroring `gc_tick_during_major`. Behaviour for `MINO_GC_MINOR` and
+`MINO_GC_MAJOR` unchanged.
+
+Regression: `tests/gc_test.clj` gains
+`gc-bang-during-incremental-major` -- warms up enough OLDs to start
+an incremental major, then calls `(gc!)` mid-mark in the transient
+shape that surfaced the bug. Fails (SIGSEGV / inc-error / hang)
+under the old ordering, passes under the fix. Full suite climbs
+from 1273 / 4555 to 1274 / 4557 (one deftest, two assertions).
+
+Local verification:
+* `./mino tests/run.clj`: 1274 / 4557 green in 1.6s.
+* `./mino_asan tests/run.clj`: 1274 / 4557 green, ASan clean.
+* `./mino task release-gate`: 17/17 probes green.
+* 5x consecutive `./mino /tmp/repro.clj` (35 transient deftests):
+  all green, no flakiness.
+
 ## v0.255.8 — Diagnostic: Per-Deftest Trace for CI Hang Investigation
 
 A diagnostic-only release: surfaces which test is running at the

src/mino.h

@@ -28,7 +28,7 @@
  */
 #define MINO_VERSION_MAJOR 0
 #define MINO_VERSION_MINOR 255
-#define MINO_VERSION_PATCH 8
+#define MINO_VERSION_PATCH 9
 
 /*
  * Human-readable version string of the *linked* runtime, e.g. "0.48.0".

src/public/gc.c

@@ -35,12 +35,18 @@ void mino_gc_collect(mino_state_t *S, mino_gc_kind_t kind)
         if (mino_current_ctx(S)->gc_stack_bottom == NULL) {
             return;
         }
-        if (S->gc_bytes_young > 0) {
-            gc_minor_collect(S);
-        }
+        /* Finish any in-flight major BEFORE running the minor. A nested
+         * minor while major's mark stack still holds OLD entries could
+         * free a YOUNG object reachable only through an OLD pointer
+         * pending on major's stack; major's next gc_trace_children
+         * would then chase the freed pointer. Same invariant the
+         * auto-tick path (gc_tick_during_major) honours. */
         if (S->gc_phase == GC_PHASE_MAJOR_MARK) {
             gc_force_finish_major(S);
         }
+        if (S->gc_bytes_young > 0) {
+            gc_minor_collect(S);
+        }
         if (S->gc_phase == GC_PHASE_IDLE) {
             gc_major_collect(S);
         }

tests/gc_test.clj

@@ -29,3 +29,37 @@
       acc
       (build__gc (- n 1) (list acc)))))
   (is (cons? (build__gc 200 42))))
+
+;; Regression for gc! during mid-major-mark: mino_gc_collect(MINO_GC_FULL)
+;; used to run the minor BEFORE finishing the in-flight major; the minor
+;; would free a YOUNG header still on the major's mark stack, and the
+;; subsequent major step would chase the freed pointer. Surfaces on CI
+;; runners as a hang in tests/transient_test (test-suite ordering puts
+;; transient-survives-gc-yield inside an in-flight major). Locally
+;; reproduced via ASan as heap-use-after-free in gc_mark_child_push.
+;;
+;; The trigger is: heat enough OLD objects to start an incremental
+;; major, then call gc! while the major's mark stack is still
+;; non-empty. The fixed mino_gc_collect now finish-majors first, then
+;; runs the minor, matching the auto-tick path's invariant.
+(deftest gc-bang-during-incremental-major
+  ;; Warm up: promote many objects so the next minor will start a
+  ;; major. Each loop iteration grows a vec; the vec survives long
+  ;; enough to be promoted.
+  (let [warmup (loop [i 0 acc []]
+                 (if (= i 5000)
+                   acc
+                   (recur (inc i) (conj acc i))))]
+    (is (= 5000 (count warmup))))
+  ;; Now run the pattern that used to corrupt: transient + gc!
+  ;; sequence under an active major mark. If the bug regresses we
+  ;; get a SIGSEGV in gc_sweep / gc_mark_child_push, or a stray
+  ;; "inc expects a number" caught by the test framework.
+  (let [t (transient [])]
+    (conj! t 1)
+    (gc!)
+    (conj! t 2)
+    (gc!)
+    (conj! t 3)
+    (gc!)
+    (is (= [1 2 3] (persistent! t)))))