Mitigate forking while having multiple threads.

If `fork()` is called when having multiple threads, the child thread
only inherits the forking thread.  Other threads simply vanish, stopping
anywhere it was executing without any cleaning up.  As a result, when
using MMTk, the modbufs of other threads will not be flushed, and the
next nursery GC will fail to keep some root-reachable objects alive, and
the program will crash later.

This commit mitigates this problem by destroying mutators of dead
threads after forking.  Concretely, the call of
`rb_mmtk_destroy_mutator` is moved into `thread_cleanup_func` which is
called not only when the thread exits normally, but also after forking
to clean up dead threads.  That ensures the modbufs are flushed.

According to the POSIX API, if `fork()` is called while there are
multiple threads, the child can only call async-signal-safe functions
until calling execve.  There are test cases (such as
`test_autoload_fork`) that forks while having multiple threads, and
there may be real-world applications doing so (see
https://bugs.ruby-lang.org/issues/14634).  They are all unsafe according
to POSIX.  So this commit only mitigates the problem, and programs that
fork while having multiple threads are still technically incorrect.

Fixes: #83

Author: wks

Diff for: internal/mmtk_support.h

@@ -44,7 +44,7 @@ void rb_mmtk_main_thread_init(void);
 
 // Flushing and de-initialization
 void rb_mmtk_flush_mutator_local_buffers(MMTk_VMMutatorThread thread);
-void rb_mmtk_destroy_mutator(MMTk_VMMutatorThread cur_thread);
+void rb_mmtk_destroy_mutator(MMTk_VMMutatorThread cur_thread, bool at_fork);
 
 // Object layout
 size_t rb_mmtk_prefix_size(void);

Diff for: mmtk_support.c

@@ -357,11 +357,16 @@ rb_mmtk_flush_mutator_local_buffers(MMTk_VMMutatorThread thread)
 }
 
 void
-rb_mmtk_destroy_mutator(MMTk_VMMutatorThread cur_thread)
+rb_mmtk_destroy_mutator(MMTk_VMMutatorThread cur_thread, bool at_fork)
 {
-    // Currently a thread can only destroy its own mutator.
-    RUBY_ASSERT(cur_thread == GET_THREAD());
-    RUBY_ASSERT(cur_thread->mutator_local == &rb_mmtk_mutator_local);
+    if (!at_fork) {
+        // A thread only destroys its own mutator when it exits normally (not at fork).
+        // But after forking, only the forking thread continue to live in the child process.
+        // The living thread will call this function to close the mutators of all dead threads.
+        // So we skip the assertions at fork.
+        RUBY_ASSERT(cur_thread == GET_THREAD());
+        RUBY_ASSERT(cur_thread->mutator_local == &rb_mmtk_mutator_local);
+    }
 
     rb_mmtk_flush_mutator_local_buffers(cur_thread);
 

Diff for: thread.c

@@ -515,6 +515,32 @@ thread_cleanup_func(void *th_ptr, int atfork)
     th->locking_mutex = Qfalse;
     thread_cleanup_func_before_exec(th_ptr);
 
+#if USE_MMTK
+    // Release MMTk-specific per-mutator structs.
+    if (rb_mmtk_enabled_p()) {
+        bool destroy_mutator = true;
+
+        if (!atfork) {
+            // If called when a thread exits normally (not at fork),
+            // the current thread should have mutator-local data,
+            // and we will destroy the mutator.
+            RUBY_ASSERT(th->mutator != NULL);
+            RUBY_ASSERT(th->mutator_local != NULL);
+        } else {
+            // If called after fork, we visit all threads,
+            // including the threads that have been "scheduled dead".
+            // We only destry the mutator if not already destroyed.
+            if (th->mutator == NULL) {
+                destroy_mutator = false;
+            }
+        }
+
+        if (destroy_mutator) {
+            rb_mmtk_destroy_mutator(th, atfork);
+        }
+    }
+#endif
+
     /*
      * Unfortunately, we can't release native threading resource at fork
      * because libc may have unstable locking state therefore touching
@@ -779,17 +805,6 @@ thread_start_func_2(rb_thread_t *th, VALUE *stack_start)
     thread_cleanup_func(th, FALSE);
     VM_ASSERT(th->ec->vm_stack == NULL);
 
-#if USE_MMTK
-    // Release MMTk-specific per-mutator structs.
-    // We are still holding the thread_sched, so other threads cannot inspect the following
-    // per-mutator structs.  It's safe for us to clean them up at this time.
-    if (rb_mmtk_enabled_p()) {
-        RUBY_ASSERT(th->mutator != NULL);
-        RUBY_ASSERT(th->mutator_local != NULL);
-        rb_mmtk_destroy_mutator(th);
-    }
-#endif
-
     if (th->invoke_type == thread_invoke_type_ractor_proc) {
         // after rb_ractor_living_threads_remove()
         // GC will happen anytime and this ractor can be collected (and destroy GVL).