Pitchfork 0.18.2: fork_sibling gets nil PID while spawning Discourse worker on Ruby 3.4

[the-par3a]

Description

I'm debugging a worker spawning failure with Pitchfork 0.18.2 in a Discourse development environment running Ruby 3.4.0.

During startup, Pitchfork fails to spawn a service worker. The parent side of fork_sibling receives nil from the pipe instead of the child's PID.

Observed output

DEBUG parent pid_str=nil
mold gen=0 pid=15371 failed to spawn a service, retrying
DEBUG parent pid_str=nil
mold gen=0 pid=15371 failed to spawn a service twice in a row - corrupted mold process?
mold gen=0 pid=15371 reaped (#<Process::Status: pid 15371 exit 1>)

The relevant code is in Pitchfork::HttpServer#fork_sibling.

I temporarily added logging immediately before the yield inside Pitchfork.clean_fork:

pid = Pitchfork.clean_fork(setpgid: setpgid) do
  w.close
  STDERR.puts "DEBUG before yield"
  STDERR.flush
  yield
  STDERR.puts "DEBUG after yield"
end

However, during the failure, DEBUG before yield is never printed.

At the same time, the parent reaches:

pid_str = r.gets

and receives:

DEBUG parent pid_str=nil

This suggests that the middle process exits before the inner clean_fork block reaches the yield, or otherwise closes the pipe without writing the grandchild PID.

Environment

• Pitchfork: 0.18.2
• Ruby: 3.4.0
• Application: Discourse
• Environment: development
• Platform: Linux container
• Pitchfork::REFORKING_AVAILABLE: true
• UNICORN_WORKERS: 3
• setpgid: false in Discourse development configuration
• spawn_timeout: 60 seconds by default

Discourse configuration

The Discourse config/pitchfork.conf.rb defines several fork callbacks, including:

before_fork do |server|
  Discourse.redis.close
  # ...
end

after_mold_fork do |server, mold|
  # ...
  Discourse.redis.close
  Discourse.before_fork
end

after_worker_fork do |server, worker|
  DiscourseEvent.trigger(:web_fork_started)
  Discourse.apply_worker_db_variables_overrides
  Discourse.after_fork
  SignalTrapLogger.instance.after_fork
end

No refork_after configuration is present in the Discourse application configuration.

Additional investigation

REFORKING_AVAILABLE is confirmed to be enabled:

REFORKING_AVAILABLE=true

The Pitchfork source shows that the Linux reforking path uses a double fork:

if middle_pid = FORK_LOCK.synchronize { Process.fork }
  w.close
  process_wait_with_timeout(middle_pid, FORK_TIMEOUT)
  pid_str = r.gets
  # pid_str is nil
else
  reset_signal_handlers
  r.close

  pid = Pitchfork.clean_fork(setpgid: setpgid) do
    w.close
    # DEBUG before yield is never reached
    yield
  end

  w.puts(pid)
  w.close
end

I also tested the basic double-fork + pipe behavior separately in the same environment, and it worked correctly. The parent received the grandchild PID from the pipe.

This makes it less likely that the problem is simply a failure of Process.fork or pipe communication in the environment.

Expected behavior

fork_sibling should successfully create the grandchild worker and write its PID to the pipe so that the parent can read it with:

pid_str = r.gets

Actual behavior

The parent receives nil:

DEBUG parent pid_str=nil

and the mold eventually exits with status 1 after retrying the worker spawn.

Questions

1. Is there any known compatibility issue between Pitchfork 0.18.2 and Ruby 3.4 involving fork_sibling, clean_fork, or fork callbacks?
2. Could a callback, thread, or signal handler in the Discourse/Rails process cause the middle process to exit before the clean_fork block reaches yield?
3. Is there a recommended way to instrument fork_sibling or clean_fork to determine why the middle process exits before w.puts(pid)?
4. Is there any additional diagnostic information that would be useful to collect?

Thanks!

[byroot]

Sorry for the late reply.

1. Is there any known compatibility issue between Pitchfork 0.18.2 and Ruby 3.4 involving

No known issue no.

Could a callback, thread, or signal handler in the Discourse/Rails process cause the middle process to exit before the clean_fork block reaches yield?

A thread shouldn't be possible, as all other threads are dead in the forked process.

It could be caused by a decorator to Process._fork. There was a similar-ish report on Ruby not long ago: https://bugs.ruby-lang.org/issues/22243

Is there a recommended way to instrument fork_sibling or clean_fork to determine why the middle process exits before w.puts(pid)?

Not particularly, but I suppose you could define your own Process._fork decorator to be able to log sooner, and perhaps even catch an exception?

Another idea is to use RUBYOPT="-d" (or swap $DEBUG = true in Ruby) so that Ruby would print all raised exceptions on stderr, you might be able to see that an exception is raised but not handled).