Elixir apps running on VirtualBox crash repetitively with `OS monotonic time stepped backwards!`

[datafoo]

Describe the bug

We are testing our Elixir apps on a different virtualization platform: Oracle VirtualBox.

Our Elixir apps running on VirtualBox crash repetitively with error message OS monotonic time stepped backwards!.

Tested on VirtualBox 7.1.4 on Windows 11 Pro 24H2 and with Elixir (1.18.1) and Erlang/OTP 27 [erts-15.2].

To Reproduce

Unknown

Our apps crashes at unpredictable times.

Expected behavior

The Elixir apps should not crash.

Affected versions

Erlang/OTP 27 (issue also exists on Erlang/OTP 25).

Additional context

Logs of the crash:

[root@myserver:~]# journalctl -b -S 10:14:40 -U 10:14:50 | cat
Feb 24 10:14:44 myserver myapp[2680]: OS monotonic time stepped backwards!
Feb 24 10:14:44 myserver myapp[2680]: Previous time: 5730208658372
Feb 24 10:14:44 myserver myapp[2680]: Current time:  5730208647858
Feb 24 10:14:44 myserver systemd-coredump[4075]: Process 2680 (beam.smp) of user 399 terminated abnormally with signal 6/ABRT, processing...
Feb 24 10:14:44 myserver systemd[1]: Started Process Core Dump (PID 4075/UID 0).
Feb 24 10:14:44 myserver systemd-coredump[4076]: Removed old coredump core.beam\x2esmp.395.f5481f856fd04a718b5ca70219bc6f9d.2255.1740388064000000.zst.
Feb 24 10:14:49 myserver systemd-coredump[4076]: Process 2680 (beam.smp) of user 399 dumped core.
                                            
                                            Module crypto_callback.so without build-id.
                                            Module crypto.so without build-id.
                                            Module argon2_nif.so without build-id.
                                            Module asn1rt_nif.so without build-id.
                                            Module dyntrace.so without build-id.
                                            Module libgcc_s.so.1 without build-id.
                                            Module libstdc++.so.6 without build-id.
                                            Module libz.so.1 without build-id.
                                            Module libncursesw.so.6 without build-id.
                                            Module beam.smp without build-id.
                                            Stack trace of thread 2721:
                                            #0  0x00007f38baefba9c __pthread_kill_implementation (libc.so.6 + 0x92a9c)
                                            #1  0x00007f38baea9576 raise (libc.so.6 + 0x40576)
                                            #2  0x00007f38bae91935 abort (libc.so.6 + 0x28935)
                                            #3  0x0000000000602401 erts_exit_epilogue.cold (beam.smp + 0x202401)
                                            #4  0x000000000072aec9 erts_exit_vv (beam.smp + 0x32aec9)
                                            #5  0x000000000072b01f erts_exit (beam.smp + 0x32b01f)
                                            #6  0x00000000007bce76 get_os_corrected_time (beam.smp + 0x3bce76)
                                            #7  0x00000000007bf9c9 erts_get_monotonic_time (beam.smp + 0x3bf9c9)
                                            #8  0x000000000061f91a erts_schedule (beam.smp + 0x21f91a)
                                            #9  0x00007f38736a4a0d n/a (n/a + 0x0)
                                            ELF object binary architecture: AMD x86-64
Feb 24 10:14:49 myserver systemd[1]: systemd-coredump@11-4075-0.service: Deactivated successfully.
Feb 24 10:14:49 myserver systemd[1]: systemd-coredump@11-4075-0.service: Consumed 2.609s CPU time, 4.8M memory peak.
Feb 24 10:14:49 myserver systemd[1]: myapp.service: Main process exited, code=dumped, status=6/ABRT
Feb 24 10:14:49 myserver systemd[1]: myapp.service: Failed with result 'core-dump'.

A bit of details regarding my environment:

[root@myserver:~]# hostnamectl 
     Static hostname: redacted
     Pretty hostname: redacted
           Icon name: computer-vm
             Chassis: vm 🖴
          Deployment: development
            Location: redacted
          Machine ID: redacted
             Boot ID: redacted
        Product UUID: redacted
      Virtualization: oracle
    Operating System: NixOS 24.11 (Vicuna)                                                             
         CPE OS Name: cpe:/o:nixos:nixos:24.11
      OS Support End: Mon 2025-06-30
OS Support Remaining: 4month 3d
              Kernel: Linux 6.6.71
        Architecture: x86-64
     Hardware Vendor: innotek GmbH
      Hardware Model: VirtualBox
     Hardware Serial: VirtualBox-redacted
    Firmware Version: VirtualBox
       Firmware Date: Fri 2006-12-01
        Firmware Age: 18y 2month 3w 4d                                                                 

[root@myserver:~]# myapp remote
Erlang/OTP 27 [erts-15.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.18.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(myapp@myserver)1> :erlang.system_info :time_correction
true
iex(myapp@myserver)2> :erlang.system_info :time_warp_mode
:multi_time_warp
iex(myapp@myserver)3> :erlang.system_info :os_monotonic_time_source
[
  function: :clock_gettime,
  clock_id: :CLOCK_MONOTONIC,
  resolution: 1000000000,
  used_resolution: 1000000000,
  extended: :no,
  parallel: :yes,
  time: 6450367121022
]
iex(myapp@myserver)4> 
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
^C
[root@myserver:~]# journalctl -b -u myapp.service --grep 'monotonic'
Feb 24 08:48:59 myserver myapp[1433]: OS monotonic time stepped backwards!
Feb 24 09:09:54 myserver myapp[2013]: OS monotonic time stepped backwards!
Feb 24 10:14:44 myserver myapp[2680]: OS monotonic time stepped backwards!

I am not sure if it is an Erlang issue, it might very well be an issue with VirtualBox.

First reported at https://elixirforum.com/t/elixir-crashes-in-virtualbox-with-monotonic-time-message/51004

[garazdawi]

Hello! This means that the OS monotonic time in your environment has stepped backwards. This is not supposed to happen and is something that Erlang relies on not happening. This is normally due to the guest VM being buggy/incorrectly configured.

If you cannot fix the VM, you can compile Erlang with --enable-ensure-os-monotonic-time. The flags has this notice in the documentation:

--enable-ensure-os-monotonic-time - Enable functionality ensuring the monotonicity of monotonic timestamps delivered by the OS. When a non-monotonic timestamp is detected, it will be replaced by the last delivered monotonic timestamp before being used by Erlang's time functionality. Note that you do not want to enable this unless the OS monotonic time source on the system fails to produce monotonic timestamps. This since ensuring the monotonicity of OS monotonic timestamps will hurt scalability and performance of the system.