uncaught exception with corrupted cache or _build

[raphael-proust]

Expected Behavior

When a build is interrupted, the _build/ dir or the cache can become corrupted and subsequent builds get a very wild error message for an uncaught exception.

• no corruption, or
• informative error message and a tool to recover manually, or
• automatic recovery

Actual Behavior

I get the following error message (I removed actual file names, replaced with […]), I left the null-byte string as is from the error message.

Description:
  ("Invalid Module_name.t",
   { s = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000" })
Raised at Stdune__Code_error.raise in file "stdune__Code_error.ml", line 11,
  characters 30-62
Called from Dune_lang__Module_name.Unique.of_string in file
  "dune_lang__Module_name.ml", line 88, characters 55-68
Called from Dune_rules__Ocamldep.parse_compilation_units.(fun) in file
  "dune_rules__Ocamldep.ml", line 82, characters 19-49
Called from Stdune__List.rev_filter_map.loop in file "stdune__List.ml", line
  22, characters 13-16
Called from Stdune__List.filter_map in file "stdune__List.ml" (inlined), line
  29, characters 26-47
Called from Dune_rules__Ocamldep.parse_compilation_units in file
  "dune_rules__Ocamldep.ml", lines 81-84, characters 2-54
Called from Dune_engine__Action_builder.T.M.map.(fun) in file
  "dune_engine__Action_builder.ml", line 45, characters 12-15
Called from Fiber__Core.O.(>>|).(fun) in file "fiber__Core.ml", line 258,
  characters 36-41
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
Re-raised at Stdune__Exn.raise_with_backtrace in file "stdune__Exn.ml"
  (inlined), line 39, characters 27-56
Called from Stdune__Exn_with_backtrace.reraise in file
  "stdune__Exn_with_backtrace.ml", line 21, characters 33-71
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
Re-raised at Stdune__Exn.raise_with_backtrace in file "stdune__Exn.ml"
  (inlined), line 39, characters 27-56
Called from Stdune__Exn_with_backtrace.reraise in file
  "stdune__Exn_with_backtrace.ml", line 21, characters 33-71
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
Re-raised at Stdune__Exn.raise_with_backtrace in file "stdune__Exn.ml"
  (inlined), line 39, characters 27-56
Called from Stdune__Exn_with_backtrace.reraise in file
  "stdune__Exn_with_backtrace.ml", line 21, characters 33-71
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
Re-raised at Stdune__Exn.raise_with_backtrace in file "stdune__Exn.ml"
  (inlined), line 39, characters 27-56
Called from Stdune__Exn_with_backtrace.reraise in file
  "stdune__Exn_with_backtrace.ml", line 21, characters 33-71
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
Re-raised at Stdune__Exn.raise_with_backtrace in file "stdune__Exn.ml"
  (inlined), line 39, characters 27-56
Called from Stdune__Exn_with_backtrace.reraise in file
  "stdune__Exn_with_backtrace.ml", line 21, characters 33-71
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
-> required by
   ("_build/default/[…]",
    ())
-> required by ("Rule.make", ())
-> required by
   ("execute-rule",
    { id = 259307
    ; info =
        From_dune_file
          { pos_fname =
              "[…]/dune"
          ; start = { pos_lnum = […]; pos_bol = […]; pos_cnum = […] }
          ; stop = { pos_lnum = […]; pos_bol = […]; pos_cnum = […] }
          }
    })
-> required by ("<unnamed>", ())
-> required by
   ("build-file",
    In_build_dir
      "default/[…]")
-> required by ("<unnamed>", ())
-> required by
   ("build-alias",
    { dir = In_build_dir "default/[…]"; name = "[…]" })
-> required by ("toplevel", ())
I must not crash.  Uncertainty is the mind-killer. Exceptions are the
little-death that brings total obliteration.  I will fully express my cases.
Execution will pass over me and through me.  And when it has gone past, I
will unwind the stack along its path.  Where the cases are handled there will
be nothing.  Only I will remain.

Note that when these failures happen, they generally happen multiple times. I'm guessing that parallel jobs in the build experience them and that all the errors are printed (before the build is interrupted by the failures).

Reproduction

I'm trying to reproduce the corruption.

Specifications

• Version of dune (output of dune --version): 3.21.1
• Version of ocaml (output of ocamlc --version): 5.4.1

[Alizter]

Could you give some more details on how you are interrupting the builds? Are you sending Ctrl+C? Is this happening during watch mode? Did something else kill dune?

[raphael-proust]

Could you give some more details on how you are interrupting the builds?

AFAICT it's hard reboots after which some files are corrupted. I don't necessarily expect dune to fully recover from such a situation. I think two possible improvements are:

• giving an informative error message ({|The module %S (required by %S) is invalid. Data on disk may have been corrupted?|} or something of the sort)
• atomic writes via writes to a temporary file followed by a rename so that the file is either not found (if interrupted before the rename) or correct (if interrupted after)

the second improvement might be incompatible with some of the optimisations that dune might be doing internally with write logic (atomic writes require a flush between write and rename) so maybe it's not the right approach

[Alizter]

The shared cache is more robust to interrupted writes, however there are some internal rule and digest caches in _build/ where this might not be the case.

If the problem persists after deleting _build/ then that means the shared cache has been poisoned which is of course more serious, but if _build/ fixes the issue then its likely the local caches have become stale due to interruptions, and I would be less worried.

The null bytes you have in the module name seems like OS added padding.

I'm not exactly sure how it is possible to trick dune this way, so I would be interested in finding out.

Could you provide the following information:

1. Which OS/arch are you using?
2. Which file system are you using?
3. Is this happening when running watch mode?
4. Does rm -rf _build/ or dune clean fix the problem?
5. Does this happen every time you do a hard reboot during a build or only sometimes?

[raphael-proust]

1. Debian/64
2. i think ext4 (i'll double check and say if otherwise
3. not watch mode, this is in the CI with a plain dune build … call
4. we've taken a sledgehammer approach of re-initialising the whole environment (incl opam workspace), but the opam repo is initialised at a different stage and has had several successful builds before the reboot/corruption/failures, i'll give those clean-up commands a try manually next time
5. only sometimes