[new release] dune (17 packages) (3.21.0)

[shonfeder]

Fast, portable, and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

Fixed

• Fix include_subdirs qualified incorrectly picking the furthest module
  instead of the closest when resolving module name ambiguities. (Fix difference between bootstrap & dune behaviour w.r.t include-subdirs ambiguity dune#12587,
  @ElectreAAS and @Alizter)

• Fix: include the module alias in the transitive dependency closure with
  (include_subdirs qualified). (fix(include_subdirs): include alias module in closure group dune#12299, @anmonteiro)

• Improve error messages for invalid version formats containing non-ASCII
  characters. Previously, non-ASCII characters in version strings (e.g., (lang dune è) or (using menhir π3.14)) would fail with a generic "Invalid file"
  error. Now they display a clear message: "Invalid atom: contains non-ASCII
  character(s). Atoms must only contain ASCII characters." The fix is
  implemented at the lexer level, providing consistent error handling across all
  s-expression parsing. (Feat/good error message extensions non ascii dune#12844, fixes Improve error message for non-ASCII characters in extension versions dune#12836, @benodiwal)

• Pass private modules with -H when this is available (fix: -H for hidden modules dune#12666, @rgrinberg)

• Allow multiple modules in (modules_flags ...), in coq.theory (Coq/Rocq: fix handling of (modules_flags ...). dune#12733, @rlepigre)

• Improve error message for invalid version formats in both (lang dune ...) and
  (using extension ...) declarations. Changes "Atom of the form NNN.NNN expected"
  to "Invalid version. Version must be two numbers separated by a dot." (fix: same error message for both ascii and non ascii invalid version cases dune#12833, @benodiwal)

• Fix crash when running dune build @check on a library with virtual modules.
  (fix: missing virtual implementation in top closure dune#12644, fixes virtual libraries: "dune build @check" bug? dune#12636, @Alizter)

• Provide a more informative error message when (pkg enabled) is put in
  dune-project instead of dune-workspace. (Fix/good error msg pkg enabled dune#12802, fixes Good error when (pkg enabled) is put in dune-project dune#12801,
  @benodiwal)

• Improve error message when invalid version strings are used in dune-project
  files. Non-ASCII characters and malformed versions now show a helpful hint
  with an example of the correct format. (fix: Fix crash on non-ASCII version in dune-project dune#12794, fixes Crash on non-ascii version in dune-project dune#12751, @benodiwal)

• Stop hiding the root_module from the include path (fix: stop hiding the root module dune#12239, @rgrinberg)

• Allow $ dune init to work on absolute paths (dune init: use source path dune#12601, fixes dune init project crashes with an absolute directory dune#7806,
  @rgrinberg)

• (include_subdirs qualified): Add missing alias dependency to module group.
  (fix(include_qualified): add missing alias dependency to module group dune#12530, @anmonteiro)

• Add Melange compilation to the @all alias in libraries (fix(melange): add melange module compilation as part of @all alias dune#12628,
  @anmonteiro)

• Fix greedy version location in lang declarations. Previously, error locations for
  invalid lang versions would span multiple bytes for multi-byte UTF-8 characters,
  causing carets to appear misaligned and seemingly include the closing
  parenthesis. Now, error locations for ASCII strings show the full length (e.g.,
  "Ali" shows ^^^), while non-ASCII strings show only the first byte (e.g., "è"
  shows ^) to avoid multi-byte character display issues. (fix: greedy version location in lang declarations dune#12869, fixes Location excerpts miscompute character positions for unicode and greedy ranges dune#12806,
  @benodiwal)

• melange support: don't emit empty JavaScript modules for generated module
  aliases. (melange: don't emit empty .js files for module aliases dune#12464, @anmonteiro)

Added

• (Experimental): Introduce the library_parameter stanza. It allows users to
  declare a parameter when using the OxCaml compiler.
  (feat(oxcaml): add the library_parameter stanza dune#11963, implements [OxCaml] Parameterized libraries: library parameter dune#12084, @maiste)

• Added the ability to scroll horizontally in TUI. (feat(tui): horizontal mouse scrolling dune#12386, @Alizter)

• Feature: Include shell command that was executed when a cram test has
  occurred in the error message (feature(cram): include failed command in cram timeouts dune#12307, @rgrinberg)

• support expanding variables in (promote (into ..)) (feat: expand variables in (promote (into ..)) dune#12832, fixes Promote into doesn't accept dune variables dune#12742,
  @anmonteiro)

• Add support for %{cmt:...} and %{cmti:...} variables to reference
  compiled annotation files (.cmt and .cmti) containing typed abstract syntax
  trees with location and type information. (feat: %{cmt} and %{cmti} artifact variables dune#12634, grants Add %{cmt} to "artifacts variables" dune#12633, @Alizter)

• Add $ dune describe tests to describe the tests in the workspace
  (@Gromototo, add dune describe tests subcommand dune#12545, fixes dune describe doesn't show tests dune#12030)

• Add argv, the process environment, and the dune version to the config event
  in the trace (Add more info to the config event dune#12909, @rgrinberg)

• Allow dune runtest to properly run while a watch mode server is running.
  (Allow running tests concurrently with watch server, again! dune#12473, grants Running tests and exec-ing executables should be possible while build server is running dune#8114, @gridbugs and @ElectreAAS)

• Use copy-on-write (COW) when copying files on filesystems that support it
  (Btrfs, ZFS, XFS, etc), under Linux. (Use copy_file_range when supported dune#12074, fixes Cache: Support Copy on Write (cp --reflink) dune#12071, @nojb)

• Add support for Tangled ATproto-based code repositories (Add support for Tangled ATproto-based code repositories dune#12197, @avsm)

• Add support for instantiating OxCaml parameterised libraries.
  (feat(oxcaml): instantiate parameterised libraries dune#12561, @art-w)

• Add a (conflict_markers error|ignore) option to the cram stanza. When
  (conflict_markers error) is set, the cram test will fail in the presence of
  conflict markers. Git, diff3 and jujutsu conflict markers are detected.
  (feature(cram): allow for conflict detection dune#12538, feat(cram): detect diff3 and jujutsu conflicts dune#12617, fix(cram): allow text after conflict markers dune#12655, fixes Issue a warning when there are conflict markers in test files dune#12512, @rgrinberg, @Alizter)

• Introduce a %{ppx:lib1+..+libn} stanza to make it possible to refer to ppx
  executables built by dune. This is useful for writing tests (Add a ppx pform to simplify writing tests dune#12711,
  @rgrinberg)

• Introduce a (dir ..) field on packages defined in the dune-project. This
  field allows to associate a directory with a particular package. This makes
  dune automatically filter out all stanzas in this directory and its
  descendants with --only-packages. All users are recommended to switch to
  using this field. (One Directory per Package dune#12614, fixes One package per directory mode dune#3255, @rgrinberg)

• Add support for DUNE_ROOT environment variable, similar to the existing
  --root CLI parameter. (fixes Add support for DUNE_ROOT environment variable dune#12399 @sir4ur0n)

• Introduce an unused-libs alias to detect unused libraries.
  (feature: add an unused-libs alias dune#12623, fixes Dune should warn about dependencies on unused libraries dune#650, @rgrinberg)

• Add --files flag to dune describe opam-files to print only the names of
  the opam files line by line. (Add --files to dune describe opam-files dune#9793, @reynir and @Alizter)

• dune exec now accepts absolute paths inside the workspace.
  (feat(exec): support absolute paths for dune exec dune#12094, @Alizter)

• Add coqdoc_header and coqdoc_footer fields to the coq field of the
  env stanza, and to the coq.theory stanza, allowing to configure a
  custom header or footer respectively in the HTML output of coqdoc.
  (Add "coqdoc_header" and "coqdoc_footer" fields. dune#11131, @rlepigre)

• Allow dune fmt to properly run while a watch mode server is running.
  Note that the --preview flag is not supported in this mode.
  ([RPC] Allow formatting while watch server is running dune#12064, @ElectreAAS)

• Support for generating _CoqProject files for coq.theory stanzas.
  (Generate "_CoqProject" files for Coq theories dune#11752, @rlepigre)

• Added (files) stanza, similar to (dirs) to control which files are visible
  to Dune on a per-directory basis. (Add (files) stanza dune#12879, @nojb)

• Add support for %{ocaml-config:ox} (Expose 'ocamlc -config' parameter 'ox' dune#12236, @jonludlam)

• Introduce dune promotion show command to display the contents of corrected
  files that are ready for promotion. This allows users to preview changes
  before running dune promote. The command accepts file arguments to show
  specific files, or displays all promotable files when called without
  arguments. (Add ‘dune promotion show’ command and tests dune#12669, fixes Add dune command to get the contents of a "corrected" file dune#3883, @MixiMaxiMouse)

• New (lang rocq) build mode for Rocq 9.0 and later. This new mode
  is very similar to the existing (lang coq), except that it doesn't
  need the coq* compatibility wrappers. As of today (lang rocq)
  doesn't support yet composed builds with Rocq itself, this will be
  added later. (lang coq) is deprecated, development is frozen, and
  will be removed at some point in the future. ([rocq] Initial support for the Rocq Prover build language dune#12035, @ejgallego,
  @Lysxia, fixes Rocq support in dune dune#11572)

Changed

• Don't run ocamldep to compute false dependencies on the root_module
  (Fix handling of root module dependencies dune#12227, @rgrinberg)

• dune format-dune-file now uses the syntax version of the Dune project that
  contains the file being formatted (if any) instead of using the latest version
  available, which remains the default if there is no Dune project in scope.
  (dune format-dune-file: use Dune lang version of current Dune project dune#11865, @nojb)

• Persistent DB and process events have been slightly modified. Persistent
  DB events have more concise names and job events always include full
  information. (Move trace events to dune_trace dune#12867, @rgrinberg)

• Removed the --trace-extended flag. Its functionality is always enabled when
  tracing is active (Remove --trace-extended dune#12908, @rgrinberg)

• The test/dune file generated by dune init proj now depends on the project library. (Add library dependency in dune init proj tests dune#12791, @shonfeder)

• Starting with version 3.21 of the Dune language, Dune no longer changes the
  default set of compiler warnings. For users that would like to keep the old
  behaviour, the variable %{dune-warnings} can be used in an (env) stanza in
  a top-level Dune file: (env (dev (flags :standard %{dune-warnings}))).
  (Stop changing the default warning set of the compiler dune#12766, @nojb)

• Fix: stop generating cmt files for cinaps binaries (fix(include_qualified): add missing alias dependency to module group dune#12530, @rgrinberg)

[shonfeder]

After quite a few attempts, we believe we have identified and fixed all regressions, and we have determined that all remaining revdeps errors are due to unrelated factors.

For records of our triage see the review of this release candidate, before we cut the point release:

• [new release] dune (17 packages) (3.21.0~alpha5) #29189
• 3.21.0 release tracker dune#12619 (comment)

I think we should be good to merge here.

[hannesm]

I appreciate your work on this. I still am curious about the decompress failures, that Marcello mentioned in #29189 (comment) and that are still in this CI run.

[shonfeder]

@hannesm ah, sorry I missed addressing that. I seem to skimmed over that part of the comment and only addressed the second issue.

The decompress failure @mseri reported is of this sort:

[ERROR] The compilation of decompress.1.4.2 failed at "dune runtest -p decompress -j 255".

#=== ERROR while compiling decompress.1.4.2 ===================================#
# context              2.5.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | file:///home/opam/opam-repository
# path                 ~/.opam/4.14/.opam-switch/build/decompress.1.4.2
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p decompress -j 255
# exit-code            1
# env-file             ~/.opam/log/decompress-7-c35180.env
# output-file          ~/.opam/log/decompress-7-c35180.out
### output ###
# File "bindings/stubs/dune", line 15, characters 22-26:
# 15 |  (forbidden_libraries unix)
#                            ^^^^
# Error: Library "unix" was pulled in.
# -> required by library "bigarray" in /home/opam/.opam/4.14/lib/ocaml
# -> required by library "ctypes" in /home/opam/.opam/4.14/lib/ctypes
# -> required by library "gen_decompress_bindings" in
#    _build/default/bindings/stubs
# -> required by executable decompress in bindings/stubs/dune:13
# -> required by _build/default/bindings/stubs/decompress.exe.o
# -> required by %{exe:decompress.exe.o} at bindings/stubs/dune:29
# -> required by _build/default/bindings/stubs/libdecompress.a
# -> required by _build/default/test/bin/.cram.bindings.t/cram.out
# -> required by alias test/bin/bindings
# -> required by alias test/bin/runtest

It looks like the fact that some solutions could produce this was reported and closed as expected in mirage/decompress#113.

I can reproduce this on the currently released version of dune (on my old Mac laptop, at the moment) with

$ opam switch create 4.14.0
$ opam dune install 3.20.2
$ opam reinstall decompress.1.4.0 --with-test --verbose

note that installation without --with-test succeeds (as the violating base-unix dep is only pulled in during tests) and that installing the latest version of decompress succeeds even with --with-test.

What ever is going on with the dependency cone to violate the forbidden unix library, does not appear to be introduced as a new regression with this dune (tho it is possible something in the ecosystem changed to introduce the problem since the last dune release).

[avsm]

Given the two outstanding issues are triaged (opam-dune-lint has an open PR and the forbidden_libraries issue also exists in the previous release), and that there have been 4 different runs on RCs of dune, it's time to merge this!

Congratulations to Shon on managing his first Dune release :-) You may want to announce this on https://discuss.ocaml.org, where we have a Community category and an announce tag for this purpose.