Skip to content

pkg: recognize default dune build commands#14773

Draft
Alizter wants to merge 1 commit into
ocaml:mainfrom
Alizter:push-xwtvotspywxw
Draft

pkg: recognize default dune build commands#14773
Alizter wants to merge 1 commit into
ocaml:mainfrom
Alizter:push-xwtvotspywxw

Conversation

@Alizter
Copy link
Copy Markdown
Collaborator

@Alizter Alizter commented May 28, 2026

Works towards #9983 in the context of #8652.

Supersedes #13592 (by @gridbugs).

Contributes to #12103: the (dune) shortcut expansion runs dune build -p <name> without -j jobs, so packages classified as is_dune_default are no longer subject to spurious rebuilds when the user varies -j. Since this PR broadens detection from one shape to ~67% of latest dune-using packages (per the survey below), it pushes the bulk of dune dependencies onto the -j-immune path.

Adds Dune_default_build_commands as the single source of truth for the shapes dune emits in generated opam files. Each shape is an OpamTypes.command list constructed in OCaml, consumed both for generation (via OpamPp.print on OpamFormat.V.command, consistent with how depends/conflicts/etc. are built) and for detection (by structural equality with the same values).

When an opam package's build: field matches one of the canonical shapes and the package carries no install: / patches: / substs: / build-env: steps, the lockfile records the (dune) keyword instead of translating the opam commands into a dune action, enabling dune to build the package in-process rather than spawning a separate dune process per dependency.

Broadens coverage from #13592's single shape to every shape dune itself emits (before_1_11, from_1_11_before_2_7, from_2_7, from_2_9, from_3_0 with subst/sites toggles, and from_3_23 with an exclusive_dir runtest target).

Why the field gate is conservative

The (dune) shortcut at src/dune_rules/pkg_rules.ml expands to a bare dune build -p <name> and cannot reproduce patch application, opam-substitute steps, or a Withenv wrapper for build-env. Packages carrying those fields fall through to the explicit action branch in src/dune_pkg/lock_pkg.ml which emits each step.

A known related gap (dune subst for dev=true packages) is tracked at #13108; this PR doesn't address it.

Refactor

Replaces opam_create.ml's inline string templates with calls to the shared module; output is byte-identical (verified by the existing opam-file-generation snapshot tests at dune-project-meta/basic-generate.t, dune-dep.t, etc., which cover lang versions 1.9 through 3.23).

Empirical justification

Survey of ocaml/opam-repository (2026-05; 29,475 opam files, 16,832 dune-using; 4,518 packages, 2,987 dune-using latest), grouped by build-shape category alone (ignoring install: / patches: / substs: / build-env:):

Shape per opam file (16,832 dune) per package latest (2,987 dune)
no-targets 45.78% 44.63%
extra-step 31.46% 16.37%
default 15.78% 30.80%
missing-runtest 3.44% 3.15%
other 2.58% 3.45%
extra-flag 0.53% 1.07%
missing-doc 0.35% 0.40%
j-literal 0.05% 0.03%
missing-install 0.02% 0.10%

Combined is_dune_default coverage (default + no-targets + from-3-23, gated on empty install / patches / substs / build-env):

  • per opam file: 53.94% (9,080 / 16,832)
  • per package latest: 66.99% (2,001 / 2,987)

The conservative field gate excludes a negligible slice: 87 of 9,167 shape-matching opam files (0.95%) and 24 of 2,025 latest packages (1.19%). The dominant disqualifier is install:; patches / substs / build-env each affect single-digit numbers of latest packages.

extra-step is the largest unmatched bucket and is dominated (~78% per latest) by rm -r vendors followed by dune build, used by tezos to clean vendored deps before building. The remainder is a long tail of ./configure / sh / mv steps. None of these can be reproduced by a bare dune build -p name, so they are correctly excluded.

Tests

  • Positive: every entry in canonical, every no_targets_variants entry, and four from_3_23 instantiations are accepted (show_all).
  • Round-trip property: every shape default_build_command can produce, across the boundary versions (0,0); (1,11); (2,7); (2,9); (3,0); (3,23) × with_subst × with_sites × exclusive_dir, is detected. Catches drift if a future generator branch isn't mirrored in detection.
  • Negative tests for every empirical survey bucket (missing-runtest, extra-step, install-section, extra-flag, missing-doc, j-literal, missing-install, other with two representative shapes) and the field gates (patches, substs, build-env).
  • Structural edges for from_3_23: wrong filter on @doc, modified install command, --promote-install-files=false without the trailing install command, bare @runtest in a from_3_23-shaped build, @doc before @runtest, wrong filter on subst.
  • show_all failure path is itself pinned, so a regression dumps the offending opam in the test diff.

  • Changelog entry at doc/changes/added/14773.md
  • User-facing documentation updated (none required: behaviour change is internal to lockfile generation)

Adds Dune_default_build_commands as the single source of truth for the
shapes dune emits in generated opam files. Each shape is an
OpamTypes.command list constructed in OCaml, consumed both for
generation (via OpamPp.print on OpamFormat.V.command, consistent with
how depends/conflicts/etc. are built) and for detection (by structural
equality with the same values).

When an opam package's build: field matches one of the canonical shapes
and the package carries no install:/patches:/substs:/build-env: steps,
the lockfile records the (dune) keyword instead of translating the opam
commands into a dune action, enabling dune to build the package
in-process rather than spawning a separate dune process per dependency.

The conservative field gate is necessary: the (dune) shortcut at
src/dune_rules/pkg_rules.ml expands to a bare 'dune build -p <name>',
which cannot reproduce patch application, opam substitute steps, or a
Withenv wrapper for build-env. Packages carrying those fields fall
through to the explicit action branch in src/dune_pkg/lock_pkg.ml
which emits each step.

Replaces opam_create.ml's inline string templates with calls to the
shared module; output is byte-identical (verified by existing
opam-file-generation snapshot tests).

Builds on ocaml#13592 by gridbugs, broadening coverage from one shape to
every shape dune itself emits (before_1_11, from_1_11_before_2_7,
from_2_7, from_2_9, from_3_0 with subst/sites toggles, and from_3_23
with an exclusive_dir runtest target).

---

Survey of ocaml/opam-repository (2026-05; 29,475 opam files,
16,832 dune-using; 4,518 packages, 2,987 dune-using latest), grouped
by build-shape category alone (ignoring install:/patches:/substs:/
build-env:):

                            per opam file     per package (latest)
                            (16,832 dune)     (2,987 dune)
     no-targets               45.78%            44.63%
     extra-step               31.46%            16.37%
     default                  15.78%            30.80%
     missing-runtest           3.44%             3.15%
     other                     2.58%             3.45%
     extra-flag                0.53%             1.07%
     missing-doc               0.35%             0.40%
     j-literal                 0.05%             0.03%
     missing-install           0.02%             0.10%

Combined is_dune_default coverage (default + no-targets + from-3-23,
gated on empty install/patches/substs/build-env):
  per opam file        53.94% (9,080 / 16,832)
  per package latest   66.99% (2,001 /  2,987)

The conservative field gate excludes a negligible slice: 87 of 9,167
shape-matching opam files (0.95%) and 24 of 2,025 latest packages
(1.19%). The dominant disqualifier is install:; patches/substs/
build-env each affect single-digit numbers of latest packages.

"extra-step" is the largest unmatched bucket and is dominated (~78%
per latest) by 'rm -r vendors' + 'dune build', used by tezos to clean
vendored deps before building. The remainder is a long tail of
./configure / sh / mv steps. None of these can be reproduced by a bare
'dune build -p name', so they are correctly excluded.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter Alizter force-pushed the push-xwtvotspywxw branch from e342424 to 263bf1f Compare May 28, 2026 18:03
@Alizter Alizter requested a review from rgrinberg May 28, 2026 18:19
@Alizter Alizter marked this pull request as draft May 29, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant