pkg: recognize default dune build commands#14773
Draft
Alizter wants to merge 1 commit into
Draft
Conversation
Adds Dune_default_build_commands as the single source of truth for the shapes dune emits in generated opam files. Each shape is an OpamTypes.command list constructed in OCaml, consumed both for generation (via OpamPp.print on OpamFormat.V.command, consistent with how depends/conflicts/etc. are built) and for detection (by structural equality with the same values). When an opam package's build: field matches one of the canonical shapes and the package carries no install:/patches:/substs:/build-env: steps, the lockfile records the (dune) keyword instead of translating the opam commands into a dune action, enabling dune to build the package in-process rather than spawning a separate dune process per dependency. The conservative field gate is necessary: the (dune) shortcut at src/dune_rules/pkg_rules.ml expands to a bare 'dune build -p <name>', which cannot reproduce patch application, opam substitute steps, or a Withenv wrapper for build-env. Packages carrying those fields fall through to the explicit action branch in src/dune_pkg/lock_pkg.ml which emits each step. Replaces opam_create.ml's inline string templates with calls to the shared module; output is byte-identical (verified by existing opam-file-generation snapshot tests). Builds on ocaml#13592 by gridbugs, broadening coverage from one shape to every shape dune itself emits (before_1_11, from_1_11_before_2_7, from_2_7, from_2_9, from_3_0 with subst/sites toggles, and from_3_23 with an exclusive_dir runtest target). --- Survey of ocaml/opam-repository (2026-05; 29,475 opam files, 16,832 dune-using; 4,518 packages, 2,987 dune-using latest), grouped by build-shape category alone (ignoring install:/patches:/substs:/ build-env:): per opam file per package (latest) (16,832 dune) (2,987 dune) no-targets 45.78% 44.63% extra-step 31.46% 16.37% default 15.78% 30.80% missing-runtest 3.44% 3.15% other 2.58% 3.45% extra-flag 0.53% 1.07% missing-doc 0.35% 0.40% j-literal 0.05% 0.03% missing-install 0.02% 0.10% Combined is_dune_default coverage (default + no-targets + from-3-23, gated on empty install/patches/substs/build-env): per opam file 53.94% (9,080 / 16,832) per package latest 66.99% (2,001 / 2,987) The conservative field gate excludes a negligible slice: 87 of 9,167 shape-matching opam files (0.95%) and 24 of 2,025 latest packages (1.19%). The dominant disqualifier is install:; patches/substs/ build-env each affect single-digit numbers of latest packages. "extra-step" is the largest unmatched bucket and is dominated (~78% per latest) by 'rm -r vendors' + 'dune build', used by tezos to clean vendored deps before building. The remainder is a long tail of ./configure / sh / mv steps. None of these can be reproduced by a bare 'dune build -p name', so they are correctly excluded. Signed-off-by: Ali Caglayan <alizter@gmail.com>
e342424 to
263bf1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Works towards #9983 in the context of #8652.
Supersedes #13592 (by @gridbugs).
Contributes to #12103: the
(dune)shortcut expansion runsdune build -p <name>without-j jobs, so packages classified asis_dune_defaultare no longer subject to spurious rebuilds when the user varies-j. Since this PR broadens detection from one shape to ~67% of latest dune-using packages (per the survey below), it pushes the bulk of dune dependencies onto the-j-immune path.Adds
Dune_default_build_commandsas the single source of truth for the shapes dune emits in generated opam files. Each shape is anOpamTypes.command listconstructed in OCaml, consumed both for generation (viaOpamPp.printonOpamFormat.V.command, consistent with how depends/conflicts/etc. are built) and for detection (by structural equality with the same values).When an opam package's
build:field matches one of the canonical shapes and the package carries noinstall:/patches:/substs:/build-env:steps, the lockfile records the(dune)keyword instead of translating the opam commands into a dune action, enabling dune to build the package in-process rather than spawning a separate dune process per dependency.Broadens coverage from #13592's single shape to every shape dune itself emits (
before_1_11,from_1_11_before_2_7,from_2_7,from_2_9,from_3_0with subst/sites toggles, andfrom_3_23with anexclusive_dirruntest target).Why the field gate is conservative
The
(dune)shortcut atsrc/dune_rules/pkg_rules.mlexpands to a baredune build -p <name>and cannot reproduce patch application, opam-substitute steps, or aWithenvwrapper forbuild-env. Packages carrying those fields fall through to the explicit action branch insrc/dune_pkg/lock_pkg.mlwhich emits each step.A known related gap (
dune substfordev=truepackages) is tracked at #13108; this PR doesn't address it.Refactor
Replaces
opam_create.ml's inline string templates with calls to the shared module; output is byte-identical (verified by the existing opam-file-generation snapshot tests atdune-project-meta/basic-generate.t,dune-dep.t, etc., which cover lang versions 1.9 through 3.23).Empirical justification
Survey of
ocaml/opam-repository(2026-05; 29,475 opam files, 16,832 dune-using; 4,518 packages, 2,987 dune-using latest), grouped by build-shape category alone (ignoringinstall:/patches:/substs:/build-env:):no-targetsextra-stepdefaultmissing-runtestotherextra-flagmissing-docj-literalmissing-installCombined
is_dune_defaultcoverage (default+no-targets+from-3-23, gated on emptyinstall/patches/substs/build-env):The conservative field gate excludes a negligible slice: 87 of 9,167 shape-matching opam files (0.95%) and 24 of 2,025 latest packages (1.19%). The dominant disqualifier is
install:;patches/substs/build-enveach affect single-digit numbers of latest packages.extra-stepis the largest unmatched bucket and is dominated (~78% per latest) byrm -r vendorsfollowed bydune build, used by tezos to clean vendored deps before building. The remainder is a long tail of./configure/sh/mvsteps. None of these can be reproduced by a baredune build -p name, so they are correctly excluded.Tests
canonical, everyno_targets_variantsentry, and fourfrom_3_23instantiations are accepted (show_all).default_build_commandcan produce, across the boundary versions(0,0); (1,11); (2,7); (2,9); (3,0); (3,23)×with_subst×with_sites×exclusive_dir, is detected. Catches drift if a future generator branch isn't mirrored in detection.missing-runtest,extra-step,install-section,extra-flag,missing-doc,j-literal,missing-install,otherwith two representative shapes) and the field gates (patches,substs,build-env).from_3_23: wrong filter on@doc, modified install command,--promote-install-files=falsewithout the trailing install command, bare@runtestin afrom_3_23-shaped build,@docbefore@runtest, wrong filter onsubst.show_allfailure path is itself pinned, so a regression dumps the offending opam in the test diff.doc/changes/added/14773.md