Upper bounds on the "dune" package prevent dune package management from solving dependencies

[gridbugs]

When dune solves dependencies, it requires that packages in the solution must be compatible with the dune package of the version of dune that's currently running. E.g. when solving a project with dune.3.20.2, only packages compatible with dune.3.20.2 can be chosen as part of the solution.

All the packages in this repo currently have upper bounds on their versions, including dune, which at the time of writing has upper bounds no higher than 3.19.1. This means that the current version of dune (3.20.2) is incapable of choosing any packages from this repository in its solutions, as they claim to be incompatible with any version of dune above 3.19.1.

I think it's correct that dune adds this constraint when solving dependencies, otherwise packages that are truly incompatible with the current dune won't build, so it's better to try to choose alternative packages that are compatible if possible.

I also think that it's generally good that packages in this repo have upper bounds on their version numbers.

I think a suitable fix for this issue would be to change the upper bounds on all the dune dependencies to < 4.

[hannesm]

Thanks for reaching out. For archiving, it is crucial that we add upper bounds. Given that dune is being developed, I don't think dune < 4 is a good solution, taking into account that dune has broken existing projects every now and then (i.e. 3.7.0).

Also, my impression is that opam itself copes very nicely with these constraints. I guess it shouldn't be too hard to adapt "dune package management" to use a similar solver to opam, or fix whatever is broken in there to get it working with the opam-repository-archive.

Of course, this is only my voice -- there is as well the group of opam-repository maintainers who may have an opinion.

[hannesm]

Ah, sorry, misread. I suspect the solver is not the issue here. But I still stand at the point that handling "dune" dependencies specially and marking them with "< 4" is a very fragile solution.

[gridbugs]

The reason this doesn't affect opam is because opam is not an opam package. Packages in this repo generally don't express an upper bound on the version of opam they are compatible with (e.g. with available: opma-version <= 2.2.0) whereas they do for dune, which of course is an opam package.

If dune 3.20.2 sees a package which claims to require Dune <= 3.19.1, do you think it's correct for dune to exclude such a package from solutions?

[shonfeder]

For archiving, it is crucial that we add upper bounds.

I'm not sure why that is actually. In the archiving policy, the only point that touches on this is:

Any dependencies without upper bounds will have upper bounds added, recording the latest available version of the dependency in the primary repo at the time of archiving.

I can see why it would be useful to record that information, but upon reflection, it seems to me that we may be abusing the dependency constraint to record useful metadata that doesn't actually reflect compatibility, but instead imposes a pessimistic constraint while meaning to just record a known correct solution?

I guess the issue is that we don't include the archive in the CI runs (that's part of the point), and so can't catch breakages that would lead us to add upper bounds only while necessary. As a result, we are making a pessimistic assumption whenever we archive that no future versions will be compatible with the archived package?

On dune's side, perhaps when it is handling package management it should ignore dependency info on itself?

[gridbugs]

On dune's side, perhaps when it is handling package management it should ignore dependency info on itself?

That sounds risky to me. When dune generates opam files it doesn't set an upper bound on the version of dune in the dependencies, so if an upper bound is present I'd assume it was put there deliberately and for good reason.

[shonfeder]

What is the risk? It seems to me that the worst that could happen would be the package would not be buildable or installable with dune? But this is just what a detected constraint on a lower version of dune what do anyhow, right? Only the the latter is pessimistic.

Isn't it a bit weird IMO to have a package that is incompatible with recent versions of a package manager?

[gridbugs]

Yes the risk is that dune will try to build packages that aren't buildable. But you're right that currently no packages with an upper bound on dune are buildable with a later dune, whereas if dune was to ignore that constraint then at least some and probably most such packages would be buildable. The ones that aren't buildable would fail to build with a possibly less-helpful error than a constraint error, but I guess dune could optimistically attempt to build such packages and if building fails then let the user know that it might be related to the constraint? There's also a small risk that a package successfully builds, but in a subtly incorrect way that manifests at runtime, though I can't imagine a practical situation where this would happen by accident.

[raphael-proust]

If dune 3.20.2 sees a package which claims to require Dune <= 3.19.1, do you think it's correct for dune to exclude such a package from solutions?

the metadata in this repository is related to opam package management, i don't understand how that should affect dune package management

the opam files recorded in this archive have bounds on their dependencies, including upper bounds on dune. but they point to tarballs which typically have a dune-project file with no bounds on dune (and, in general, looser bounds on all their deps because the opam repo bounds are refined when the opam repository ci fails and the opam repository managers act upon it).

is the issue arising because you are bootstrapping a repository of dune packages using the opam packages' metadata? or is dune package management otherwise reading opam package metadata? basically: through what mechanism is dune affected by the contents of the opam files indicating a constraint on the opam package dependency on dune? i'm trying to understand this so i can reply more accurately to your questions and comments.

depending on your use case (e.g., copying package metadata for dune packages), then i'd suggest you make a script which instead copies the bounds based on the state inside the main repo at the commit that the package was archived (as indicated in the x-opam-repository-commit-hash-at-time-of-archival field).
archived package → commit hash of main repo at time of archival → last known working loose bounds → whatever you want to use these for

---

on a more general note, the archive repository is a way to preserve metadata about unmaintained software in a way which preserve isntallability as per when it was archived. the main intended use case is "you add this repo, you type opam install somepackage.someoldversion, it's installed". archeology made trivial.

you seem to be hinting at a slightly different use case where one would use the content of the archive repo within a current ongoing project. as in working on a current ongoing project that depends on an archived package?

if your project is current and ongoing, it shouldn't depend on archived packages. you can:

• ask for the archived project you depend on to be unarchived (i.e., moved back to the main repo)
  • at this point the upperbounds can be removed
  • the CI will ensure the bounds provide a good installability story for all users
  • you'll have to convince the maintainer to put the effort into maintaining this specific version or take upon the maintenance yourself

[shonfeder]

the metadata in this repository is related to opam package management, i don't understand how that should affect dune package management

dune's package management can use this package data. In general, afaiu the aim is to have a repository of high quality package data, that can potentially be used by different tools. So dune devs have an interest in the quality of and properties of the data format.

on a more general note, the archive repository is a way to preserve metadata about unmaintained software in a way which preserve isntallability as per when it was archived

For the record, In our dev meeting, I voiced the opinion that we already have this data without the addition of upper bounds, because we also record the commit hash of the repo at the time of archiving. But I was not able to sway others that this would suffice :) -- I am OK with that.

the main intended use case is "you add this repo, you type opam install somepackage.someoldversion, it's installed". archeology made trivial

I noted that this will only stay on the happy path so long as the lower bounds don't induce dependency hell in conflict with some other package you may want installed in your environment.

But I don't think there is one best solution there, just tradeoffs.

w/r/t to dune package management, I still hold the view that dune should not check version constraints of the dune dependency in packages when it is acting as a package manager. But this is an issue for the dune repo.

[raphael-proust]

dune's package management can use this package data. In general, afaiu the aim is to have a repository of high quality package data, that can potentially be used by different tools. So dune devs have an interest in the quality of and properties of the data format.

Indeed the opam-repository is an open source license (CC01) project and it's entirely acceptable for any project to re-use the metadata in the opam-repository for their own packaging (or other) needs. But it doesn't imply that the metadata in the opam-repository should be changed to accomodate other package management systems.

If the use of the metadata through other package management systems reveals existing issues with the metadata we can address them. On the other hand, if the metadata is fine for opam but unsuitable for other package management system we would only change the metadata if it didn't break the intended use of opam.

More generally, I was wondering through which mechanism does dune package management ends up depending on the package metadata of the opam-repository, or in this case of the opam-repository archive. As suggested in my message, there is a work-around for finding the looser set of constraints based on the content of the archived package. The suggestion might or might not work depending on how dune consumes the opam-repo archive.

I voiced the opinion that we already have this data without the addition of upper bounds, because we also record the commit hash of the repo at the time of archiving

The record of the commit hash also gives you the looser set of constraints:
cat the archived package
grep the x-opam-repository-commit-hash-at-time-of-archival key
sed the value out of that key
use git show $commithash:$packagename on the main repo

I think this is simpler than recovering the tighter set of constraints which would look something like:
cat the archived package
grep the x-opam-repository-commit-hash-at-time-of-archival key
sed the value out of that key
for each of the dependencies of the package do
_ list the subdirs in packages/$packagename/ at revision $comnmithash
_ find the highest version (not sure what cli tool there is for it, maybe just get the code from opam and package it in a tiny cli)
oh and you need to do the same thing to each of the packages transitively!

This seems unreasonable and so you instead have to recommend to users to
get the comnmit hash
add a remote that's the main opam repo with a #$commithash suffix
make sure the added remote has higher priority
install the package

So the question is not whether something is possible (each approach make the other possible), but rather: what do we want to make frictionless. Or in your own words:

just tradeoffs

as per

I noted that this will only stay on the happy path so long as the lower bounds don't induce dependency hell in conflict with some other package you may want installed in your environment.

I don't see how that's better with a looser set of bounds. Say you want to install a recent package that depends on ppxlib.0.36+ and an archive package that depends on ppxlib and was archived before ppxlib.0.36+ was published. Then you'd get some compilation errors in a dependency of the archived package. If you try to mix-and-match you're likely to get errors.

Not only that, but with the looser set of bounds, you are likely to get errors even if you just want to install an archived package, with no other packages whatsoever. Indeed, at some point, a breaking release of ppxlib, cmdliner, or whatever other package makes it into the archive repo. When this happens, all packages in there which depend on this breaking pacakge need to have an upperbound to stay buildable.