Lock the plugins, build time dependencies

[roberth]

What are you suggesting?

I think sbt-dependency-lock should lock the versions of used build plugins.
Sbt plugins versions can obviously have a significant effect on the build output, so they should be included in the lock file.

This may be a hard problem to solve.

How will this benefit other users?

This will make sure that they will notice changes to their build system. Furthermore, it works towards making the build hermetic; accounting for all inputs that affect the result.

Is there a workaround that can do this now?

Another tool, Sbtix has been reasonably successful at achieving this. Its goal is to make the build completely hermetic, proven by running inside the Nix package manager, which enforces hermeticity.
Its main limitation is that it requires to be run as a separate tool; not just an sbt plugin in the regular build. The author did intend to fix this problem in a rewrite, but did not get around to its completion.

Will this require a lock file change?

This will add more entries to the lockfile; perhaps in a new field. This might either count as an addition or a change.

Other details

Besides benefits for regular sbt users, this project has the potential to solve the packaging of sbt-based projects in Nix, for NixOS, home-manager, etc. It seems that a number of people are interested in this; those participating in zaninime/sbt-derivation#5 and some others I know.

For this to be work, the lock file has to contain all dependencies that will be fetched over the wire. I don't think members of this project need to know anything about Nix. It might turn out to be useful for this project's testing in the future, but not at this stage.

Current state of packaging sbt software with Nix

This is quite a rabbit hole. Feel free to skip this.

At the moment, such software is either not packaged at all, not buildable from source, or built using a fragile solution that does fetch all dependencies but fails whenever the entire local 'm2 repo' doesn't match the recorded Merkle hash. This gives no insight into the cause of the problem, requires a magical hash value to be maintained and is inefficient because it duplicates all dependencies every time.

Sbtix is usable for private projects as long as they can avoid bad combinations of dependency overrides. This is feasible for medium-sized projects but perhaps only when using sbtix from the start (because that helps avoiding aforemention bug). This appears to have made it unsuitable for use in Nixpkgs, the central package repo used by NixOS etc.

Perhaps a combination of an Sbtix-like approach for the plugins and the current sbt-dependency-lock for regular dependencies might be necessary. I am by no means an expert on sbt internals though, so I hope a better solution is possible.

[nrdxp]

note that you can already do this by adding the plugin one layer deeper in the project. That is, to project/project/plugins.sbt where project/plugins.sbt has your normal project plugins in it.

Then you can do something like this in build.sbt to lock everything at once.

addCommandAlias(
  "lockUpdate",
  """;dependencyLockWrite
    |;reload plugins
    |;dependencyLockWrite
    |;reload return""".stripMargin
)

The real issue here, at least as far as Nix is concerned, is that the lockfile doesn't contain a full url to the artifacts in question. Without that one would have to implement a convoluted fetching mechanism that knows where SBT expects these artifacts and attempts to fetch them from several different mirror locations. While not impossible, it'd be much nicer and simpler if the lock format just included a full url. Especially because of sbt/sbt#6479

Another issue I see is that it's hashing using sha1 which should be considered as unsafe at this point, and though I haven't tried, I'm not even sure if it would work with Nix.

Finally, this lock file only specifies the jar's but in order to rebuild a maven repo that sbt can use offline, we'd also need either the pom.xml and/or ivy.xml. Even if the jar exists in an expected location, SBT still errors out if it can't find the metadata file. This bit could fairly easily be worked around though, if we had a full url, since their location is always in a well specified location relative to the jar. Still, some dependencies come maven style, others ivy style, so without knowing which it is, we'd have to try both, so it'd still be nicer to just have it included here.

Another important caveat is that SBT likes to fetch itself and the version of the Scala library it was built against at boot up, so it order to have a fully complete locking mechanism, these dependencies would need to be included as well.

Finally, as just a nicety, I feel it'd be much better if we could just merge all the lock files into one, instead of spreading them all throughout the repo.

With all that said, after trying everything I could think of to get SBT and Nix to play nicely together, including making a hand-rolled plugin resolver, and trying to resurrect Sbtix, I think this plugin is the closest we've gotten to being able to do it, if we could just resolve all the above points, we'd have what we need.