Decoupling of Mill core and libs bundles

[lefou]

As of now, Mill releases always contain the core tool as well as a lot of bundled language and tool libraries. While it is very convenient to develop and use, it also limits how fast each part can evolve. Providing some degree of binary compatibility means, we can't adapt specific languages at a different pace as the core runtime, and vice versa.

But essentially, this tight coupling isn't necessary. The Mill launcher, daemon, CLI, and the various integrations like javalib, scalalib or kotlinlib, to name a few, don't need to be coupled that tight. Especially, the various language libs could have a faster release speed when it comes to binary compatibility. We could improve the user experience faster by cutting new minor and major versions and deprecate blocking stuff faster. Since Mill 1.0 release, we have dropped wanted changes multiple times because of binary compatibility issues in the modules, but not in the core APIs of Mill. We also added a lot of cruft and workarounds to keep some compatibility but at the price of clarity and maintainability. Given, that Mill modules are Scala traits and changes to traits often require recompilation for all downstream derives, we are technically limited.

I therefore propose to split up the language modules (Java, Scala, Kotlin, ScalaJS, Scala Native, Groovy) from the Mill core platform (runner, evaluator, cli, launcher, daemon, and the core API including tasks and modules). This means we would have different versioning and binary compatibility guarantees for each.

I think for most of the issues I typically witnessed it's sufficient to keep the various JVM languages in the shared release, so we essentially end up with three release bundles:

• Mill core
• Mill libs
• Mill contrib

Having the freedom to bump Mill libs from e.g. 1.1.5 to 2.0 to apply cleanups and improvements, also gives users the freedom to stay on Mill libs 1.x for a while and choose the best date for the switch. And in case they need to stay on Mill libs 1.x, they could still update Mill core to the newest compatible version to profit from improvements and fixes.

[lefou]

So, how should the usability story look like? We want to keep the batteries included experience.

We no longer want to implicitly add the mill-libs dependency equivalent to the classpath. Instead, it should be explicitly added to mvnDeps of the meta build. By default, we add it with a preconfigured version. but if a user adds mill-libs with a newer (yet compatible) version, coursier should reconcile the version to use that instead.

[lihaoyi]

Since Mill 1.0.0 the mill-libs artifact and it's dependencies have stopped being special-cased as unmanaged assembly jars, and should already be resolved individually via mvnDeps and Coursier. So I think we should already be able to add a //| mvnDeps: header to specify the same artifact with a newer version, and it should get picked up and just work

[lefou]

I didn't find any mvnDeps override in MillBuildRootModule, so I assume we still hardcode it somewhere.

[lefou]

Oh, it's in mandatoryMvnDeps.

[lefou]

So, there are no real technical limitations at this point. But we still need to discuss, agree on and plan the split. And of course, we need to adapt the build and release process.

[lihaoyi]

Should be over here I think

mill/runner/meta/src/mill/meta/BootstrapRootModule.scala

Line 52 in a78ba93

mvn"com.lihaoyi::mill-libs:${Versions.millVersion}",

[lihaoyi]

Splitting libs definitely seems like a big thing that we'd want to experiment with a bunch before committing to it. It's not obvious whether it's a good idea or not

[lefou]

Now, if done right, it's definitely something I want. E.g. the evolution of features that resided in external plugins is/was definitely easier. I assume, that the Mill core APIs that affect the libs won't change that dramatically, so having a longer maintenance level for that seems plausible.

And I think we still have a lot of refactoring to do in the libs part and these would require definitely more than one overhaul round until they're settled. And I assume, with every new language or feature there are new needs for change and improvement. Having the liberty to cut a new major release without the need to drop support for the previous version would make things easier, but also raises the maintenance overhead. But that should have been clear from the beginning, as popular build tools always need to be bleeding edge and support the legacy as well.