Revamping the cross-platform support

[kubukoz]

(moved from Discord)

I saw the docs, and so far the most comprehensive example for cross-building with platform axes is https://mill-build.org/mill/scalalib/web-examples.html#_publishing_cross_platform_scala_modules and https://mill-build.org/mill/scalalib/web-examples.html#_publishing_cross_platform_scala_modules_alternative.

and it appears to be a little bit too verbose... like, for each module you need to duplicate this amount of code inside

  object jvm extends Cross[JvmModule](scalaVersions)
  trait JvmModule extends Shared {
    object test extends ScalaTests with SharedTestModule
  }

  object js extends Cross[JsModule](scalaVersions)
  trait JsModule extends SharedJS {
    object test extends ScalaJSTests with SharedTestModule
  }

and that's just 2 platforms - most OSS projects now have 3.

If this isn't a lot, let's add cross-module dependencies:

  object jvm extends Cross[JvmModule](scalaVersions)
  trait JvmModule extends Shared {
    def moduleDeps = Seq(bar.jvm())
    def ivyDeps = super.ivyDeps() ++ Agg(ivy"com.lihaoyi::upickle::3.0.0")

    object test extends ScalaTests with SharedTestModule
  }

  object js extends Cross[JsModule](scalaVersions)
  trait JsModule extends SharedJS {
    def moduleDeps = Seq(bar.js())

    object test extends ScalaJSTests with SharedTestModule
  }

is it me or is this not going to scale? Having to configure each variant individually seems way more troublesome than "just" using sbt-projectmatrix and having entire rows configured with a single line.

Is this something you'd be interested in revisiting in future Mill versions?

[kubukoz]

At this point, the problematic parts I see are:

1. the duplication of the 4 lines for each platform in each project (5 projects * 3 platforms == 60 extra lines). Ideal solution: 1 line per platform per project, e.g.

trait FooModule extends ... {
  object jvm extends JVMModule
  object js extends JSModule
  object native extends NativeModule
}

Perhaps this is already possible and I just didn't find the right way to implement the pattern yet.

Edit: this seems to work, actually...

trait SharedJVM extends Shared {
  object test extends ScalaTests with SharedTestModule
}

trait SharedJS extends Shared with ScalaJSModule {
  def scalaJSVersion = "1.16.0"
  object test extends ScalaJSTests with SharedTestModule
}

val scalaVersions = Seq("2.13.14", "3.3.6")

object bar extends Module {
  // 2 lines per axis, sounds doable
  object jvm extends Cross[JvmModule](scalaVersions)
  trait JvmModule extends SharedJVM

  object js extends Cross[JsModule](scalaVersions)
  trait JsModule extends SharedJS
}

2. declaring dependencies on project level rather than inside each platform, e.g. trait FooModule extends ... { def crossModuleDeps = Seq(bar) }

[kubukoz]

Another thing which @ghostbuster91 reminded me about: the Cross.ModuleN functionality looks very very nice (the ability to compute a build matrix by using a for comprehension, while filtering out combinations you don't want - amazing stuff), but we found it hard to apply with platforms:

package build
import mill._

val crossMatrix = for {
  crossVersion <- Seq("2.10", "2.11", "2.12")
  platform <- Seq("jvm", "js", "native")
  if !(platform == "native" && crossVersion != "2.12")
} yield (crossVersion, platform)

object foo extends mill.Cross[FooModule](crossMatrix)
trait FooModule extends Cross.Module2[String, String] {
  val (crossVersion, platform) = (crossValue, crossValue2)
  def suffix = Task { "_" + crossVersion + "_" + platform }
}

def bar = Task { s"hello ${foo("2.10", "jvm").suffix()}" }

The example itself doesn't elaborate any further, and I'm not even sure it's actually possible to combine this example with ScalaJSModule and friends. It would be very very useful!

[ghostbuster91]

👍 The current approach is not only verbose but also potentially error-prone, as cross-module dependency declarations can easily become inconsistent across platforms. In most cases, this will surface as a compile-time error, but in scenarios involving dynamic class loading, it may lead to subtle runtime failures.

While it's possible to mitigate this by introducing additional abstraction layers to express dependencies in a platform-agnostic way, doing so adds complexity. Moreover, given the fast-paced evolution of the Mill project, maintaining such abstractions across version upgrades can become particularly challenging.

[lihaoyi]

Cross modules are a poor fit for this use case, because cross modules are by definition homogenously typed and dynamically enumerated, while cross platform modules are typically heterogenouslyntyped and statically enumerated.

I think the solution to get a more SBT-esque ease of use would be to define a ScalaJsJvmModule that inherits all the members from both ScalaModule and ScalaJsModule, and define .jvm and .js submodules that use those members as their defaults but can be overridden on each submodule as necessary.

Can definitely be done, would take some legwork but I don't see any blockers

[lihaoyi]

Basically such a definition would look something like

object foo extends ScalaJsJvmModule{
  // shared settings
  def moduleDeps = ???
  def scalaVersion = ???
  // optional platform specific settings
  object jvm extends ScalaJsSubModule{
    def moduleDeps = ???
    def ivyDeps = ???
  }
}

[kubukoz]

How about a trait that includes all platforms, and then you can somehow exclude the ones you don't want? Thinking of various combinations of jvm/js/native.

[lihaoyi]

Yes that could work

[lolgab]

I wrote a plugin to simplify cross-platform support. It was before Mill adopted its own PlatformModule.
It needs to be updated to latest mill, but I think the reflection-based approach allowed cleaner builds at that time.
Haven't fully studied the new PlatformModule to compare with current Mill's approach though.

https://github.com/lolgab/mill-crossplatform

[lefou]

It would help in many situations, when ScalaJSModule and ScalaNativeMoudle would just define their own configuration and other specific tasks and avoid overriding and re-using the Jvm-specific parts of ScalaModule. Ideally, they are designed as sub-module-mixins from the start. And we either have a default-platform, in which case we can avoid the {jvm,js,native} submodule or not.

[lolgab]

Then you would need to triplicate and prefix every target, since they need to be compiled separately, need to have separate dependencies, etc.

[lihaoyi]

The approach i mentioned above would basically keep the outer module properties as the defaults, and allow you to override the inner .jvm .js or .native submodules and override their properties if you want to customize the inner submodules (or you can just leave them as the default). That is a bit more tedious bit would avoid having to prefix and duplicate all the properties three times

[lefou]

@lolgab I was rather thinking of some ScalaBaseModule, and each platform specific module being a trait that can be mixed in. Either in-place (when you only have one platform) or as sub-module, when you have more than one platform. Each platform trait could have a reference to baseModule which is ModuleRef(this) in the single-platform case or ModuleRef(outer) in the sub-module case. Of course there are a lot of details I have in mind but am currently not able to line out. But it is almost in the same direction what @lihaoyi just suggested.