Add support for cross-values support at the task-level

[alexarchambault]

I'd like to propose to add support for things along those lines to handle cross-values support:

object myModule extends ScalaModule {
  def scalaVersion = Task.Cross("scala")("2.12.20", "2.13.16", "3.5.2")
}

which currently needs to be written as

object myModule extends Cross[MyModule]("2.12.20", "2.13.16", "3.5.2")
trait MyModule extends ScalaModule with Cross.Module[String] {
  def scalaVersion = crossValue
}

In particular, this moves the concern of handling cross values from the module level to the task level.

IMO, the proposed syntax is more straightforward. Thanks to it, no refactoring is needed when introducing cross values - it's just a matter of adding a new task, or redefining one.

Users could then do things like

$ ./mill 'myModule.compile,scala=2.13.16'

(I'm not sure what would be the best way to pass such cross values on the command-line. This syntax is just a proposal, to set ideas. I'm also not considering default values for these at this point.)

I'm going to post below messages discussing a possible implementation for that, and problems that might arise with it.

[alexarchambault]

Implementation

I think something along those lines should allow to implement that:

• introduce CrossValue or CrossValues, a new implementation of mill.define.NamedTask, for tasks providing cross-values (like scalaVersion in the example above)
• in mill.api.Ctx, add fields for
  • cross-values used by input tasks (or by input tasks' input tasks, etc.)
  • cross-values passed by users but not used by input tasks

About cross-values added in Ctx, the first set (cross values used by inputs) MUST be used by targets to serialize their result on disk. Just like today, when compiling the second example above, we get out/myModule/2.13.16/compile.dest/…, compile evaluations in the first example should disambiguate the value of "scala" used by input tasks. So we could use something along those lines out/myModule/cross/scala/2.13.16/compile.dest/.

If the leaf tasks have values in the second added field in Ctx, that means some cross values were not used. We could either warn users or error out in that case.

As a Task, CrossValues have no input tasks. They simply read their value in the Ctx, and fail if no such value is found, or if it's not among their set of predefined values.

[alexarchambault]

Issues / points to pay attention to, that come to mind

(Possibly sparse) Matrices of cross values

When several cross-values are involved, users might want to restrict the value of one of the cross value depending on the value of the other. This should be achievable with the proposal above, we should allow to implement things like this.

object myModule extends Module {
  // Cross task definition that reads two cross values from the context at once
  def crossValues = Task.Cross.sparse("foo", "bar")(
    ("1.2", "2.3"),
    ("1.3", "3.3"),
    ("1.4", "4.3")
  )
  def fooVersion = Task {
    crossValues()._1
  }
  def barVersion = Task {
    crossValues()._2
  }
}

Cross values with the same name but different value list

A cross value with the same name can be defined at different places (think scalaVersion defined for each ScalaModule). Sometimes, different value sets might be used at different places. The proposal above should allow to fail like today, when no compatible cross value can be found.

object mod1 extends ScalaModule {
  def scalaVersion = Task.Cross("scala")("2.12.20", "2.13.16", "3.6.2")
}
object mod2 extends ScalaModule {
  def scalaVersion = Task.Cross("scala")("2.13.16", "3.6.2")
}

object mod3 extends ScalaModule {
  def scalaVersion = Task.Cross("scala")("2.12.20", "2.13.16", "3.6.2")
  def moduleDeps = Seq(
    mod1,
    mod2
  )
}

This should fail

$ ./mill mod3.compile,scala=2.12.20

This should succeed

$ ./mill mod3.compile,scala=2.13.16

Cross values set by tasks

object mod1 extends ScalaModule {
  def scalaVersion = Task.Cross("scala")("2.12.20", "2.13.16")
}

object thing extends Module {
  // Here we want to compile a cross-compiled module
  // only for a specific hardcoded Scala version. Users don't need
  // to specify that cross value on the command-line when running
  // that command.
  def compile213Stuff() = Task.Command {
    mod1.compile.cross("scala" -> "2.13.16")()
  }
}

// This module isn't cross-compiled, so it tries to
// enforce its Scala version to its moduleDeps beforehand -
// this value doesn't need to be specified on the command-line
// by users in particular
object cli extends ScalaModule {
  private def sv = "2.13.16"
  def scalaVersion = sv
  def moduleDeps = Seq(
    mod1.cross("scala" -> sv)
  )
}

scalaVersion.cross("scala" -> "2.13.16") in the example above basically returns a new Task that updates the context it's passed with new cross values, before passing it to scalaVersion.

If we warn users for unused cross parameters, we should warn here if the cross parameter ends up not being used by mod1.scalaVersion.cross("scala" -> "2.13.16") (not sure how that should be implemented yet)

Making the syntax mod1.cross("scala" -> "2.13.16") valid might require more changes. It basically states that if the module cli is going to evaluate tasks from mod1, it should augment their context with extra cross values. The current signature of JavaModule#moduleDeps (def moduleDeps: Seq[JavaModule]) doesn't really allow to augment references to modules this way. Maybe it could be changed to something like def moduleDeps: Seq[ModuleRef[JavaModule]], and ModuleRef could accept those extra cross values. Then, everywhere JavaModule uses a task defined on one of these modules, it should pass the extra cross values to the task (maybe a helper method on ModuleRef could help for that).

[alexarchambault]

One nice aspect of this proposal is that it's somewhat orthogonal to the current cross modules, so we can have both implementations around for some time. That way, users are not forced to switch to it during a Mill version bump. That should give time to Mill users to switch to the new system, and allow to address any shortcoming that might arise.

Such a cross value system makes dealing with cross-values more user friendly IMO, and, perhaps more subjectively, removes what feels to me like an idiosyncrasy in Mill: its handling of cross values, which feels at odd with the overall sleekness of the Mill task system.

IMO, such cross values should help increase the adoption of Mill in the Scala open source ecosystem, but also by Mill users more broadly.

In the Scala open source ecosystem, the vast majority of projects support multiple Scala versions via cross-compilation. Up to now, the way Mill handled cross values felt somewhat less user friendly than how sbt handles those (which is a pity, given how much more convenient the Mill task system is). This proposal aims at changing that.

For Mill users at large, it lowers the cost of adding / removing cross values: no need to introduce intermediary traits if one adds a cross value type, it's just a matter of defining a new task.

[lihaoyi]

I'm in favor of this proposal in general. Bazel provides something similar with Configurations and Transitions, and in general it works great. It definitely will be a big improvement in ergonomics over use Cross.Modules all the time, and as you said could live alongside it with Cross.Modules still useful for things like per-folder modules or test suites (e.g. Mill's own integration tests)

Namespacing

In terms of implementation, it would mean that tasks are no longer uniquely keyed by their Segments selector, but by a tuple of (Segments, values-assigned-to-all-transitive-upstream-Cross-tasks). This would take some refactoring of the internal code paths and on-disk representation of the out/ folder, but it seems it could be done.

As far as syntax goes, at the declaration site I think we might be able to get by without introducing another namespace for the cross values, by using their task selector as the name. e.g.

def scalaVersion = Task.Cross("2.12.20", "2.13.16", "3.5.2")
object myModule extends ScalaModule {
  def scalaVersion = build.scalaVersion()
}

$ ./mill 'myModule.compile,scalaVersion=2.13.16'

Or inline

object myModule extends ScalaModule {
  def scalaVersion = Task.Cross("2.12.20", "2.13.16", "3.5.2")
}

$ ./mill 'myModule.compile,myModule.scalaVersion=2.13.16'

CLI Syntax

The syntax is a big clunky and we could probably come up with something better? e.g.

$ ./mill myModule.scalaVersion=2.13.16 myModule.compile

In either case, we could make missing the myModule.scalaVersion=2.13.16 configuration either an error, or allow the user of Task.Cross to provide a default.

In-Code References

As is the case with Bazel's transitions, any task referencing another task should be able to set a cross value, e.g.

def scalaVersion = Task.Cross("2.12.20", "2.13.16", "3.5.2")
object myModule extends ScalaModule {
  def scalaVersion = build.scalaVersion()
}

def allJars = Task{
  os.copy(myModule.jar(build.scalaVersion := "2.12.20").path, Task.dest / "out-212.jar")
  os.copy(myModule.jar(build.scalaVersion := "2.13.16").path, Task.dest / "out-213.jar")
  os.copy(myModule.jar(build.scalaVersion := "3.5.2").path, Task.dest / "out-35.jar")
}

A task should be able to reference an upstream task with multiple different, as shown above, which would compile myModule and build it's jar three times for 2.12.20/2.13.16/3.5.2 and copy all three jars into out/allJars.dest/. As tasks are now uniquely identified (Segments, values-assigned-to-all-transitive-upstream-Cross-tasks), this would not result in any conflict as myModule.compile,scalaVersion=2.12.20, myModule.compile,scalaVersion=2.13.16, etc. would each have different outputs and different paths on disk for their .dest/.json/.log files

[alexarchambault]

As far as syntax goes, at the declaration site I think we might be able to get by without introducing another namespace for the cross values, by using their task selector as the name. e.g.

That could work, except it might be a problem if different modules support different Scala versions:

def scalaVersion = Task.Cross("2.12.20", "2.13.16", "3.5.2")
def restrainedScalaVersion = Task.Cross("2.12.20", "2.13.16")
object mod1 extends ScalaModule {
  def scalaVersion = build.scalaVersion()
}
object mod2 extends ScalaModule {
  def moduleDeps = Seq(mod1)
  def scalaVersion = build.restrainedScalaVersion()
}

The Scala version being restrainedScalaVersion for mod2, it needs a way to pass it as just scalaVersion to the tasks of mod1 that it evaluates.

[alexarchambault]

But maybe the name could be overridden somehow

def scalaVersion = Task.Cross("2.12.20", "2.13.16", "3.5.2")
def restrainedScalaVersion = Task.Cross("2.12.20", "2.13.16").withName(scalaVersion.name)

So that build.restrainedScalaVersion would just pick the value of build.scalaVersion that's around, and evaluating mod2.scalaVersion with build.scalaVersion=3.5.2 would fail.

[lihaoyi]

Yes that definitely seems like an issue: we want different modules to be able to use a shared name for the cross task, while allowing different sets of values.

One way we could resolve that would be to have separate def scalaVersion = Task.CrossKey() that just provides the identifier, then Task.Cross(scalaVersion)("2.12.20", "2.13.16") would provided the set of values usable at this point.

There may be some other ways of doing it too, might need to see what else we could come up with