Spurious `error overriding method may not override non-experimental member`

[lihaoyi]

Compiler version

3.7.0-RC1

Minimized code

git checkout pull/4745/head
git fetch origin pull/4745/head:4745
git checkout 4745
./mill -w 'example.large.multi[10-multi-file-builds].local.server'

Sorry this isn't minimized yet, wanted to open the issue for discussion before I invested the effort to do so

Output

[error] -- [E164] /Users/lihaoyi/Github/mill/out/example/large/multi/10-multi-file-builds/local/server/test.dest/worker-0/sandbox/run-1/foo/package.mill:7:7
 [error] 7 │  def ivyDeps = Seq(ivy"com.lihaoyi::mainargs:0.4.0")
[error]   │      ^
[error]   │error overriding method ivyDeps in trait JavaModuleBase of type => mill.define.Target[Seq[mill.scalalib.Dep]];
[error]   │  method ivyDeps of type => mill.define.Target[Seq[mill.scalalib.Dep]] may not override non-experimental member

Somehow, adding a @experimental annotation on the enclosing object at the override site seems to make the problem go away, which seems to be at odds with the error message that suggests the override is already experimental:

lihaoyi mill$ git diff
diff --git a/example/large/multi/10-multi-file-builds/foo/package.mill b/example/large/multi/10-multi-file-builds/foo/package.mill
index 8cf21e39614..3648cf1f9a2 100644
--- a/example/large/multi/10-multi-file-builds/foo/package.mill
+++ b/example/large/multi/10-multi-file-builds/foo/package.mill
@@ -1,7 +1,7 @@
 package build.foo
 import mill._, scalalib._
 
+@scala.annotation.experimental object `package` extends RootModule with build.MyModule {
-object `package` extends RootModule with build.MyModule {
   def moduleDeps = Seq(build.bar.qux.mymodule)
   def ivyDeps = Seq(ivy"com.lihaoyi::mainargs:0.4.0")
 }

Expectation

There are two problems here:

1. All definitions of ivyDeps are compiled with -experimental, so they should all be experimental anyway. This works for the rest of the codebase, not sure why this one case doesn't work

2. Why are we blocking downstream experimental members from overriding upstream non-experimental members? The ability to override things is a core part of the Java/Scala languages and JVM runtime. AFAICT this basically requires any experimental library to have every transitive dependency be experimental, which seems unreasonable

I don't see any mention of overridelimitations in the docs (https://docs.scala-lang.org/scala3/reference/experimental/index.html) so it's unclear what the expected semantics around this restriction are meant to be

This is currently blocking the ability for Mill to experiment with new experimental features scala/improvement-proposals#100 in Scala 3.7.0-RC1

[sjrd]

2. Why are we blocking downstream experimental members from overriding upstream non-experimental members? The ability to override things is a core part of the Java/Scala languages and JVM runtime. AFAICT this basically requires any experimental library to have every transitive dependency be experimental, which seems unreasonable

This at least is definitely by design. If we allowed overriding a non-experimental member with an experimental member, we would break the LSP. You would be able to smuggle a dependency on experimental features from non-experimental code through virtual dispatch.

(There is a related known "hack" to use @deprecated features from non-deprecated code without warnings: override a non-deprecated method with a deprecated one, from which you call the deprecated stuff. For deprecations the consequences are benign, so it's fine; but for experimental, the checks are stricter.)

[lihaoyi]

I guess the question is what is the granularity of experimental? is it attached to values that get passed around, or is it attached to codebases that depend on each other?

If it is the former, what you say makes sense. If it is the latter, then it doesn't.

IMO it should be the latter: the goal is to ensure experimental libraries are only used by other experimental libraries. The goal isnt to prevent people from passing values from one experimental part of their program to another non-experimental part within the same JVM

[lihaoyi]

Put another way, does that mean that it should be impossible to implement a Function0 that calls an experimental feaure? After all, Function0 is a non-experimental interface implemented in the standard library, and once something experimental is wrapped in a function we can pass it anywhere and call it anywhere, including non-experimental code. So experimental features must never be used in lambdas since we cannot guarantee where those lambdas will end up

[sjrd]

No that is not the same. The lambda captures its experimental flag from its environment. You can't create the lambda if you cannot call the method that defines it, so it's all good. The apply method of the lambda remains non-experimental.

[lihaoyi]

Added a mitigation step to the issue description. TBH what's going on doesn't seem to make sense to me: it's giving me an error that I'm trying to override a non-experimental def with an experimental override, but then when I add a @experimental annotation on the object enclosing the override the error goes away? And all code in both library and test is compiled with -experimental anyway, so it shouldn't matter