Is it possible to invalidate compiled classes if an env var changed?

[Rogach]

Compilation of my project is dependent on some env vars (there's a Scala macro that checks env vars and changes generated code accordingly).

Howerver, currently if env var is changed, already compiled source files are not recompiled - since compiler assumes that nothing changed.

Is there a way to force Mill to recompile all files if task dependency changed?

I tried the following:

  def myEnvInput = Task.Input {
    Task.env.getOrElse("X42", "")
  }
  override def compile = Task {
    println("compile.X42=" + myEnvInput())
    super.compile()
  }

However, super.compile() is oblivious to the fact that override depends on env var - dependencies of super.compile() are the same and so it doesn't recompile.

[lefou]

Not really, since the compiler isn't supposed to forward env-vars to plugins and macros, or however you are using them. Even, if you can convince the compiler to re-run, but do not clean the previous run, you still risk an inconsistent build due to incremental compilation support in Zinc (used for Java and Scala) or Kotlinc, which will very likely not detect the env changes as invalidation reason.

Instead, you should make your environent dependencies explicit. One way I can think of is converting them into compile-time resources. E.g. you could generate a env.properties file as part of the compile resources using the actual env values and then read and use that properties file during compilation.

  def myEnvInput = Task.Input {
    Task.env.getOrElse("X42", "")
  }
  override def compileResources = Task {
    os.write(Task.dest / env.properties", s"compile.X42=${myEnvInput()}")
    super.compileResources() ++ Seq(PathRef(Task.dest))
  }

[lefou]

I overlooked, that your Scala macro is able to check the actual env value. But due to the fact, that we keep a long-running Scala compiler, and Java processes don't get notified about changed environment variables once they started, the only chances you have is to either replace the Zinc-based compiler by invoking scalac directly and as short lived process or Run Mill without a daemon (--no-daemon or --interactive options), which results in Mill not re-using a cached instance.

[Rogach]

I've found a better way (posted as an answer below). But your suggestions helped me a lot, thank you!

[Rogach]

Found an easy way to solve my issue - just inject the env vars into a generated source file.

This nicely handles all problems through standard cache invalidation mechanisms - macros get invalidated because their dependency changed, macro-generated code gets invalidated because macro changed, so all affected items are cleanly recompiled (as a bonus - this works nicely with incremental compilation, source files that don't use these macros are not recompiled).

This approach can probably be used for any other situation where compilation is dependent on env vars (or other inputs).

Full example:

  def myEnvInput = Task.Input {
    Task.env.getOrElse("X42", "")
  }
  override def generatedSources = Task {
    os.write(
      Task.dest / "MacrosEnv.scala",
      s"""// generated on ${System.currentTimeMillis()}
         |object MacrosEnv {
         |  val X42 = "${myEnvInput()}"
         |}
         |""".stripMargin
    )
    super.generatedSources() ++ Seq(PathRef(Task.dest))
  }

[Rogach]

Okay, well, it did work for a while, but seems that was some kind of a fluke - after a while it stopped working, macros module is recompiled but module that is dependent on it isn't.

[Rogach]

What did work was to dump the env vars into the same file as macros definitions - this finally ensures that Macros.scala is recompiled (previously it didn't recompile because outputs would be the same - still the same references to MacrosEnv).

So the final example:

  // in build.mill
  def envX42 = Task.Input { Task.env.getOrElse("X42", "") }
  def macrosTemplate = Task.Source(moduleDir / "tmpl" / "Macros.scala")
  override def generatedSources = Task {
    os.write(
      Task.dest / "Macros.scala",
      os.read(macrosTemplate().path)
        .replace("$X42", envX42())
    )
    super.generatedSources() ++ Seq(PathRef(Task.dest))
  }

  // in modules/macros/tmpl/Macros.scala
  object Macros {
    val X42 = "$X42"
    // ... macro definitions
  }