Very slow resolve step (1-5+ minutes)

[bherw]

Hi, I've been experimenting with migrating a big Maven monorepo to Mill. I found Mill pretty nice to work with so far, and got a lot of the base libraries migrated over without too much pain. I started with mill init and then made myself a Deps class, added some "default" libraries to the mvnDeps of the ProjectBaseModule, and migrated a few .mill.yaml to .mill. I also got a protobuf/grpc source generation plugin built, and it seems to be caching its outputs correctly--a rebuild finishes in < 1s. What I have so far is a build that can compile and test, not much more.

Unfortunately, now that I'm testing the build on some of the modules that build applications (so they have big dependency trees), the resolve step for submodule.test is taking a really long time, on the order of 1-5 minutes, depending on the submodule.

Some context:

• Mill version 1.1.5
• JDK: temurin 25.0.2
• Kotlin: 2.3.20
• about 500K lines of code.
• 88 submodules
• around 170 dependencies
• Mostly Java with Lombok, and a bit of Kotlin.
• My ProjectBaseModule is using KotlinMavenModule and PublishModule.

For comparison:

• maven clean build (including tests) around 2 minutes. Maven 3.9.9
• gradle clean build with tests around 50 seconds. Gradle 9.4.0
• mill test on an application submodule with cached build step: 171s to resolve, 184s total.
• mill test on the entire project: stuck in resolve forever

Based on some testing, and looking at mill-dependency-tree and the chrome profiler the curve on increasing task count seems pretty steep, though I'm not certain if that's the culprit:

mill-dependency-tree.json line count	resolve time
277	145 ms
514	117 ms
1833	16.6 s
4670	85 s
5780	171 s
14512	1361 s

It seems that, given a dependency tree A -> B -> C, if I run A, then run B, B doesn't need to re-resolve and finishes instantly, but then if I run A again, or some unrelated submodule D, then try to run A again, the dependency tree has to be recomputed from scratch. Once it finishes the resolve step it's pretty fast, but it's pretty unusable for a full project build at this point. The last element in the table there is __.test. At 1361 seconds just to resolve, I originally thought the build had hung on the resolve step.

Is this expected behavior or am I doing something obviously wrong? Given the benchmarks you have with mill itself (way more submodules than me) and Netty (around the same line count), I wouldn't expect this weird scaling behaviour. Any ideas for what I could test?

[lihaoyi]

@bherw could you get JProfiler, run a CPU profile of the MillDaemonMain process while its stuck, and upload it here? That would give us a lot finer grained insight into where the time is going, and how we can fix it either in your build or upstream in Mill

[lihaoyi]

14512 tasks isnt a large enough number I'd expect slow resolution, so the problem must be somewhere else in the Mill algorithm or structure of your build.

More details, including the precise commands run that exhibit/do-not-exhibit the slowdowns, would also be helpful for debugging

[lihaoyi]

Also a minimization of the codebase, e.g. just the build files with all source code deleted, would also help me investigate the issue

[bherw]

Hi, thanks for the reply.

I don't have JProfiler on me at the moment, but I ran a quick profile in IntelliJ and got the following:

26% of total time is spent the in moduleDeps and runModuleDeps in the generated object for my utility submodule, which is at the root of the dependency tree--almost everything depends on it either directly or indirectly. This module is using package.mill.yaml format as produced by the init tool; I haven't translated it to programmatic yet.

  def resolveModuleDeps[T <: Module](
      rootModule: Module,
      modulePath: String,
      fieldName: String,
      default: => Seq[T]
  )(implicit ct: ClassTag[T]): Seq[T] = {
    val classLoader = rootModule.getClass.getClassLoader
    // 10% of total time
    val content = os.read(os.resource(using classLoader) / "mill/module-deps-config.json")
    // 57% of total time
    val configFromClasspath = upickle.default.read[Map[String, ModuleDepsConfig]](content)

My guess would be that somehow the classes generated from yaml are missing caching on the moduleDeps, so as the dependency tree grows, it's having to evaluate the dependencies for my utils module once for every time it's referenced transitively.

It's a bit late here, but I'll see if I can get a minimal sample for you next week.

[lihaoyi]

@bherw I tagged a development version 1.1.5-5-511b2c with an attempted fix (#6970), could you try it out and see if it helps?

[bherw]

It worked great, thanks! 👍

Doing rm -r out; ./mill resolve _, the resolve step now completes in (5 sample runs):

• 130 ms
• 134 ms
• 136 ms
• 139 ms
• 141 ms

I think there's still something slightly wonky going on, though. If I kill the mill daemon and run MILL_VERSION=1.1.5-5-511b2c ./mill __.test, mill always recomputes resolvedMvnDeps and test.testForked for a lot of submodules (unsure if it's rerunning all of them or just some), even though no files have been changed. If I don't kill the daemon, and I run the same command several times with the daemon running, then I don't see mill recomputing resolvedMvnDeps anymore, but it does rerun test.testForked on a similar number of submodules, which takes about 5s. That's not a dealbreaker by any means, but it is not the same behavior reported for the netty/mockito tests.

Is it possible that if the test step finds zero test sources to run, then it can't cache the result? I tried adding a deliberately invalid test and it looks like it's not actually executing the tests in src/test/java.

Here's a sample with just the build files if you'd like to take a look: mill-sample.tar.gz

Thanks again.

[bherw]

Oops, needed to use KotlinMavenTests. So it finds my tests now, which is great.

I see mill rerunning all tests each time I run mill __.test. Is that the expected behavior or am I misunderstanding something?

[lihaoyi]

@bherw year __.test is not cached. __.testCached should be

[bherw]

__.testCached indeed is cached. Thanks again. 👍

[bherw]

Just as a followup note, after migrating to package.mill everywhere (which I planned to do anyway so I could centralize the dependency declarations), I have great performance on resolving with 1.1.5. In general, I suspect most large builds would want to do so, so this issue might only affect someone using mill init to do their initial automated migration.

[lefou]

For those who intent to use mill init but want to start with .mill files instead of .mill.yaml, you can use the --declarative false option.