Is it possible to develop a custom Mill plugin to run tasks within Docker containers, rather than on the system executing Mill?

[weaversam8]

Question is in the title. I am curious about Mill as a general purpose build tool, as described in @lihaoyi's recent blog. Many of these build tools (like Earthly) use Docker under the hood to provide a reproducible build environment.

Would it be possible today to create a plugin/trait that did this? If so, how would it interact with other plugins?

E.g. would there be a way to use the existing import pythonlib._ but have the pip install occur within a Docker container? I'm not particularly familiar with Scala, so I'm not sure whether this would be something another plugin could do natively (perhaps with a meta-build?) or whether each other plugin would need a Docker version (e.g. import pythonlib_docker._)

[gamlerhart]

That depends on what you imagine Mill doing with "Docker"/containerization. And what the actual goal is.

• Of course, you can run mill .... inside a docker environment, so that it runs on a standardized environment.
• Of course, you can run a docker ... command as part of a build step.

But I assume you expect some more 'fine-grained' / principled approach. Where there is some `magic' to run parts of the build implicitly in Docker. I don't think there is a good way to write such a plugin. On the other hand, I can imagine for example having Mill 'launch' the build inside Docker. It already runs things inside a deamon. I could imagine that that deamon runs inside a Docker container to 'standardize' the build more.

Anyway, I think the important part is to write your use cases. What problem should it solve? Is there a good example of what you would want to achive?

[weaversam8]

Sure - I think what I'm referring to is the idea of some "steps" within the build running within Docker (e.g. os.call(...) commands running inside a docker container) while retaining integration with the larger mill "graph".

A concrete example:

1. Task A runs on my computer natively and builds a Typescript project into compiled JS artifacts
2. Task B and C build docker containers based on two different Dockerfiles
3. Task D and E each launch one of the docker containers that runs tests on the JS generated by Task A and generates report.json
4. Task F runs on my computer natively and generates final_report.json using report.json from Task D and E.

flowchart LR
TaskB -- generates container 1 for --> TaskD
TaskA -- compiles JS for --> TaskD
TaskA -- compiles JS for --> TaskE
TaskC -- generates container 2 for --> TaskE
TaskD -- generates report.json for --> TaskF
TaskE -- generates report.json for --> TaskF
style TaskD fill:#22F
style TaskE fill:#22F

Loading

I think if each of these tasks were developed from scratch, I think I'd understand how to do it. What I'm confused about is if there's a way to do this that would allow a task written as a part of another plugin (e.g. TypeScriptTests) to execute both natively on my computer and within a docker environment.

In the diagram above, Task D and E would be running TypeScriptTests in a docker container generated by Tasks B and C.

[gamlerhart]

Ok, you want to run a specific task within Docker. The question:

• Is plain 'running' a command inside Docker good enough? Imaginary synax:
  def myTaskE = Task{
  docker.image(...).call(...)
  }
  Then this is possible. At the moment you need to write manually the Docker comment/setup. However, a plugin could provide helper methods to make that easier. Like it could do the mapping of files etc for you.

• Or is it an actual Mill code snippet to run there, not a plain command? Imaginary syntax:

  def myTaskE = Task.withDocker(....){
     // inside a docker environment
  }
  

  This certainly would need a lot more work / native support. And many hard question what the actual meaning / behavior is.

[weaversam8]

I think both could be interesting. "Plain 'running'" would certainly be a good starting point... but it could be nice to provide hooks for support across the board. That way even if plugin developers didn't implement it immediately, there could be a standardized interface for folks to ensure their plugins work under a Docker environment.