Make `out/` folder contents (more) reproducible and filesystem layout agnostic (1500USD bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 1500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

The goal of this ticket is to make the out/ folder contents more reproducible, such that it contains the same bytes and hashes regardless of the user's filesystem layout outside of that folder. This is would allow re-using the out/ folder as a build cache between different machines that may have the checkout in different place (e.g. /Users/alice/my-repository vs /Users/charlie/my-repository), both coarse grained (e.g. by sending over a zip file) and fine grained (via the bazel remote cache protocol)

The main thing that needs to happen is that every os.Path and mill.api.PathRef that is serialized within a "known" directory needs to be normalized to a path relative to an abstract reference to that known directory. e.g.

• /Users/alice/my-repository/out/foo/bar.dest/qux should be serialized as $WORKSPACE/out/foo/bar.dest/qux
• /Users/lihaoyi/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar should be serialized as $COURSIER_CACHE/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar
• /Users/alice/thing-outside-repository should be serialized as $HOME/thing-outside-repository

AFAIK the necessary known roots should all be available globally (e.g. mill.api.workspace.WorkspaceRoot.workspaceRoot, os.home, sys.env("COURSIER_CACHE")). It should be easy enough to add to the serialization logic:

• mill.api.PathRef serialization

  mill/main/api/src/mill/api/PathRef.scala

  Lines 175 to 197 in e0a2c93

  	implicit def jsonFormatter: RW[PathRef] = upickle.default.readwriter[String].bimap[PathRef](
  	p => p.toString(),
  	s => {
  	val Array(prefix, valid0, hex, pathString) = s.split(":", 4)
  	val path = os.Path(pathString)
  	val quick = prefix match {
  	case "qref" => true
  	case "ref" => false
  	}
  	val validOrig = valid0 match {
  	case "v0" => Revalidate.Never
  	case "v1" => Revalidate.Once
  	case "vn" => Revalidate.Always
  	}
  	// Parsing to a long and casting to an int is the only way to make
  	// round-trip handling of negative numbers work =(
  	val sig = java.lang.Long.parseLong(hex, 16).toInt
  	val pr = PathRef(path, quick, sig, revalidate = validOrig)
  	validatedPaths.value.revalidateIfNeededOrThrow(pr)
  	pr
  	}
  	)

• os.Path serialization

  mill/main/api/src/mill/api/JsonFormatters.scala

  Lines 27 to 31 in e0a2c93

  	implicit val pathReadWrite: RW[os.Path] = upickle.default.readwriter[String]
  	.bimap[os.Path](
  	_.toString,
  	os.Path(_)
  	)

Apart from PathRef and Path, we will also need to deal with:

• Files in out/ which are naturally non-deterministic: mill-profile.json, mill-chrome-profile.json, mill-server/* and mill-no-server/*, etc.

• Modified times are also expected to vary. These may need to be zeroed out in the process of making zip and jar files such that they do not affect the byte contents, and ignored as part of any equivalence comparison

• Any foo.json files belonging to workers can also be expected to differ since they contain the toString of the worker, and may need to be renamed to foo.worker.json or similar to make them identifiable.

• There will also be inherent differences between files generated on different platforms (e.g. native binaries). This is fine for now, and likely unavoidable.

• There may be other files that need to be made reproducible that are not listed here

The success criteria would be a test in integration/feature/ that:

• Copies the code in example/scalalib/web/5-webapp-scalajs-shared into two separate subfolders.
  • The choice of example/scalalib/web/5-webapp-scalajs-shared is somewhat arbitrary, but should give us good coverage of a variety of Mill module and task types, exercising a wide range of code paths
• Runs ./mill runBackground && ./mill clean runBackground && ./mill jar && ./mill assembly in each folder
  • (one with a custom COURSIER_CACHE and -Duser.home passed in),
• Does a file-by-file and byte-for-byte comparison against the two outfolders with some normalization criteria (ignoring the expected-to-differ files and ignoring mtimes), to assert that the out/ folder is byte-for-byte identical

Related issues with prior discussion:

• Reuse out folder among different machines/directories  #2101
• Remote Caching #2153
• Support remote cache #1886

[rahat2134]

Trying this issue, it looks interesting...

[lihaoyi]

@rahat2134 got for it! Feel free to ask here if you have any questions

[ajaychandran]

Does reproducibility have to be a property? What if it is defined as a transformation?

A filesystem agnostic image can be created by replacing (string) values with "known env vars" in task output file copies.
At the point of ingestion, a reverse substitution would be applied to recreate the files for a different environment.

[albassort]

I would be interested in working on this if the issue is still open?

Edit: Upon further reading, I am requesting a lock on the bounty--- for at 1 week. If I don't have anything mostly done by then, you can free it. Seeing as how its from October, I don't think that is an unreasonable amount of time. Thank you.

Update 2:

I have studied the project in the 5 days now since, and have a working prototype. I should, in 2 days have a hopefully, somewhat working prototype. And the time after will be left for writing tests :)

Update 3:

https://github.com/albassort/mill

I predict it will be mostly be done by Wednesday, maybe the test will be done on Wednesday

[sequencer]

Our large Scala projects workaround the reproducibility issues with this hacky tool: https://github.com/Avimitin/mill-ivy-fetcher and finally made mill-based solutions reproducible. It has been fully integrated into chipsalliance/t1.

I think we may help the out dir reproducibility, and wanna hear more suggestions from haoyi.

[albassort]

#4870

PR Submitted