Add assemblyUnzipDirectory to AssemblyOption to use a different directory for unzipping jars

[er1c]

Adding as a draft for just reference for now

This is a draft implementation for #445

• Use hardlinks as basis for copying from unzip cache directory when enabled
• Rebased ontop of Drop sbt 0.13.x support #449

TODO

• Add/update README.md examples
• Create a flag for the hardlink feature? or have a way to explicitly turn it off (or default off)
• Run through my work's CI system for extra validation

[er1c]

Found one issue with the hardlinks, one of my assumptions of the caches being on the same devices is flawed :)

[error] java.nio.file.FileSystemException: /local/data_importer/target/streams/_global/assembly/_global/streams/assembly/2669ae58f340f420d4802cc2289f5ba407c4e2bc_a9a7907ae54d79c51d9dd7e07bc71162aa0c71a2_0289763ca8ca20fbc99cf2a3bcea50af3cd6be74/META-INF/MANIFEST.MF -> /.cache/sbt/v1/assembly-dependencies/2669ae58f340f420d4802cc2289f5ba407c4e2bc_a9a7907ae54d79c51d9dd7e07bc71162aa0c71a2_0289763ca8ca20fbc99cf2a3bcea50af3cd6be74/META-INF/MANIFEST.MF: Invalid cross-device link
[error] 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
[error] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[error] 	at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
[error] 	at java.nio.file.Files.createLink(Files.java:1086)
[error] 	at sbtassembly.AssemblyUtils$.copyFile(AssemblyUtils.scala:174)
[error] 	at sbtassembly.AssemblyUtils$.copyImpl(AssemblyUtils.scala:152)
[error] 	at sbtassembly.AssemblyUtils$.$anonfun$copy$1(AssemblyUtils.scala:138)
[error] 	at scala.Function$.$anonfun$tupled$1(Function.scala:79)

This either needs to be caught and then just copied, or have a toggle earlier up to see if hardlink is supported

[er1c]

Got some corner case that I need to figure out:

[info] Copying shapeless_2.12-2.3.3.jar from unzip cache: /home/docker/.cache/sbt/v1/assembly/dependencies/91e75de52000c787900cf3b917ad2ffe0e5035ea_6041e2c4871650c556a9c6842e43c04ed462b11f_806e468a010b7087977fac295351e79a95d7c832, to: /shared/text_factor/scala/text_tokenizer/target/streams/_global/assembly/_global/streams/assembly/91e75de52000c787900cf3b917ad2ffe0e5035ea_6041e2c4871650c556a9c6842e43c04ed462b11f_806e468a010b7087977fac295351e79a95d7c832, for project: text-factor/text_factor
[info] Calculate mappings...
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Strategy 'deduplicate' was applied to 6 files (Run the task at debug level to see details)
[info] Strategy 'discard' was applied to 417 files (Run the task at debug level to see details)
[info] Strategy 'filterDistinctLines' was applied to 6 files (Run the task at debug level to see details)
[info] Strategy 'rename' was applied to 6 files (Run the task at debug level to see details)
[info] SHA-1: d1fbb4a0902d74b3746362fdb849620f582b548e
[error] java.lang.IllegalArgumentException: requirement failed: Source file '/home/docker/.cache/sbt/v1/assembly/dependencies/346af639499c3d3ed6bb01314995d424e58af9b2_ac435f9d09ce25fc2c3dcade9b1a567466f6245a_806e468a010b7087977fac295351e79a95d7c832/META-INF/maven/com.amazonaws/aws-java-sdk-serverlessapplicationrepository/pom.properties' does not exist.
[error] 	at scala.Predef$.require(Predef.scala:281)
[error] 	at sbtassembly.AssemblyUtils$.copyFile(AssemblyUtils.scala:193)
[error] 	at sbtassembly.AssemblyUtils$.copyImpl(AssemblyUtils.scala:178)
[error] 	at sbtassembly.AssemblyUtils$.$anonfun$copy$1(AssemblyUtils.scala:164)
[error] 	at scala.Function$.$anonfun$tupled$1(Function.scala:79)
[error] 	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[error] 	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error] 	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error] 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error] 	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
[error] 	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[error] 	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error] 	at sbtassembly.AssemblyUtils$.copy(AssemblyUtils.scala:164)
[error] 	at sbtassembly.AssemblyUtils$.copyDirectory(AssemblyUtils.scala:152)
[error] 	at sbtassembly.Assembly$.$anonfun$processDependencyJars$12(Assembly.scala:462)
[error] 	at sbt.State$StateOpsImpl$$anon$1.call(State.scala:389)
[error] 	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:113)
[error] 	at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:91)
[error] 	at xsbt.boot.Locks$GlobalLock.$anonfun$withFileLock$1(Locks.scala:119)

[er1c]

I really didn't want to add a lock here (I originally tried locking on individual jar file), but this seemed to be the only reliable place to put it in a very complex project.

[er1c]

This is a problem, if say you have a /home/packer/.cache/sbt that then gets mounted into a docker container as /home/docker/.cache/sbt and then get something like:

~/.cache/sbt/v1/assembly/dependencies $ cat fdf034b77ea4c96874cbd087f89a92d9c6169839_66a4cd59bc0daa39b67b0b5e496cee078b835d51_c3c65759ec2815d6ff9909c6e810928dba1ad3a0.jarName
/home/docker/.cache/coursier/cache/http/nexus-aws.dot-data.net/repository/maven-public/org/typelevel/spire-macros_2.12/0.17.0-M1/spire-macros_2.12-0.17.0-M1.jar

Trying to decide/figure out if there is an easy way to canonicalize based upon what the current cache directories are, rather fan full absolute path.

[er1c]

I think its safer to keep this in there, rather than do any canonicalization

[eed3si9n]

Why is Contraband dropped here?

[er1c]

It's not used anymore, I can add it back, I had a question earlier about whether it should be used or not. I'll ping you a link

[eed3si9n]

Could we put this back to debug?