Can't compile java with `--add-exports` option in Maven

[mnd999]

When I try to compile a module with some dubious module usage, I get errors in bloom:

markspc% bloop compile --verbose annotations         
[D] Loading workspace settings from bloop.settings.json
[D] Computing sources and classpath hashes for annotations
[D] Scheduling compilation for annotations...
[D] Ignoring analysis for annotations, directory /home/mark/neo4j/neo4j-4.2/public/annotations/target/classes-empty-annotations is missing
[D] Increasing counter for /home/mark/neo4j/neo4j-4.2/public/annotations/target/classes-empty-annotations to 1
[D] External classes directory /home/mark/neo4j/neo4j-4.2/public/annotations/target/bloop-bsp-clients-classes/classes-bloop-cli
[D] Read-only classes directory /home/mark/neo4j/neo4j-4.2/public/annotations/target/classes-empty-annotations
[D] New rw classes directory /home/mark/neo4j/neo4j-4.2/public/annotations/target/bloop-internal-classes/classes-bloop-cli-6br8Q2p_QPGbdSBfk9e1_A==
[D] No previous setup found, invalidating everything.
[D] Full compilation, no sources in previous analysis.
[D] All sources are invalidated.
[D] Initial set of included nodes: 
[D] Recompiling all sources: number of invalidated sources > 50.0% of all sources
Compiling annotations (12 Java sources)
[D] Attempting to call com.sun.tools.javac.api.JavacTool@75554fa9 directly...
[D] Invoking javac with -g -target 11 -source 11 -encoding UTF-8 -proc:none --add-exports jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED -Xpkginfo:always -d /home/mark/neo4j/neo4j-4.2/public/annotations/target/bloop-internal-classes/classes-bloop-cli-6br8Q2p_QPGbdSBfk9e1_A== -classpath /home/mark/neo4j/neo4j-4.2/public/annotations/target/classes/META-INF:/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes/META-INF:/home/mark/neo4j/neo4j-4.2/public/annotations/target/bloop-internal-classes/classes-bloop-cli-6br8Q2p_QPGbdSBfk9e1_A==:/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes-empty-annotations:/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes:/home/mark/.m2/repository/org/eclipse/collections/eclipse-collections/10.2.0/eclipse-collections-10.2.0.jar:/home/mark/.m2/repository/org/eclipse/collections/eclipse-collections-api/10.2.0/eclipse-collections-api-10.2.0.jar:/home/mark/.m2/repository/org/apache/commons/commons-lang3/3.10/commons-lang3-3.10.jar:/home/mark/.m2/repository/junit/junit/4.13/junit-4.13.jar:/home/mark/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.11/scala-library-2.12.11.jar
[D] Java compilation took 0.095945566 s
[W] Error reading API from class file: org.neo4j.annotations.api.PublicApiDoclet$FilteringDocletEnvironment : java.lang.IllegalAccessError: superclass access check failed: class org.neo4j.annotations.api.PublicApiDoclet$FilteringDocletEnvironment (in unnamed module @0x33cd053) cannot access class jdk.javadoc.internal.tool.DocEnvImpl (in module jdk.javadoc) because module jdk.javadoc does not export jdk.javadoc.internal.tool to unnamed module @0x33cd053
Compiled annotations (115ms)
[E] Unexpected error when compiling annotations: 'superclass access check failed: class org.neo4j.annotations.api.PublicApiDoclet$FilteringDocletEnvironment (in unnamed module @0x33cd053) cannot access class jdk.javadoc.internal.tool.DocEnvImpl (in module jdk.javadoc) because module jdk.javadoc does not export jdk.javadoc.internal.tool to unnamed module @0x33cd053'
[T] java.lang.IllegalAccessError: superclass access check failed: class org.neo4j.annotations.api.PublicApiDoclet$FilteringDocletEnvironment (in unnamed module @0x33cd053) cannot access class jdk.javadoc.internal.tool.DocEnvImpl (in module jdk.javadoc) because module jdk.javadoc does not export jdk.javadoc.internal.tool to unnamed module @0x33cd053
[T]     java.base/java.lang.ClassLoader.defineClass1(Native Method)
[T]     java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
[T]     java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
[T]     java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
[T]     java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
[T]     java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
[T]     java.base/java.security.AccessController.doPrivileged(Native Method)
[T]     java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
[T]     java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
[T]     java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[T]     java.base/java.lang.Class.getDeclaredClasses0(Native Method)
[T]     java.base/java.lang.Class.getDeclaredClasses(Class.java:2196)
[T]     sbt.internal.inc.ClassToAPI$.structure(ClassToAPI.scala:184)
[T]     sbt.internal.inc.ClassToAPI$.x$2$lzycompute$1(ClassToAPI.scala:130)
[T]     sbt.internal.inc.ClassToAPI$.x$2$1(ClassToAPI.scala:130)
[T]     sbt.internal.inc.ClassToAPI$.instance$lzycompute$1(ClassToAPI.scala:130)
[T]     sbt.internal.inc.ClassToAPI$.instance$1(ClassToAPI.scala:130)
[T]     sbt.internal.inc.ClassToAPI$.$anonfun$toDefinitions0$1(ClassToAPI.scala:137)
[T]     xsbti.api.SafeLazyProxy$$anon$1.get(SafeLazyProxy.scala:30)
[T]     xsbti.api.SafeLazy$Impl.get(SafeLazy.java:64)
[T]     sbt.internal.inc.ClassToAPI$.$anonfun$process$2(ClassToAPI.scala:32)
[T]     scala.collection.immutable.List.foreach(List.scala:392)
[T]     scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:38)
[T]     scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:38)
[T]     scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:47)
[T]     sbt.internal.inc.ClassToAPI$.process(ClassToAPI.scala:32)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.readAPI$1(AnalyzingJavaCompiler.scala:169)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$20(AnalyzingJavaCompiler.scala:190)
[T]     sbt.internal.inc.classfile.Analyze$.readInheritanceDependencies$1(Analyze.scala:179)
[T]     sbt.internal.inc.classfile.Analyze$.$anonfun$apply$15(Analyze.scala:185)
[T]     sbt.internal.inc.classfile.Analyze$.$anonfun$apply$15$adapted(Analyze.scala:99)
[T]     scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:912)
[T]     scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
[T]     scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
[T]     scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
[T]     scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
[T]     scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
[T]     scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:911)
[T]     sbt.internal.inc.classfile.Analyze$.apply(Analyze.scala:99)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$19(AnalyzingJavaCompiler.scala:190)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$19$adapted(AnalyzingJavaCompiler.scala:188)
[T]     scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:912)
[T]     scala.collection.immutable.List.foreach(List.scala:392)
[T]     scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:911)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$17(AnalyzingJavaCompiler.scala:188)
[T]     scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.timed(AnalyzingJavaCompiler.scala:234)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.compile(AnalyzingJavaCompiler.scala:188)
[T]     sbt.internal.inc.javac.AnalyzingJavaCompiler.compile(AnalyzingJavaCompiler.scala:73)
[T]     sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$9(BloopHighLevelCompiler.scala:179)
[T]     scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[T]     sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$1(BloopHighLevelCompiler.scala:72)
[T]     bloop.tracing.BraveTracer.traceInternal(BraveTracer.scala:64)
[T]     bloop.tracing.BraveTracer.trace(BraveTracer.scala:38)
[T]     sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.timed$1(BloopHighLevelCompiler.scala:71)
[T]     sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$8(BloopHighLevelCompiler.scala:172)
[T]     scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[T]     monix.eval.internal.TaskRunLoop$.monix$eval$internal$TaskRunLoop$$loop$1(TaskRunLoop.scala:187)
[T]     monix.eval.internal.TaskRunLoop$RestartCallback$1.onSuccess(TaskRunLoop.scala:119)
[T]     monix.eval.Task$.$anonfun$forkedUnit$2(Task.scala:1463)
[T]     java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
[T]     java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
[T]     java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[T]     java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
[T]     java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
[T]     java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

If I compile the same module with Maven it's fine.

The bloop config for this was generated using my branch https://github.com/mnd999/bloop/tree/generate-maven-java-modules as bloop won't support this out of the box.

It could be that I'm just generating the config wrong.

[mnd999]

Module config:

{
    "version": "1.4.0",
    "project": {
        "name": "annotations",
        "directory": "/home/mark/neo4j/neo4j-4.2/public/annotations",
        "workspaceDir": "/home/mark/neo4j/neo4j-4.2",
        "sources": [
            "/home/mark/neo4j/neo4j-4.2/public/annotations/src/main/java"
        ],
        "dependencies": [
            
        ],
        "classpath": [
            "/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes",
            "/home/mark/.m2/repository/org/eclipse/collections/eclipse-collections/10.2.0/eclipse-collections-10.2.0.jar",
            "/home/mark/.m2/repository/org/eclipse/collections/eclipse-collections-api/10.2.0/eclipse-collections-api-10.2.0.jar",
            "/home/mark/.m2/repository/org/apache/commons/commons-lang3/3.10/commons-lang3-3.10.jar",
            "/home/mark/.m2/repository/junit/junit/4.13/junit-4.13.jar"
        ],
        "out": "/home/mark/neo4j/neo4j-4.2/public/annotations/target",
        "classesDir": "/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes",
        "resources": [
            "/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes/META-INF",
            "/home/mark/neo4j/neo4j-4.2/public/annotations/target/classes/META-INF"
        ],
        "scala": {
            "organization": "org.scala-lang",
            "name": "scala-compiler",
            "version": "2.12.10",
            "options": [
                
            ],
            "jars": [
                
            ],
            "setup": {
                "order": "mixed",
                "addLibraryToBootClasspath": true,
                "addCompilerToClasspath": false,
                "addExtraJarsToClasspath": false,
                "manageBootClasspath": true,
                "filterLibraryFromClasspath": true
            }
        },
        "java": {
            "options": [
                "-g",
                "-target",
                "11",
                "-source",
                "11",
                "-encoding",
                "UTF-8",
                "-proc:none",
                "--add-exports",
                "jdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED",
                "-Xpkginfo:always"
            ]
        },
        "test": {
            "frameworks": [
                {
                    "names": [
                        "com.novocode.junit.JUnitFramework"
                    ]
                },
                {
                    "names": [
                        "org.scalatest.tools.Framework",
                        "org.scalatest.tools.ScalaTestFramework"
                    ]
                },
                {
                    "names": [
                        "org.scalacheck.ScalaCheckFramework"
                    ]
                },
                {
                    "names": [
                        "org.specs.runner.SpecsFramework",
                        "org.specs2.runner.Specs2Framework",
                        "org.specs2.runner.SpecsFramework"
                    ]
                },
                {
                    "names": [
                        "utest.runner.Framework"
                    ]
                },
                {
                    "names": [
                        "munit.Framework"
                    ]
                }
            ],
            "options": {
                "excludes": [
                    
                ],
                "arguments": [
                    {
                        "args": [
                            "-v",
                            "-a"
                        ],
                        "framework": {
                            "names": [
                                "com.novocode.junit.JUnitFramework"
                            ]
                        }
                    }
                ]
            }
        },
        "platform": {
            "name": "jvm",
            "config": {
                "home": "/usr/lib/jvm/java-11-openjdk",
                "options": [
                    
                ]
            },
            "mainClass": [
                
            ]
        },
        "tags": [
            "library"
        ]
    }
}%        

[jvican]

Thanks for reporting! It's the first time I see that error.

I recommend you try to move a minimal example of your javac setup in Maven to a build tool like gradle or sbt and check the generated configuration file. Off the top of my head, I'm not sure what can be missing. The compile order you configure your project with might also matter here, try playing with it too.

[mnd999]

I added more stack trace, I don't think it's Maven specific, running javac with the bloop generated CLI manually seems to work but the bloop / zinc compiler seems to trip up when it hits the bit of code that requires the add exports.
At this point, I've only figured out how to build the Maven plugin so I don't think I can take this any further quickly.

[jvican]

bloop / zinc compiler seems to trip up when it hits the bit of code that requires the add exports.

I don't think zinc has ever been tested with this javac option so I'm not surprised it doesn't work. Maybe Maven is doing something special here to make this option work. I don't have the time to have at this but if I was to keep investigating I would look at the maven-scala plugin and look if they are doing something there to make this option, or the scala-java compile integration, work.

[jvican]

@mnd999 I think what's going on here is that this option adds magic imports to Javac and therefore the Scala compiler that can compile Java signatures fails because it doesn't recognize this option. This is an error that should be reported upstream to scala/scala.

The reason why this works in Maven though is because Maven is compiling this module only with the Java compiler, thanks to java->scala being the default compile order in Maven. The way you can force this in bloop is by exporting the java->scala option to the compile order field of the bloop configuration file. Let me know if this works!

[mnd999]

@jvican I'm not convinced. I've added some detection and setting of the compile order to my branch https://github.com/mnd999/bloop/tree/generate-maven-java-modules and it doesn't seem to make much difference. Even in the log above, javac is getting invoked and completing successfully. It seems to be some other process coming in after that to do some analysis that seems to trip up. Maybe it is an upstream thing, but I don't think I understand it enough just yet.

If I bloop compile a module that depends on the one that fails, for example, it compiles annotations, then the dependent module and after that it comes back to do this next step and then blows up.