Support JVM transitive dependencies with `provided` scope at compile-time

[alonsodomin]

Is your feature request related to a problem? Please describe.
This is somewhat related to #12794 although since this is regarding transitive dependencies, it didn't feel right to add this to that discussion. I'm therefore creating the ticket as a placeholder in case some other users find themselves puzzled by this behaviour. Since this is a somewhat niche use case, this may be something that may be decided to not support in Pants, but maybe worth discussing anyway.

The scenario would one in which a project has a 3rd party dependency where some of its depedencies are defined as provided in their POM. Since the provided scope has the semantics that end user will provide the artifact for runtime, Coursier totally ignores them during resolution and not even mentions them in the JSON report. Many user will not even notice this behaviour as they will indeed provide with an artifact when building their final runnable. However there is one scenarion in which they won't: when the provided dependency is a compile-time only dependency, i.e.: compile-time Java annotation processors or the likes.

This was spotted in our side when using this library: https://search.maven.org/artifact/de.flapdoodle.embed/de.flapdoodle.embed.mongo/3.4.5/bundle.

As you can see in the POM of the mentioned library, they have two provided dependencies on the immutables toolkit, which is mostly a combination of Java annotations and annotation processors that syntetise code at compile-time. The following Coursier command also shows how Coursier ignores those transitive dependencies:

cs fetch de.flapdoodle.embed:de.flapdoodle.embed.mongo:3.4.5 -j resolved.json

To reproduce the bug you will need the following settings:

1. A jvm_artifact target referencing the previously mentioned library
2. A Java file that uses the given library
3. Enable all linting warnings in javac
4. Enable treating warnings as errors in javac

3rdparty/jvm/flapdoodle/BUILD:

jvm_artifact(
    name="embedded",
    group="de.flapdoodle.embed",
    artifact="de.flapdoodle.embed.mongo",
    version="3.4.5",
    packages=["de.flapdoodle.embed.**"],
)

pants.toml:

[javac]
args = [
  "-deprecation",
  "-encoding", "utf-8",
  "-Xlint:all",
  "-Xlint:-processing",
  "-Werror",
]

src/main/java/mypackage/BUILD:

java_sources()

src/main/java/mypackage/EmbeddedMongo.java:

package mypackage;

import de.flapdoodle.embed.mongo.config.*;
import de.flapdoodle.embed.mongo.packageresolver.Command;

class EmbeddedMongo {
    public static void main(String[] args) {
        RuntimeConfig cfg = Defaults.runtimeConfigFor(Command.MongoD).build();
    }
}

When compiling Java code that uses that library, the following warnings regarding missing annotations are shown:

__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/config/store/ImmutableDownloadConfig.class): warning: Cannot find annotation method 'from()' in type 'Generated': class file for org.immutables.value.Generated not found
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/config/store/ImmutableDownloadConfig.class): warning: Cannot find annotation method 'generator()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/config/store/ImmutableDownloadConfig$Builder.class): warning: Cannot find annotation method 'from()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/config/store/ImmutableDownloadConfig$Builder.class): warning: Cannot find annotation method 'generator()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/extract/ImmutableDirectoryAndExecutableNaming.class): warning: Cannot find annotation method 'from()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/extract/ImmutableDirectoryAndExecutableNaming.class): warning: Cannot find annotation method 'generator()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/extract/ImmutableDirectoryAndExecutableNaming$Builder.class): warning: Cannot find annotation method 'from()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/extract/ImmutableDirectoryAndExecutableNaming$Builder.class): warning: Cannot find annotation method 'generator()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/store/ImmutableExtractedArtifactStore.class): warning: Cannot find annotation method 'from()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/store/ImmutableExtractedArtifactStore.class): warning: Cannot find annotation method 'generator()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/store/ImmutableExtractedArtifactStore$Builder.class): warning: Cannot find annotation method 'from()' in type 'Generated'
__cp/96e4570a8be5/de.flapdoodle.embed_de.flapdoodle.embed.process_3.1.10.jar(de/flapdoodle/embed/process/store/ImmutableExtractedArtifactStore$Builder.class): warning: Cannot find annotation method 'generator()' in type 'Generated'

If the repository in which these warnings are appearing has configured javac to consider all warnings as errors (using the -Werror flag), then those compilation warnings prevent building the sources even though there is nothing wrong with the actual source code. Once that error creeps in, there are two possible ways of moving forward:

1. Removing the -Xlint:all and/or -Werror flags from javac.
2. Adding the missing transitive dependencies to the target that is reporting the warnings.

Both options look suboptimal, in the first case, there are many legitimate use cases to have as many linter warnings reported as possible and in a large project can lead to a lot of legitimate warnings being missed. The second option it's a better workaround as it solve the issue locally, but it forces the developers to investigate the spourious warnings.

Describe the solution you'd like
Ideally Pants should be able to resolve the compile-time classpath inclusing this provided transitive dependencies but then ignore them when (unless explicitly mentioned) when packaging targets deploy_jar or jvm_war.

This however requires the following feature from Coursier so it can generate a JSON report that informs about those transitive provided dependencies:
coursier/coursier#1846

Describe alternatives you've considered
So far the only alternatives evaluated are the ones mentioned in the main description body, which feel a bit more like workarounds than real alternatives.

Additional context
N/A

[stuhood]

Thanks for the thorough writeup!

Ideally Pants should be able to resolve the compile-time classpath inclusing this provided transitive dependencies but then ignore them when (unless explicitly mentioned) when packaging targets deploy_jar or jvm_war.

This however requires the following feature from Coursier so it can generate a JSON report that informs about those transitive provided dependencies: coursier/coursier#1846

I sortof agree with @alexarchambault in this case though: these probably should be explicitly added by the user. Part of that is because provided doesn't actually signal why something is provided... it's not necessarily "provided-but-should-be-present-at-compile-time". In some cases these are dependencies which will only be used reflectively at runtime.

But regardless: the case you are describing is definitely related to #12794, in that regardless of how the dep actually ends up declared, it needs to be declared as compile scope (again: in this case, but not in some others). It's also easy to imagine exposing coursier/coursier#1846 as a flag once both it and #12794 exist.