KafkaConnect build does not use custom repository for maven dependency

[cyrmue]

Describe the bug
When specifying a custom repository for a maven dependency in the build section of a KafkaConnect resource, the custom repository it is not used for all dependencies. We have no direct internet connection to any repository (we need to "proxy" all download of all dependencies)

To Reproduce
Steps to reproduce the behavior:

1. Create a KafkaConnect resource like the following:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: test
spec:
  replicas: 1
  bootstrapServers: kafka-bootstrap:9092
  build:
    output:
      type: docker
      image: <internalRegistry>
      pushSecret: <pushCredential>
    plugins:
      - name: elastic-sink
        artifacts:
          - type: maven
            repository: https://<ourRepoServer>/remote-repos/
            group: io.confluent
            artifact: kafka-connect-elasticsearch
            version: 14.0.2

2 . See error

Downloading from custom-repo: https://<ourRepoServer>/remote-repos/com/fasterxml/oss-parent/43/oss-parent-43.pom
Downloaded from custom-repo: https://<ourRepoServer>/remote-repos/com/fasterxml/oss-parent/43/oss-parent-43.pom (24 kB at 1.6 MB/s)
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: commons-codec:commons-codec:jar -> duplicate declaration of version 1.15 @ line 121, column 21
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: commons-codec:commons-codec:jar -> version 1.15 vs ${commons.codec.version} @ line 126, column 21
[ERROR] Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.6.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.6.2 @
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project io.confluent:kafka-connect-elasticsearch:14.0.2 (/tmp/elastic-sink/0242b419/pom.xml) has 1 error
[ERROR]     Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.6.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.6.2: Could not transfer artifact kr.motd.maven:os-maven-plugin:pom:1.6.2 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.6.2/os-maven-plugin-1.6.2.pom: Unknown host repo.maven.apache.org: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:

Expected behavior
All downloads should use the repository URL.

Environment (please complete the following information):

• Strimzi version: 0.33.1
• Installation method: Helm chart
• Kubernetes cluster: 1.23
• Infrastructure: On-premise cluster

Additional context
We thought it would be fixed with:
#7259
#7676

But in our case the settings.xml should not be with profile but with mirror. Something like:
KafkaConnectDockerfile.java Line 159

String settingsXml =  "<settings xmlns=\"http://maven.apache.org/SETTINGS/1.0.0\"><mirrors><mirror><id>customRepo</id><mirrorOf>*</mirrorOf><url>" + escapeXml(repo) + "</url></mirror></mirrors></settings>";

Mirror settings in maven doc
=> "You want to replace a particular repository with your own internal repository which you have greater control over".

[scholzj]

The repository is not intended for mirrors but for custom Maven repositories.

You also don't share the full log, so it is quite hard to see what your issue is. It would be also good if you can create a proper reproducer that we can use to reproduce it. Without that, it is quite hard to understand if the issue is really Strimzi or your own setup.

[cyrmue]

Thanks. Sorry.
I Upload the whole build logfile (just replaced local hostnames in it).

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: test
spec:
  replicas: 1
  bootstrapServers: kafka-bootstrap:9092
  build:
    output:
      type: docker
      image: artifactory.dom.local/localdocker/strimzi-connect/my-connect-cluster:latest
      pushSecret: artifactory-push-secret
    plugins:
      - name: elastic-sink
        artifacts:
          - type: maven
            repository: https://artifactory.dom.local/remote-repos/
            group: io.confluent
            artifact: kafka-connect-elasticsearch
            version: 14.0.2

Expected, that all downloads should use the repository address. But as you have written the setting is only for custom Maven repositories. So if we want to achieve this it should be an additional property for the central mirror..
The Problem is the download on line 49 kafkaconnect-connect-build.log
(central repository from an external address)

[scholzj]

The situation for which this feature was designed is a configuration like this:

      - name: elastic-sink
        artifacts:
          - type: maven
            repository: https://packages.confluent.io/maven/
            group: io.confluent
            artifact: kafka-connect-elasticsearch
            version: 14.0.3

Where the connector lives only in some custom Maven repositories (such as Confluent one, in this case -> they do not publish to Maven central) but some of its dependencies still live only in the central Maven repo. And for that, I think the current solution works as expecrted.

Your use-case seems different as you seem to be in a disconnected network and have some local Maven repository mirror? TBH, I have no such environment to try it out. I'm not sure why does it not work in your case. For me, it tries to pull it from both the custom and the remote repo:

Downloading from confluent: http://packages.confluent.io/maven/kr/motd/maven/os-maven-plugin/1.6.2/os-maven-plugin-1.6.2.pom
Downloading from central: https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.6.2/os-maven-plugin-1.6.2.pom
Downloaded from central: https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.6.2/os-maven-plugin-1.6.2.pom (6.3 kB at 144 kB/s)

In your log that does not seem to be the case which confuses me a bit. Could it be that this is simply because your mirror does not have the kr/motd/maven/os-maven-plugin/1.6.2 artifact?

---

If your use-case requires some additional options/settings in Maven, we can consider it and track this as an enhancement. But it is a bit hard as it requires special infrastructure to develop and test.

[cyrmue]

Thanks a lot!
I upload two "mvn dependency:copy-dependencies" with the output. The first one which is how the build currently runs and the second with the mirror in settings.xml for central.

Current situation (which does not work in our env):

curl -f -L --create-dirs --output /tmp/elastic-sink/0242b419/pom.xml https://artifactory.dom.local/artifactory/remote-repos/io/confluent/kafka-connect-elasticsearch/14.0.2/kafka-connect-elasticsearch-14.0.2.pom
echo '<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"><profiles><profile><id>download</id><repositories><repository><id>custom-repo</id><url>https://artifactory.dom.local/artifactory/remote-repos/</url></repository></repositories></profile></profiles><activeProfiles><activeProfile>download</activeProfile></activeProfiles></settings>' > /tmp/0242b419.xml
mvn dependency:copy-dependencies -s /tmp/0242b419.xml -DoutputDirectory=/tmp/artifacts/elastic-sink/0242b419 -f /tmp/elastic-sink/0242b419/pom.xml
curl -f -L --create-dirs --output /tmp/artifacts/elastic-sink/0242b419/kafka-connect-elasticsearch-14.0.2.jar https://artifactory.dom.local/artifactory/remote-repos/io/confluent/kafka-connect-elasticsearch/14.0.2/kafka-connect-elasticsearch-14.0.2.jar

the output of the "mvn dependency:copy-dependencies" with the error.
mvn_error.log

If I do exactly the same but with the mirror entry in the settings.xml:

curl -f -L --create-dirs --output /tmp/elastic-sink/0242b419/pom.xml https://artifactory.dom.local/artifactory/remote-repos/io/confluent/kafka-connect-elasticsearch/14.0.2/kafka-connect-elasticsearch-14.0.2.pom
echo '<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"><profiles><profile><id>download</id><repositories><repository><id>custom-repo</id><url>https://artifactory.dom.local/artifactory/remote-repos/</url></repository></repositories></profile></profiles><activeProfiles><activeProfile>download</activeProfile></activeProfiles><mirrors><mirror><id>artifactory</id><mirrorOf>central</mirrorOf><url>https://artifactory.dom.local/artifactory/remote-repos</url></mirror></mirrors></settings>' > /tmp/0242b419.xml
mvn dependency:copy-dependencies -s /tmp/0242b419.xml -DoutputDirectory=/tmp/artifacts/elastic-sink/0242b419 -f /tmp/elastic-sink/0242b419/pom.xml
curl -f -L --create-dirs --output /tmp/artifacts/elastic-sink/0242b419/kafka-connect-elasticsearch-14.0.2.jar https://artifactory.dom.local/artifactory/remote-repos/io/confluent/kafka-connect-elasticsearch/14.0.2/kafka-connect-elasticsearch-14.0.2.jar

and the output of the "mvn dependency:copy-dependencies" without the error.
mvn_ok.log

I see also, that this is not a bug (sorry for reporting so).

One option for testing in in "open" environment could be (if working with k8s, disallow NW policy for repo.maven.apache.org and using a mirror? (one from https://repo.maven.apache.org/maven2/.meta/repository-metadata.xml) ? Or on a "normal" client (point repo.maven.apache.org to a wrong ip for example in the hosts file?)

[scholzj]

Triaged on 23.2.2023: Sounds like something what might be useful. But the solution is a bit unclear as it is not easy to reproduce. Custom repositories are not the same as mirrored repositories. So it might need a new separate feature. A part of it should be also some easy way to test it - does not need to be fully automated, but we need something to be able to verify the PR and any future releases when needed. Proposal might be needed depending on how complex the solution is.

[j005u]

Running into the same issue here because all our requests from inside the cluster have to go through an Artifactory virtual repository.

The way to do this properly is by utilizing the maven mirror setting. Alternatively the ability to pass arbitary options to maven should also help.

The way to validate this I think is to just set the mirror as something other than the maven default and validate that package downloads actually use the mirror repo URL. Or alternatively if you need a fail case then set it to a URL that doesn't exist - that should break any appropriate tests.

I would also add that for docker repos the same situation can be dealt with through the operator env vars, but it sure would be nice if we could just set an arbitary prefix to replace quay.io instead of having to manually update each env var on it's own.

[scholzj]

I would also add that for docker repos the same situation can be dealt with through the operator env vars, but it sure would be nice if we could just set an arbitary prefix to replace quay.io instead of having to manually update each env var on it's own.

I don't think it is really clear what you mean by this. You can use any repository you want.

[j005u]

I would also add that for docker repos the same situation can be dealt with through the operator env vars, but it sure would be nice if we could just set an arbitary prefix to replace quay.io instead of having to manually update each env var on it's own.

I don't think it is really clear what you mean by this. You can use any repository you want.

Point being it would be extremely nice to be able to replace the base registry globally with a single setting, rather than have to update each image type env var one-by-one, but I digress. The maven deps issue is an actual blocker.

I don't suppose there's a clean way to pass environment variables to the maven builder? I think this might be doable through MAVEN_ARGS. Edit: never mind, I don't think this can be done via MAVEN_ARGS.

[scholzj]

Point being it would be extremely nice to be able to replace the base registry globally with a single setting, rather than have to update each image type env var one-by-one, but I digress.

Yeah, ok. But there is always someone who wants to override it this way, someone who wants to override it that way etc. The problem is that we cannot maintain 10 different ways to configure the images and keep maintaining and testing all of them. If needed, you can quite easily automate this for example by writing some simple installation script. I think you can do this in the Helm Chart as well if needed.

The maven deps issue is an actual blocker.

It is not exactly the same, but you can always build the image yourself using a Dockerfile. So I do not think it is really a blocker.