[Bug] Enable Dependabot on the repository

[ppkarwasz]

Search before asking

• I had searched in the issues and found no similar issues.

Environment

Other

EventMesh version

Other

What happened

Since EventMesh has a lot of dependencies and the upgrades are done manually, the TAR distribution contains many libraries with known vulnerabilities.

For examples Jackson 2.13.0 is included multiple times in the distribution.

One way to solve these problems is to enable Dependabot (cf. documentation) or another dependency manager on the repository. Dependabot is highly configurable. You can upgrade every dependency or just those that are vulnerable.

How to reproduce

Check the contents of the distribution archive and look for Jackson 2.13.0.

Debug logs

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct *

[Pil0tXia]

We have many PRs submitted by dependabot, but we can't merge it directly, because it will cause dependency&license check fail. Do you have any suggestions?

[ppkarwasz]

We have a merge-dependabot-reusable.yaml Github Actions script that merges Dependabot PR's automatically if they pass the tests. @vy, any ideas how to adapt this to EventMesh?

The main problem, as I see it that EventMesh has a binary distribution with third-party dependencies, while Log4j doesn't. Therefore your merge Dependabot script must be more complex to adhere to the Apache "Assembling LICENSE and NOTICE' policy.

I can try to enhance your check-dependencies.sh so that it does not fail if a dependency version changes. Most of the time (i.e. if the new version does not have new dependencies) it should be enough to merge the Dependabot PR automatically.

IIRC Apache Airflow also has a complex CI deployment to upgrade several Python packages a day. @potiuk, is there some part of your infrastructure that can be adapted to a Java project?

[vy]

If I understand it right, for a particular dependency (e.g., Log4j), these four locations must be aligned:

1. build.gradle
2. tools/third-party-licenses/NOTICE#L626
3. tools/dependency-check/known-dependencies.txt
4. tools/third-party-licenses/licenses/java/LICENSE-log4j-api.txt

[Please correct me if I am missing any above.]

You can follow a strategy as follows:

1. Break down the tools/third-party-licenses/NOTICE file into multiple tools/third-party-licenses/notices/*/NOTICE-*.txt files. This is similar to what you already have for licenses, i.e., tools/third-party-licenses/licenses/*/LICENSE-*.txt.
2. Enrich known-dependencies.txt to contain the link patterns formatted by version to external NOTICE and LICENSE files. For instance, you can use the following link pattern for the Log4j NOTICE file: https://github.com/apache/logging-log4j2/blob/rel/<version>/NOTICE.txt.
3. Implement a dependabot-specific GHA workflow such that if NOTICE and LICENSE files still do match and tests pass, update the version in build.gradle and known-dependencies.txt files.

If you are interested in, I can implement this as a contractor and contribute to EventMesh.

Disclaimer: I am a long time Log4j contributor and Apache Logging Services (log4j, log4cxx, etc.) PMC Chair.

[Pil0tXia]

@vy

these four locations must be aligned

another location shall be aligned: tools/third-party-licenses/LICENSE.

Thank you for proposing the solution. I find it feasible.

@xwm1992 @qqeasonchen , do you think we can further streamline the five files mentioned above?

[ppkarwasz]

For licensing problems there is also an Apache Creadur Whisker project, although I don't know what is it worth.

[Pil0tXia]

@vy

In your experience and knowledge, do you think our LICENSE and dependency management are excessively configured?

For example, we attach a txt file for every third-party dependency (e.g., tools/third-party-licenses/licenses/java/LICENSE-log4j-api.txt). Is it necessary to declare the licenses of these dependencies again in tools/third-party-licenses/LICENSE?

Regarding dependency management, I haven't seen any other projects using the tools/dependency-check/known-dependencies.txt file. The RocketMQ project doesn't perform dependency checks, while Kafka uses a Gradle plugin with minimal configuration. Does this mean that the tools/dependency-check/known-dependencies.txt file is redundant?

[vy]

In your experience and knowledge, do you think our LICENSE and dependency management are excessively configured?

@Pil0tXia, I wouldn't say so. There are no plug-n-play tooling to make the life easier for 1) Gradle/Maven application developers 2) that need to provide a binary distribution 3) complying with the ASF distribution requirements. Airflow, Kafka, etc. they all roll out their own solutions. These efforts are mostly manual, that is, people occasionally check the validity of these files and update them as they see fit.

Note that point (2) – that is, the fact that you distribute binaries – is crucial. For instance, Log4j doesn't distribute binaries containing dependencies. Hence, there, we don't have this problem. We only maintain a NOTICE manually, which is almost fixed for decades.

For example, we attach a txt file for every third-party dependency (e.g., tools/third-party-licenses/licenses/java/LICENSE-log4j-api.txt). Is it necessary to declare the licenses of these dependencies again in tools/third-party-licenses/LICENSE?

No, I don't think so.

Regarding dependency management, I haven't seen any other projects using the tools/dependency-check/known-dependencies.txt file. The RocketMQ project doesn't perform dependency checks, while Kafka uses a Gradle plugin with minimal configuration. Does this mean that the tools/dependency-check/known-dependencies.txt file is redundant?

I don't think so. You have license check as a part of the build. This is great. You also partially implemented a framework to automatically include the LICENSE, etc. files while creating a distribution. But updates to this framework are not automated yet.

In conclusion, I don't think you are over-engineering. I think you are trying to do the right thing. I will consult to members@apache.org on this matter and see if we can simplify the process before getting down to implement an automation pipeline around it.

[ppkarwasz]

For libraries that don't bundle their dependencies there is some automatic tooling: maven-apache-resources. It contains Velocity templates that generate META-INF/LICENSE, META-INF/NOTICE and META-INF/DEPENDENCIES files.

For example the last one in log4j-core looks like this:

From: 'com.github.luben'

  - zstd-jni (https://github.com/luben/zstd-jni) com.github.luben:zstd-jni:jar:1.5.5-11
    License: BSD 2-Clause License  (https://opensource.org/licenses/BSD-2-Clause)


From: 'Conversant Engineering' (http://engineering.conversantmedia.com)

  - com.conversantmedia:disruptor (https://github.com/conversant/disruptor) com.conversantmedia:disruptor:jar:1.2.15
    License: The Apache License, Version 2.0  (http://www.apache.org/licenses/LICENSE-2.0.txt)


From: 'FasterXML' (http://fasterxml.com)

  - Woodstox (https://github.com/FasterXML/woodstox) com.fasterxml.woodstox:woodstox-core:bundle:6.5.1
    License: The Apache License, Version 2.0  (http://www.apache.org/licenses/LICENSE-2.0.txt)
...

and we don't have to do anything to generate it (the maven-apache-parent contains the necessary logic).

I believe that for projects that bundle their dependencies a similar mechanism could be used. Notice files are more complex (but fortunately uncommon), but licenses IMHO could be listed in a file like the example above (and bundled in a single copy). IANAL

[vy]

I have shared this issue in members@apache.org. (Note that it is only accessible by ASF members.)

[vy]

@Pil0tXia, I would like to share some updates from the members@apache.org thread:

1. You are not over engineering, you are doing the right thing.
2. To my surprise, almost all projects manually collect and check the license information.
3. You can indeed massively ease dependency updates: hook up to dependabot PRs, auto-merge the PR iff build succeeds and license of the new version matches with the one of the old. Otherwise, block the PR and add a PMC-review-needed label.

I figured that license information is available in pom.xmls and can easily be accessed/downloaded, no need for manually storing link patterns, etc. as I suggested earlier. No need to store licenses, etc. either. They can be download while creating the distribution.

I am still interested in this gig. Let me know what you think.

[Pil0tXia]

@vy

I agree with your idea. Thank you very much for the insights you've provided.

When EventMesh creates a distribution, we execute the dist and installPlugin tasks from build.gradle. We can create a new task that automatically downloads LICENSE files, and then utilize its outcome in the dist task.

[Pil0tXia]

May we make some progress here? ☺️