Http 429 - "Too many requests" during maven build

[ChrisCreatesCoolCode]

Hello there,

I am using your plugin in my build process. Lately I discovered that some licenses fail to download with a "429 - Too Many Requests" as shown in the image below.
The rejection ultimately is decided by the server, but maybe there is a problem with the rate of requests to these servers? If so, maybe this rejection could be avoided by reducing the rate of download requests?

Best regards
Christian

[dade-hms]

Same here, running a simple mvn org.codehaus.mojo:license-maven-plugin:2.4.0:aggregate-download-licenses

[r-sop]

Same problem. Accessing the url manually works.

It also shows a lot of duplicated urls, so I think caching the response should fix this.

[slawekjaranowski]

There is a class org.codehaus.mojo.license.download.Cache maybe should be improved.

Do you have an issue in single or multiple modules Maven project?

[r-sop]

@slawekjaranowski For my case it's a multi-module Maven project.

Some additional information: We also use the licenseUrlReplacements, here our (cleaned up) configuration:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>license-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>aggregate-add-third-party</id>
            <goals>
                <goal>aggregate-add-third-party</goal>
            </goals>
        </execution>
        <execution>
            <id>aggregate-download-licenses</id>
            <configuration>
                <licenseUrlReplacements>
                    <licenseUrlReplacement>
                        <regexp>\Qhttps://glassfish.java.net/public/CDDL+GPL_1_1.html\E</regexp>
                        <replacement>https://oss.oracle.com/licenses/CDDL+GPL-1.1</replacement>
                    </licenseUrlReplacement>
                    <licenseUrlReplacement>
                        <regexp>\Qhttps://glassfish.dev.java.net/nonav/public/CDDL+GPL.html\E</regexp>
                        <replacement>https://oss.oracle.com/licenses/CDDL+GPL-1.1</replacement>
                    </licenseUrlReplacement>
                    <licenseUrlReplacement>
                        <regexp>\Qhttps://openjdk.java.net/legal/gplv2+ce.html\E</regexp>
                        <replacement>https://github.com/openjdk/jfx/blob/master/LICENSE</replacement>
                    </licenseUrlReplacement>
                    <licenseUrlReplacement>
                        <regexp>\Qhttps://glassfish.dev.java.net/public/CDDLv1.0.html\E</regexp>
                        <replacement>https://javaee.github.io/LICENSE</replacement>
                    </licenseUrlReplacement>
                    <licenseUrlReplacement>
                        <regexp>\Qhttp://opensource.org/licenses/mit-license\E</regexp>
                        <replacement>https://opensource.org/license/mit</replacement>
                    </licenseUrlReplacement>
                </licenseUrlReplacements>
            </configuration>
            <goals>
                <goal>aggregate-download-licenses</goal>
            </goals>
        </execution>
    </executions>
</plugin>

[ChrisCreatesCoolCode]

@slawekjaranowski In my case it is also a multi-module-project

[vitalijr2]

I have tried to run the command

mvn org.codehaus.mojo:license-maven-plugin:2.4.0:aggregate-download-licenses

here.

The result is same:

$ mvn org.codehaus.mojo:license-maven-plugin:2.4.0:aggregate-download-licenses
[INFO] Scanning for projects...
[INFO] 
[INFO] ---------------< org.codehaus.mojo:license-maven-plugin >---------------
[INFO] Building License Maven Plugin 2.5.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] ----------------------------[ maven-plugin ]----------------------------
[INFO] 
[INFO] --- license:2.4.0:aggregate-download-licenses (default-cli) @ license-maven-plugin ---
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
[WARNING] No URL for license at index 0 in dependency aopalliance:aopalliance:1.0
[WARNING] No license information available for: org.codehaus.plexus:plexus-i18n:1.0-beta-10
[WARNING] No URL for license at index 0 in dependency org.tukaani:xz:1.9
[WARNING] Unable to retrieve license from URL 'https://glassfish.dev.java.net/public/CDDLv1.0.html' for dependency 'javax.annotation:jsr250-api:1.0': Connect to glassfish.dev.java.net:443 [glassfish.dev.java.net/137.254.56.48, glassfish.dev.java.net/64:ff9b:0:0:0:0:89fe:3830] failed: Network is unreachable (connect failed)
[WARNING] 'http://wiki.github.com/shellac/java-rdfa/licence' returned 400 Bad Request
[WARNING] 'http://www.gnu.org/licenses/lgpl-3.0.txt' returned 429 Too Many Requests
[WARNING] There were 6 download errors - check the warnings above
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.098 s
[INFO] Finished at: 2024-11-17T09:51:49+02:00
[INFO] ------------------------------------------------------------------------

[vitalijr2]

There is a class org.codehaus.mojo.license.download.Cache maybe should be improved.

Do you have an issue in single or multiple modules Maven project?

I run it in debug, looks like the cache works as expected. But the mojo does strange steps:

• first run something like dry-run and does not put anything to cache but in the end writes "Finished populating cache"
  

  license-maven-plugin/src/main/java/org/codehaus/mojo/license/AbstractDownloadLicensesMojo.java

  Line 868 in 922e838

  downloadLicenses(licenseDownloader, depProject, true);

• next run the same but with downloading licenses and putting them to cache, each license is downloading once
  

  license-maven-plugin/src/main/java/org/codehaus/mojo/license/AbstractDownloadLicensesMojo.java

  Line 876 in 922e838

  downloadLicenses(licenseDownloader, depProject, false);

  On second run some sites returns 429 error even on first request, maybe because I run this goal some times in a row for test purposes.
  As an example when gnu.org returns 429 two one time on first request:
  https://gist.github.com/vitalijr2/6cc3b2ad2e21e5062f0a316f4bb265af

[dade-hms]

I have tried here with one of our single module project.

Every query to gnu.org gets the error. From the first one, like somehow the tool was detected by their server.

It's not an IP blocking, I can query the same url at the same time using wget or my browser.

mvn -X org.codehaus.mojo:license-maven-plugin:2.4.0:aggregate-download-licenses | grep gnu.org
[DEBUG] About to download 'https://www.gnu.org/licenses/lgpl-3.0.html'
[WARNING] 'https://www.gnu.org/licenses/lgpl-3.0.html' returned 429 Too Many Requests
[DEBUG] About to download 'http://www.gnu.org/licenses/lgpl.html'
[WARNING] 'http://www.gnu.org/licenses/lgpl.html' returned 429 Too Many Requests
[DEBUG] About to download 'https://www.gnu.org/licenses/gpl-2.0.txt'
[WARNING] 'https://www.gnu.org/licenses/gpl-2.0.txt' returned 429 Too Many Requests
[DEBUG] About to download 'https://www.gnu.org/software/classpath/license.html'
[WARNING] 'https://www.gnu.org/software/classpath/license.html' returned 429 Too Many Requests
[WARNING] 'https://www.gnu.org/software/classpath/license.html' returned 429 Too Many Requests
[DEBUG] About to download 'https://www.gnu.org/licenses/old-licenses/lgpl-2.1'
[WARNING] 'https://www.gnu.org/licenses/old-licenses/lgpl-2.1' returned 429 Too Many Requests

Also, maven infos:

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Maven home: *******
Java version: 17.0.5, vendor: Eclipse Adoptium, runtime: /home/dade/.sdkman/candidates/java/17.0.5-tem
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "6.8.0-49-generic", arch: "amd64", family: "unix"