wip: attempts at caching Maven local repo

[dduportal]

Ref. jenkins-infra/helpdesk#4525

this PR is currently a playground (hence the draft + WiP status) to evaluate if the caching through the S3 PVC is worth it

[MarkEWaite]

I would love it if you would be willing to include the JGit 7.2 and Mina SSHD 2.15.0 changes from:

• Test Mina SSHD 2.15.0 and JGit 7.2.0 #4663

Also no problem if you don't include it, but it would help me know that the changes in those two pull requests are not harmful

[dduportal]

I would love it if you would be willing to include the JGit 7.2 and Mina SSHD 2.15.0 changes from:

* [Test Mina SSHD 2.15.0 and JGit 7.2.0 #4663](https://github.com/jenkinsci/bom/pull/4663)

Also no problem if you don't include it, but it would help me know that the changes in those two pull requests are not harmful

I'm not sure to understand your request @MarkEWaite ?
The goal of the PR here is to build a PoC for loading some part of the cache and validate it has no regression using it.

[dduportal]

Retrying following @alecharp pointer: running the command mvn -pl sample-plugin dependency:go-offline -Dmaven.repo.local="${CUSTOM_M2_REPO}" to populate a cache based on the list of plugins.

Without using ACP, it tooks around 8m17 to generate a ~1.6Gb maven repo, mapping to a 1.2 Gb tar archive in the S3 PVC.

[dduportal]

Retrying following @alecharp pointer: running the command mvn -pl sample-plugin dependency:go-offline -Dmaven.repo.local="${CUSTOM_M2_REPO}" to populate a cache based on the list of plugins.

Without using ACP, it tooks around 8m17 to generate a ~1.6Gb maven repo, mapping to a 1.2 Gb tar archive in the S3 PVC.

With ACP: 2min26

[dduportal]

Retrying following @alecharp pointer: running the command mvn -pl sample-plugin dependency:go-offline -Dmaven.repo.local="${CUSTOM_M2_REPO}" to populate a cache based on the list of plugins.
Without using ACP, it tooks around 8m17 to generate a ~1.6Gb maven repo, mapping to a 1.2 Gb tar archive in the S3 PVC.

With ACP: 2min26

Now it has been bootstrapped: 6s \o/

Let's wait for the result in #4669 (comment) (1.2 instead of 6 Gb file which changes the way S3 behaves - < 5 Gb)

[dduportal]

New retry with b0372d6 (and 7b8c4ae) based on @alecharp pointers:

• Iterate over 3 lines (first PoC have the line specified statically, we could generate the lines from the prep system as future improvement)
• Expand the dependencies scope to runtime, compile and test

=> resulting m2 local repo is 1.9 Gb, with a 1.6 Gb archive

[basil]

jenkins-infra/helpdesk#4442 (comment) FTR

[dduportal]

jenkins-infra/helpdesk#4442 (comment) FTR

Thanks for the reminder. We had this in mind from the beginning also (you explained it to me during FOSDEM which was a discovery as It helped us understand why the "pre warm" step in "prep" was not sufficient enough).

We did load the cache during all the tests with the following patterns:

• Ran a (1 line x 50 PCT runs) sequentially with a Maven "go offline" command for each. It generated a 6 Gb archive which is clearly too much.
• @alecharp helped me to experiment around populating the cache partially with a single or a few command(s) using the "plugin-sample" subproject:
• First try with 1 line and no scope options (wip: attempts at caching Maven local repo #4667 (comment)) generated a 1.2 Gb archive which was missing 2 lines and test dependencies (at least!). But a good first try which showed promising results on the cache usage.
• Second try with 3 lines, and adding more scope options (wip: attempts at caching Maven local repo #4667 (comment)) which generated a 1.6 Gb archive: cache usage is still showing good results was used to evaluate the (positive) impact on ACP.

That being said: this cache population is not good enough (as we still had ACP errors even when using it). I want to explore running a modified PCT (3 lines x all PCTs all in parallel) which would only run a basic test to populate the cache without wasting too much machine time then rsync the M2 repo to a shared FS to aggregate all (and limit repetition). Finally, generate the cache archive from the shared FS.

=> Besides the need for many agents, the risk is to have a too big archive at the end. But this could be circumvented by providing the cache in the node VM template (Adding 10 Gb to an AMI is roughly adding 10s to 20s on the VM boot time). It would avoid using S3 or any centralized system (as it would be distributed on each node).

[dduportal]

jenkins-infra/helpdesk#4442 (comment) FTR

If it proves worth and scalable, we could explore your proposal by building the cache from many other plugins before entering the BOM.

[dduportal]

Closing as the implementation will be somewhere else. Works very well!