Multiple copies of IDE distribution in Gradle cache + reindexing problem

[chashnikov]

What happened?

I migrated a project to IntelliJ Platform Gradle Plugin 2.0 and specified intellijIdeaUltimate("2024.1") in build.gradle.kts and didn't change it. However, after a few hours of work, I found that the Gradle cache contains six copies of IDEA Ultimate distribution:

nik@nik-workstation:~/.gradle/caches/transforms-3$ find -name ideaIU-2024.1
./e0d45b10a0ea56c67b8eef1a3248a586/transformed/ideaIU-2024.1
./d89f9fc4990ceec8ab9fc252d20afe96/transformed/ideaIU-2024.1
./80050e3b4d7632fe5e49c5042913887a/transformed/ideaIU-2024.1
./7167bc64f2f585d0ea26534a74c92e9a/transformed/ideaIU-2024.1
./b91bb87b6350df68a77c91dc42a6215e/transformed/ideaIU-2024.1
./172cec470fbf4c2fa3c0a0f57e6b823c/transformed/ideaIU-2024.1

Since each distribution occupies 3.3Gb, it's quite a lot of space.

Steps to reproduce

Not when exactly new copies are created.

Gradle IntelliJ Plugin version

2.0.0-SNAPSHOT

Gradle version

8.5

Operating System

Linux

[hsz]

I'm aware of this problem and investigating the root cause. Expected behavior is that Gradle reuses already transformed artifact even if build configuration changes.

[JojOatXGME]

I encountered the same problem. I ended up with 197 GiB worth of JetBrains IDEs in ~/.gradle/caches/transforms-4 after only a some days of experimenting with 2.0.

PS: #1639 look like a duplicate of this issue.

I am using convention plugins, maybe that is part of the problem? I could imagine that Gradle uses the classpath of the plugin which defines the transformer as part of hash. After all, the transformer might have been changed between two versions of the plugin. With convention plugins, the transformer becomes effectively part of the convention plugin, which would mean Gradle would have to re-run the transformer every time I change my build logic in the convention plugin. But that is just a guess.

[Undin]

@hsz any updates on this (I've read #1639 (comment))?

[hsz]

The Kotlin Script I've created for purging IntelliJ Platform extracted archives from the Gradle Transformers Cache:

https://gist.github.com/hsz/0fc45e1a6fc9ef73d4e4f5960058bded

[hsz]

Hello, Jonathan! I am very well aware of this problem. Let me explain to you the state of the current implementation:

Adding the IntelliJ Platform dependency to the project resolves it from the IJ Maven Repository or CDN (download.jetbrains.com).
Both sources provide the IntelliJ Platform archive — Maven gives ZIP and CDN, depending on your OS: DMG, ZIP, or TAR.GZ.
Gradle fetches such an archive into the cache directory, like ~/.gradle/caches/modules-2/files-2.1/....

The Gradle IntelliJ Plugin 1.x is extracting content next to the archive, polluting the cache, but it works.
With IntelliJ Platform Gradle Plugin 2.0, I decided to do it correctly and involve the artifact transformers mechanism provided by Gradle. The extracted content goes into a dedicated ~/.gradle/caches/transforms-4/[HASH] location, and the dependency can be correctly formed using native Gradle features.

After the implementation, it turned out, that Gradle calculates [HASH] using the project build classpath. This means that this hash changes whenever you update the IntelliJ Platform Gradle Plugin plugin or any other Gradle plugin in your project (or have a buildSrc local setup that changes).
As a side effect, the IntelliJ Platform archive gets extracted again, the cache grows, and the IDE reindexes the IntelliJ Platform dependency again.

I already had a chat with Gradle folks about this issue, but there's no solution possible to keep relying on the artifact transformers feature.
I'm currently trying to figure out another solution, and most likely, we'll introduce a custom cache location, such as ~/.intellijPlatform/ides/, to keep just a single copy of the extracted IDE.
Keep your fingers crossed!

#1639 (comment)

[hsz]

Unfortunately, it is impossible to extract the resolved IntelliJ Platform artifact (no matter if this is installer or a ZIP archive from the IntelliJ Maven repository) to a custom directory and reuse it later with Gradle dependencies mechanism without bReaking some of Gradle foundations.

I consider implementing the fix in the Gradle build system as the only possible solution.

[Vanco]

Maybe use the HASH of file (zip, dmg, tag.gz) instead of Gradle calculates one. @hsz

[hsz]

I have no impact on what is taken as an input:

https://github.com/gradle/gradle/blob/7457e89eed50aa0b9dbab3a521141f6b8ce4a073/platforms/software/dependency-management/src/main/java/org/gradle/api/internal/artifacts/transform/DefaultTransform.java#L683

Gradle considers the whole implementation classpath of the build script. The idea is to limit the input classpath and isolate it by specifying dependencies that affect the transformer output, such as:

dependencies {
    registerTransform(MyTransform::class.java) {
        from...
        to...
        isolated {
             classpath("transform:dependency-1:1.0")
             classpath("transform:dependency-2:1.0")
        }
    }
}

Having that implemented, I could pass the IntelliJ Platform dependency input, and Gradle will calculate the hash of the archive you mentioned.
But that requires changes in the build system.