[bug] Conan 2 omits empty directories when compressing a package

[MichalCermakx]

Describe the bug

Hi Conan team, James,

I have a use-case, where I am using conan for a tool with empty folders that are required for proper tool execution.
I found this old ticket #8013 , discussing the same issue in conan 1. Based on the response, I understand this was a planed update for the conan 2.

But even conan 2 is not handling empty folders. Those are in the package folder but are not propagated into conan_package.tgz

Please check the attached example -
empty212ed808c0279.zip

Package folder:

conan_package.tgz (note missing empty_folder folder):

Thank you,
Michal

How to reproduce it

Add a empty folder to package and check if it is inside of conan_package.tgz

[memsharded]

Hi @MichalCermakx

Thanks for your report.

It is true that #8013 mentioned back then that Conan 2 might do that, but unfortunately, this wasn't the case, and Conan 2 keeps packaging files stored in the conanmanifest.txt, not folders (folders are not represented in manifests).

I am checking what could be done now, as changing this kind of behavior tends to be risky and can break users, but I'd like to ask a quick question just in case: is just that the folder needs to be there, even if empty, but it will not be modified by the execution of the application? Or is it the case that it will be modified by the execution of the application? Because if that is the case, that breaks packages as it breaks the manifest (a conan cache check-integrity will fail for this package), this could be a possible use case of the install() feature, something that has been discussed in the past and it is planned to be tried in 2.X roadmap.

[MichalCermakx]

Thank you for the feedback.
Regarding the modification. It is not changing the conan cache. The content I am talking about is something like build template, which is copied into dependency build folder, where it is utilized to perform a build.

If empty folders are not compressed, I am thinking about making a list of folders (with relative path), storing those in conandata.yml, and recreate those on the client. But this would again lead to a broken package. Or not? I recall something like
empty folders do not count.
Another solution is to add dummy files into the empty folders. The last would be a specific one, which would recreate those folders by a dependency package, during its build.

[memsharded]

Regarding the modification. It is not changing the conan cache. The content I am talking about is something like build template, which is copied into dependency build folder, where it is utilized to perform a build.

Not fully clear what yo u mean with "dependency build folder", because those are not visible by the consumers, and are not really related to the "package folder" which is the one that is compressed.

But this would again lead to a broken package. Or not? I recall something like
empty folders do not count.

No, lets say that your package_info() creates some empty folders in the "package_folder", I think that shouldn't be a problem, because the integrity-checks do also check only files and their checksums, but not empty folders.

Another solution is to add dummy files into the empty folders. The last would be a specific one, which would recreate those folders by a dependency package, during its build.

Yes, this is a simple and effective solution, putting an empty file inside those folders will work, and unless it is strictly necessary that the folder is empty and it doesn't affect the application that dummy file there, that should work well too.

[MichalCermakx]

Thank you, I have all the information I need. As conclusion, those are alternatives with a potential update on conan side:

1. Update conan compression.
2. Use conandata.yml + package_info() to recreate empty folders. Conan developers might consider adding a helper functions to support this.
3. Add a warning into copy method. During copy into package folder, user might be notified empty folders are not compressed.

Alternative without conan update:

1. Add dummy files into empty folders.

[memsharded]

Use conandata.yml + package_info() to recreate empty folders. Conan developers might consider adding a helper functions to support this.

This can be an alternative without any Conan modification, and it doesn't really need a change in Conan. Users can do a mkdir() in their package_info() method to force the creation of an empty folder in the package_folder, unless I have misunderstood something.

[MichalCermakx]

This is true. When I mentioned helper methods, I was thinking about adding this into conan:

1. "capture_empty_folders_in_package()" OR a parameter of a copy method OR conanfile.py attribute. This will lead to capture of a list of empty folders in conandata.yml.
2. recreate those folders in package_info() by calling "recreate_empty_folders()". Or automatic way when the list of empty folders is found in conandata.yml

[memsharded]

I am submitting a PR in #16424 to explore the possibility of letting the Conan internal conan_package.tgz to also store empty folders. It is a draft to discuss with the team, no guarantees it will be possible to merge it, but at least lets discuss.

[MichalCermakx]

Hi @memsharded ,

I really appreciate it. Feel fee to close this ticket, I believe it is not needed anymore.

Michal

[memsharded]

Well, the PR #16424 will close this ticket automatically if merged. If we decide to not merge it because of risk or whatever, then we might manually close this ticket, maybe with some note in the documentation to clarify behavior and suggest alternatives.

But better leave the ticket open and close it when a final path is taken.
Just to make sure, if #16424 is merged, this is something that you will still find useful, isn't it?

[MichalCermakx]

Yes, I was just thinking about you want to keep the number of open tickets low.
I definitely want to get this update, and I hope even others will appreciate it.

[memsharded]

#16424 was merged, it will be in next Conan 2.5