Copy external directory to the rootfs

[Marcel-Lambacher]

Expected vs actual behavior

I'm not able to include an entire folder into the OSTree from within the treefile.
I always have to add each file separately.

"add-files": [
 ["subfolder/test1.txt", "/usr/share/subfolder/test1.txt"],
 ["subfolder/test2.txt", "/usr/share/subfolder/test2.txt"],
 ["subfolder/test3.txt", "/usr/share/subfolder/test3.txt"]
]

This behaviour is okay if you only have to work with a few files.
But if you want to include several directories, it is really annoying.

Expected:

I would be very happy if there is a feature, where you can include an entire directory (recursively) into the treefile:

"add-files": [
 ["subfolder/", "/usr/share/subfolder/"],
]

Would you like to work on the issue?

From my first understanding, this seems like an easy task. If this is a feature that should be implemented I would like to give a try.

[jlebon]

That seems reasonable to me. Or maybe globs instead/in addition? One note though: if you have a lot of data that could change like this, it might make more sense to package them as an RPM instead to make sure it's versioned with the rest of the host. We do try to make sure add-files works (e.g. checksumming them to detect changes), but overall RPM-OSTree really likes RPMs. :)

[Marcel-Lambacher]

Yep, I also like the idea to support globs.

Maybe to give you an idea, how we're using OSTree. Basically, we've for different customers different OSTree branches which we're deploying continuously to there target systems. So, if we need to update an package or update some configuration files, etc... we're deploying a new OSTree. Probably you're right and we should rather use own RPM packages for this. But using OSTree directly seems to be a bit easier without the overhead of creating and maintaining different RPM packages for each customer.

Back to topic, is this feature something that's worth to create a PR for it?

[jlebon]

Back to topic, is this feature something that's worth to create a PR for it?

Yup, definitely wouldn't be opposed to it! Do note that handling of actually opening the add-files was recently moved to the Rust side of the codebase, so this will likely involve some work both on the Rust and C side.

[Marcel-Lambacher]

All right, I just quickly run over the code.
IMHO the most simple solution is to "flat out" the glob expression within the treefile parsing in fn treefile_parse<P: AsRef<Path>> and add all matching files directly to the add_files HashMap<String, fs::File>.

I didn't covered all relevant source files, but I think that I'm not required to change any c sources. But we'll see later :)

Is this idea good enough for the implementation, our should I better try to implement the glob expressions at the latest point possible when the actual files will be copied?

[cgwalters]

I'm not able to include an entire folder into the OSTree from within the treefile.

I'm not opposed to this but...an entirely different approach would be for us to support merging in libostree branches. Something like:

add-ostree-refs: 
  - examplecom/x86_64/somecontainer
  - examplecom/x86_64/anothercontainer

Like add-files, these would be merged in after all the packages are installed.

A major advantage of this approach is that change detection is trivial - we just look at whether the commit object has changed, whereas for an external directory it's a lot more expensive.

This would be a natural extension for us to support because this project is after all...rpm-ostree, and so far we've mostly supported RPMs as input on the compose side. In --unified-core mode we basically translate things to this mode because we import packages as ostree refs.

See this post on ostree-list about a custom build system that I think does something like this. And the original gnome-continuous did too.

[dustymabe]

I'm not opposed to this but...an entirely different approach would be for us to support merging in libostree branches. Something like:

Does what you describe above have any overlap with server side package layering? #442

[dustymabe]

@Marcel-Lambacher - any chance you are working on implementing this?

[Marcel-Lambacher]

@dustymabe Ah, I totally have forgotten this issue, I'm sorry.
Well, I fixed this problem by providing a simple template engine on top of the tree definition file which is responsible for rolling out given directory paths to absolute file paths.

However, I'm still interested to implemented this feature. Unfortunately, I can't give you a concrete due date.

Cheers
Marcel

[sayanchowdhury]

@Marcel-Lambacher Do mind if I pick up the issue to work or do you want to continue working on the issue?

[cgwalters]

PR for this in #1830

[jlebon]

A major advantage of this approach is that change detection is trivial - we just look at whether the commit object has changed, whereas for an external directory it's a lot more expensive.

Though one could say that it just pushes the change detection to a separate layer. :) To be fair though it's likely that whatever env/pipeline would leverage this has a smarter way to detect changes than we do (e.g. git commit hash, tarball hash, etc...).