Approach with store on shared folder

[kyle-messier]

Hi, my teams has been using the targetopia for a variety of projects. Currently we use targets, tarchetypes, crew, crew.cluster and geotargets. We utilize an internally maintained data storage and slurm managed cluster. For projects with multiple members contributing to the code base for a large pipeline, we think gittargets could be an important piece. I have a few questions and comments about the recommended practices here.

We are utilizing a shared file path set in the _targets.yaml such as

main: store: /common/path/for/us/all

So, if I understand correctly, a good use case of gittargets would be for the team members to work on their local versions of the pipeline that has the _targets/objects to the common store, and then to use gittargets to work on separate branches, so that the main branch is not changed everytime someone makes a change. Can you confirm if that is a good use case?

If that is the case, then I have a question about setting it up for what is currently an active targets pipeline. Given the recommended workflow, it seems we can skip (1) since the pipeline has run and the data store exists. We also have a git repo going already, so does that mean we can skip 2. tar_git_init() or is it doing something to connect targets to git that is required for proper functionality? If not, perhaps we should run tar_git_init(git_lfs = FALSE) since we don't want to commit the target objects.

Lastly, as a point of clarification, it looks like we are intended to still use the standard git procedures for managing the code base and then just call tar_git_snapshot() occasionally to add a group of commits to the gittargets snapshot, correct?

[wlandau]

So, if I understand correctly, a good use case of gittargets would be for the team members to work on their local versions of the pipeline that has the _targets/objects to the common store, and then to use gittargets to work on separate branches, so that the main branch is not changed everytime someone makes a change. Can you confirm if that is a good use case?

I created gittargets as a way to implement historical versioning for a single user. I am not sure what would happen if multiple users made snapshots to the same data store, but if it did work, I don't think it would fit cleanly. As described at https://docs.ropensci.org/gittargets/articles/git.html#snapshot-model, gittargets already creates data branches to align with code commits, and those data branches are an internal thing. It does not seem feasible to then make these branches user-specific and then think about pull requests/merges.

What I think may help more is some sort of content-addressable storage (CAS) as discussed at ropensci/targets#1232 (comment). As described in the thread, this may be developed at some point by the folks at rOpenSci.

If that is the case, then I have a question about setting it up for what is currently an active targets pipeline. Given the recommended workflow, it seems we can skip (1) since the pipeline has run and the data store exists. We also have a git repo going already, so does that mean we can skip 2. tar_git_init() or is it doing something to connect targets to git that is required for proper functionality? If not, perhaps we should run tar_git_init(git_lfs = FALSE) since we don't want to commit the target objects.

You can skip (1). I am less sure about tar_git_init() since it's been a long time since I worked on gittargets.

Lastly, as a point of clarification, it looks like we are intended to still use the standard git procedures for managing the code base and then just call tar_git_snapshot() occasionally to add a group of commits to the gittargets snapshot, correct?

Yes, that is the recommended workflow.