List public databases in our internal git-annex server

[sandrinebedard]

Some of our datasets (eg: spine-generic, whole-spine) are hosted on public cloud servers (eg: AWS, OpenNeuro). To facilitate their reusability it would be convenient to have them listed in our internal git-annex database. Is this something possible?

Moreover, this approach would avoid having duplicated dataset. E.g., currently, whole-spine is physically hosted on both our internal git-annex server, and on OpenNeuro.

[sandrinebedard]

@mguaypaq I am wondering this sine I want to add new GT for spinal cord segmentation, and it would be more simple to just push on one of them!

[mguaypaq]

I think we can at least de-duplicate the image storage, using git-annex special remotes, like we currently do for spine-generic. For spine-generic:

• The small files and the folder structure and pull requests are hosted on github.
• The contents of the large files are hosted on AWS.

In this case, we would have:

• The small files and folder structure and pull requests are hosted on data.neuro.polymtl.ca.
• The contents of the large public files are hosted in the cloud, and are available from both the public repo, and our internal repo.
• If we have large private files (for example, derivatives) to add, those ones can still be stored on data.neuro.polymtl.ca. Git-annex is able to track multiple sources for different files.

This would also be useful in the case where we want to take a public dataset, but change the file and folder names.

(If you want to purely have a link, so that we don't even have to duplicate the small files and folder structure, but still have the repository show up in searches, I don't think that's possible.)

[jcohenadad]

If you want to purely have a link, so that we don't even have to duplicate the small files and folder structure, but still have the repository show up in searches, I don't think that's possible

Ah, interesting. This is what we intended. Isn't it possible because of the nature of git-annex to only use pointer for big files? What if we specify in .gitattributes that all files should be annexed?

[mguaypaq]

If you want to purely have a link, so that we don't even have to duplicate the small files and folder structure, but still have the repository show up in searches, I don't think that's possible

Ah, interesting. This is what we intended. Isn't it possible because of the nature of git-annex to only use pointer for big files? What if we specify in .gitattributes that all files should be annexed?

This is more a limitation of (Neuro)gitea than git-annex.

We could specify in .gitattributes that git-annex should handle the contents of all files, not just big files, that's true. But that's just the file contents, not the file and folder names (and not the history, branches, commit messages, etc.). Regardless of git-annex, there still needs to be a git repository.

I think what you want is called a repository mirror, and it looks like Gitea does support it, which I didn't know about. Essentially, Gitea clones the existing repository (which must be publicly accessible), and it periodically pulls changes from the public source. I'm not sure yet how this would interact with git-annex, but I think it would only pull the un-annexed stuff (file names, folder names, commit messages, etc.) and the content availability information (that is, info about which servers have the large file contents).

[jcohenadad]

This is more a limitation of (Neuro)gitea than git-annex.

ok, thank you for the clarification. And what is our need to keep gitea instead of just using git-annex? Is it for the visual display of the images on a web browser, and the possibility to open issues/PRs? If so, does the lab use these features? (I personally don't, except for the Praxis databset, but that's separate from data.neuro.polymtl.ca)

[mguaypaq]

Gitea handles a lot for us:

• It's the entire web interface (so, listing datasets, browsing the contents and the README).
• It does pull requests, which students/postdocs and me have been using a lot.
• It handles user logins and authentication.
• It handles read/write/admin permissions for the repositories.
• It provides the endpoint that git clone/push/pull etc. talks to over ssh.
• Eventually, it should handle automated checks and validation for pull requests.

In particular, it's the reason we have a listing of datasets at all. If we didn't want gitea, we might as well list external datasets as just an HTML link on the lab manual.

[hermancollin]

I am preparing to send one of our datasets to https://dandiarchive.org/. It's currently sitting in the internal data server. We should de-duplicate by deleting the data from the git-annex server. Has this been done before?

I don't know what the procedure should be in this case.

[mguaypaq]

I am preparing to send one of our datasets to https://dandiarchive.org/. It's currently sitting in the internal data server. We should de-duplicate by deleting the data from the git-annex server. Has this been done before?

That depends on which parts you want to de-duplicate, and how much you trust dandiarchive to still be working in a few years.

• The most radical option would be to delete your dataset completely from data.neuro.polymtl.ca, and then to replace it with a mirror of the version on https://github.com/dandisets. This would probably lose some of the git history of the dataset during development, and it would certainly lose the information on pull requests, etc.
• Since dandiarchive supports datalad, another option would be to keep the git repository on data.neuro.polymtl.ca, and simply add dandiarchive as a special remote in git-annex. Then, we could de-duplicate only the files which are on dandiarchive, while keeping everything else.

[sandrinebedard]

Okay so, from now on, we should delete the whole-spine in our internal dataset and only update the version on open neuro. @mathieuboudreau gave me the writting acess for now!

[mguaypaq]

Ok, so I've archived our internal, non-mirror dataset whole-spine, and we can delete it completely in ~1 month (in case we change our minds). The whole-spine-mirror is still there.

[mguaypaq]

Update: I have un-archived the whole-spine repo, and this is where we should track future changes at the lab. The whole-spine-mirror repo still exists so that we can track the OpenNeuro releases, but we shouldn't make any extra changes there.

In the future, we should be able to:

1. Make more pull requests in whole-spine.
2. Eventually decide to make another OpenNeuro release, at which point we'll (notionally) make a tarball of the master branch and upload it to OpenNeuro.
3. This will cause OpenNeuro to add some commits and a tag in their branch.
4. We can git fetch these extra commits into whole-spine, and git merge them into our master branch, to make sure that the two histories don't diverge again in the future.

Here are the steps I took to integrate the git histories of both repositories in whole-spine's master branch:

1. I cloned whole-spine so that origin was our internal git-annex repo, and added OpenNeuro's ds005616 repo as a second remote called openneuro. That way, I was able to git fetch both histories.
2. I figured out that the 0cf984c3cd55ee959bf7f88b121fbd07066260fc commit from NeuroPoly was the most likely point used to make the first OpenNeuro release e2d2144e3344e3d1dfcb12ff697a2252765575f1. There were a lot of changes between the two.
3. On the NeuroPoly side, I added commits from 0cf984c3cd55ee959bf7f88b121fbd07066260fc to 24808119094495cbbb6bde3b24b617ba2ee51b72 to re-create the changes made to the dataset as it was being published:
   • removing an outdated bids-validator config file
   • removing private images that we can't publish (sub-amuED, sub-amuJD, sub-amuJW)
   • adding the changelog, readme, dataset_description from OpenNeuro
   • adding labels and fieldmaps from OpenNeuro
4. On the OpenNeuro side, I added commits from e2d2144e3344e3d1dfcb12ff697a2252765575f1 to 0c07f09f4f4ac9b4b18973255c0e89babf44b7f2 to harmonize their git-annex config with ours:
   • removing the datalad config (we can't push to it)
   • un-annexing some small text files (there's truly no point)
   • running git annex unlock (to have pointer files instead of symlinks, which is our lab-wide default)
5. At this point the two branches had the same files, so I created a merge commit da575f022527f1aec32fd953c97ff84fd44b8ea6 to join the two unrelated git histories.
6. After that, I was able to git merge each OpenNeuro tag into the NeuroPoly master branch without much trouble.
   • Each modified .nii.gz file created a merge conflict, but I was able to git add and git annex unlock the correct one.
7. Finally, I was able to git merge the changes made to whole-spine at NeuroPoly since the splitting-off commit 0cf984c3cd55ee959bf7f88b121fbd07066260fc.

[jcohenadad]

Thank you for these investigations @mguaypaq. I'm wondering how we could flag that a git-annex repos is also on openneuro (or any other cloud host such as compute canada), such that when we make a local change, we don't forget to also update the cloud repos. Maybe a new label 'cloud' (or 'openneuro') on gitea?

[mguaypaq]

In the case of openneuro, we probably don't want to update it every time we make an internal change; we probably want to wait until we have a coherent set of changes and then update openneuro when we make a release.

For this project specifically, we still have the two repos whole-spine and whole-spine-mirror on our data server, so they will both show up together when searching. Also, I just added a reference between them in the repo description (in both directions):

[jcohenadad]

For this project specifically, we still have the two repos whole-spine and whole-spine-mirror on our data server, so they will both show up together when searching. Also, I just added a reference between them in the repo description (in both directions):

ok cool. But people need to know which one they should update. Is it whole-spine?