Allow to pre-fetch and then point/reuse all online materials (like hugging face models)

[yarikoptic]

to

• facilitate reuse across runs and datasets by reusing already existing local resource
• embedding to facilitate YODA style organization of materials for archival and reproducibility
• address needs a boost for timeout of hugging face downloads #64

HOWTO

• datalad create -c text2git aind-pipelines to create the super-dataset which would point to everything from online materials necessary, listed below:
• local cache of models (currently from HuggingFace)
  • HF Cache folder? (also could be under datalad)
• local container images (current from ghcr.io)
  • OCI images for all of them via datalad-container (PR version needed) Add skopeo-based adapter for working with OCI images datalad/datalad-container#277
  • oci::docker://nextflow/nextflow to run nextflow which would then run docker ATM
• local git repos (currently from github.com)
  • git repo with submodules . That would
    • facilitate "platform release and versioning", e.g. AIND-Pipelines 1.0.0 which could point to different versions/releases of individual pipelines.
    • singular installation of entire collection and instantiation of only needed submodules
    • introducing local patches for troubleshooting etc.

Sample invocation which worked

[CodyCBakerPhD]

I also take this to include all the git-based capsule-repos (which are still my blocking issue on local WSL runs since all efforts to use a self-contained git invocation have so far failed)

[CodyCBakerPhD]

Note this idea could also go down the road of a 'sandboxed' mode where all internal operations can be run without internet access, which is metered/managed only through input/output steps

[CodyCBakerPhD]

Just noting that while working on this, GitHub itself had a temporary outage that prevented basic operation of the pipeline - further indicating this is a good idea...

[alejoe91]

Just noting that while working on this, GitHub itself had a temporary outage that prevented basic operation of the pipeline - further indicating this is a good idea...

I think what we could do is the following:

• allow users to specify a capsule_versions_user/custom.env, which takes precedence over the capsule_versions.env if present
• in this file, instead of git hashes one could point to local folders
• the new unifying function for git clone could check if the hash (second argument) exists as a folder, and if so use that instead of cloning.

How does that sound?

[CodyCBakerPhD]

For the second 2 points, that is what I imagined - yes

Unsure about the need for an additional .env file