Enable remote data downloading#142
Open
kaitj wants to merge 8 commits into
Open
Conversation
- Enables fetching from OSF urls; currently want full URLs to be able to handle other storage cases - Update existing codebase to handle fetching from either remote (if not exist) or local cache - Update existing tests (new tests to be added)
- Update fetcher docstring to reflect error raised - Remove unnecessary pytest cli args - Add new tests for osf fetcher TODO: Add integration tests for fetching from remote uri's once graph is populated
- Do this in one go rather than one at a time
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
- Added human nodes from OG neuromaps sources. Note neuromaps used the method of passing the target as the source and source as the target to perform inverse transformations between the spaces (this is hard-coded in their code). Files were extracted from zipped OSF so singular path could be provided - Added NCBR resource node with placeholder paths. - NA: Generate midthickness surfaces in these human spaces as a reference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the ability to download data from OSF remote storage and facilitates the ability to add on additional storage locations in the future. This also changes how the
data_diris used, from pointing to where the data is originally stored to the cache directory. This means that:self.data_dir / file_path.Data still needs to moved to OSF, would hold off on merging this until we have the data uploaded and / or the graphs updated to absolute paths / uris
Partially addresses #70; still need to move the data