Skip to content
This repository was archived by the owner on Aug 30, 2020. It is now read-only.

Conversation

@alexfoias
Copy link
Contributor

jcohenadad and others added 7 commits July 30, 2020 07:10
Add manually corrected dwi SC segmentations
- remove sapienza04 gmseg
- correct gmseg for sapienza01
manual correction of t1w and t2w images
* fix defaced image

* change permissions

* add fslPrisma dataset

* rename sapienza to fslAchieva

* deface images

* fix defacing dresden02_T2w

* remove other defaced images

* rename test

* fix rename
@alexfoias
Copy link
Contributor Author

Output from current master:

Warning missing following subjects from participants.tsv: 
['sub-fslAchieva01',
 'sub-fslAchieva02',
 'sub-fslAchieva03',
 'sub-fslAchieva04',
 'sub-fslAchieva05',
 'sub-fslAchieva06',
 'sub-fslPrisma01',
 'sub-fslPrisma02',
 'sub-fslPrisma03',
 'sub-fslPrisma04',
 'sub-fslPrisma05',
 'sub-fslPrisma06']

Warning missing data for subjects listed in participants.tsv: 
['sub-sapienza01',
 'sub-sapienza02',
 'sub-sapienza03',
 'sub-sapienza04',
 'sub-sapienza05',
 'sub-sapienza06']
Missing jsonSidecar: ./derivatives/labels/sub-mgh01/anat/sub-mgh01_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-cmrra01/anat/sub-cmrra01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vuiisIngenia05/anat/sub-vuiisIngenia05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-milan01/anat/sub-milan01_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-milan01/anat/sub-milan01_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vuiisIngenia03/anat/sub-vuiisIngenia03_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-sherbrooke03/anat/sub-sherbrooke03_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w02/anat/sub-tokyo750w02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w02/anat/sub-tokyo750w02_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w05/anat/sub-tokyo750w05_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w05/anat/sub-tokyo750w05_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tehranS03/anat/sub-tehranS03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford05/anat/sub-stanford05_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford02/anat/sub-stanford02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mpicbs02/anat/sub-mpicbs02_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tehranS02/anat/sub-tehranS02_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-strasbourg01/anat/sub-strasbourg01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w03/anat/sub-tokyo750w03_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mniS05/anat/sub-mniS05_T2w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mniS05/anat/sub-mniS05_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford03/anat/sub-stanford03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford04/anat/sub-stanford04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford04/anat/sub-stanford04_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-barcelona05/anat/sub-barcelona05_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno08/anat/sub-brno08_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno08/anat/sub-brno08_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vallHebron07/anat/sub-vallHebron07_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno06/anat/sub-brno06_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-beijingVerio02/anat/sub-beijingVerio02_T2w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-dresden02/anat/sub-dresden02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno01/anat/sub-brno01_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mniPilot1/anat/sub-mniPilot1_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-perform06/anat/sub-perform06_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-perform06/anat/sub-perform06_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-balgrist02/anat/sub-balgrist02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-beijingVerio03/anat/sub-beijingVerio03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno07/anat/sub-brno07_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno07/anat/sub-brno07_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib09/anat/sub-oxfordFmrib09_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil03/anat/sub-nottwil03_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil03/anat/sub-nottwil03_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil04/anat/sub-nottwil04_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil04/anat/sub-nottwil04_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib07/anat/sub-oxfordFmrib07_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-queensland03/anat/sub-queensland03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva05/anat/sub-fslAchieva05_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-geneva04/anat/sub-geneva04_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordOhba05/anat/sub-oxfordOhba05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordOhba05/anat/sub-oxfordOhba05_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia01/anat/sub-tokyoIngenia01_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib01/anat/sub-oxfordFmrib01_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia06/anat/sub-tokyoIngenia06_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib06/anat/sub-oxfordFmrib06_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil05/anat/sub-nottwil05_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib08/anat/sub-oxfordFmrib08_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nwu06/anat/sub-nwu06_T2w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nwu01/anat/sub-nwu01_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordOhba03/anat/sub-oxfordOhba03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva03/anat/sub-fslAchieva03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva04/anat/sub-fslAchieva04_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva04/anat/sub-fslAchieva04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-cmrra05/anat/sub-cmrra05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia05/anat/sub-pavia05_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-sherbrooke01/anat/sub-sherbrooke01_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia02/anat/sub-pavia02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mgh05/anat/sub-mgh05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mgh02/anat/sub-mgh02_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia03/anat/sub-pavia03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia03/anat/sub-pavia03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia04/anat/sub-pavia04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-pavia04/anat/sub-pavia04_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib10/anat/sub-oxfordFmrib10_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-beijingPrisma01/anat/sub-beijingPrisma01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford06/anat/sub-stanford06_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford06/anat/sub-stanford06_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford06/anat/sub-stanford06_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-stanford06/anat/sub-stanford06_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mpicbs01/anat/sub-mpicbs01_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w06/anat/sub-tokyo750w06_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w06/anat/sub-tokyo750w06_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyo750w01/anat/sub-tokyo750w01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mpicbs07/anat/sub-mpicbs07_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-cardiff05/anat/sub-cardiff05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mniS01/anat/sub-mniS01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-mniS08/anat/sub-mniS08_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tehranS01/anat/sub-tehranS01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vuiisAchieva01/anat/sub-vuiisAchieva01_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vuiisAchieva01/anat/sub-vuiisAchieva01_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tehranS06/anat/sub-tehranS06_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-beijingGE04/anat/sub-beijingGE04_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-vallHebron03/anat/sub-vallHebron03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-perform04/anat/sub-perform04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno02/anat/sub-brno02_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno02/anat/sub-brno02_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno05/anat/sub-brno05_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno05/anat/sub-brno05_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoSkyra04/anat/sub-tokyoSkyra04_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-amu02/anat/sub-amu02_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-amu05/anat/sub-amu05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-balgrist01/anat/sub-balgrist01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-perform02/anat/sub-perform02_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-perform02/anat/sub-perform02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno04/anat/sub-brno04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno03/anat/sub-brno03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brno03/anat/sub-brno03_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-amu04/anat/sub-amu04_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-cmrrb03/anat/sub-cmrrb03_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-cmrrb03/anat/sub-cmrrb03_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva01/anat/sub-fslAchieva01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva01/anat/sub-fslAchieva01_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva01/anat/sub-fslAchieva01_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-fslAchieva06/anat/sub-fslAchieva06_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-juntendo750w02/anat/sub-juntendo750w02_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia03/anat/sub-tokyoIngenia03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia03/anat/sub-tokyoIngenia03_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia03/anat/sub-tokyoIngenia03_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib03/anat/sub-oxfordFmrib03_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia04/anat/sub-tokyoIngenia04_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-brnoPrisma04/anat/sub-brnoPrisma04_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nwu02/anat/sub-nwu02_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-geneva01/anat/sub-geneva01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia05/anat/sub-tokyoIngenia05_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-oxfordFmrib05/anat/sub-oxfordFmrib05_T1w_RPI_r_labels-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-tokyoIngenia02/anat/sub-tokyoIngenia02_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-queensland01/anat/sub-queensland01_acq-T1w_MTS_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil01/anat/sub-nottwil01_T2star_rms_gmseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil01/anat/sub-nottwil01_T2w_csfseg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-nottwil01/anat/sub-nottwil01_T1w_RPI_r_seg-manual.json
Missing jsonSidecar: ./derivatives/labels/sub-juntendo750w03/anat/sub-juntendo750w03_T1w_RPI_r_seg-manual.json

@alexfoias
Copy link
Contributor Author

@jcohenadad This PR is ready for review.

Copy link
Member

@jcohenadad jcohenadad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please do not merge-- we need to wait for @kousu to tell us what to do

@alexfoias
Copy link
Contributor Author

Yes. I won't, I'll wait for your approval.

@kousu
Copy link
Contributor

kousu commented Aug 13, 2020

Sorry! You're going to have to remake this over in the remade https://github.com/spine-generic/data-multi-subject. And you'll have to add a git annex get line to the CI script, now (or will you? does the bids validator actually check that the niftis are valid images?).

I also would caution about running that on every CI run because that means downloading 10G to travis every time. That is going to make CI slow. Travis supports caching specific dirs: https://docs.travis-ci.com/user/caching/ but they have a warning that caching doesn't do much unless you're caching processing steps -- because each VM needs to download the data fresh each time; still it might be worth experimenting at least for an hour or two with. What if we only ran the validator in the nightly cronjob?

@kousu kousu mentioned this pull request Aug 13, 2020
@kousu
Copy link
Contributor

kousu commented Aug 13, 2020

(or will you? does the bids validator actually check that the niftis are valid images?).

Answered my own question: https://travis-ci.com/github/spine-generic/data-multi-subject-2/builds/179667411. Hard yes:

1: [ERR] This file is too small to contain the minimal NIfTI header. (code: 36 - NIFTI_TOO_SMALL)

  ./sub-amu01/anat/sub-amu01_T1w.nii.gz
  ./sub-amu03/anat/sub-amu03_T2star.nii.gz
  ... and 1030 more files having this issue (Use --verbose to see them all).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants