Test procedure for candidates helping with manual correction of segmentations

[jcohenadad]

Context

We need people to help manually correcting the segmentations. In order to select the right candidates for the job, we need to put together a small test period so we can assess the quality of the performed task. The purpose of this issue is to describe the workflow for the test period.

Suggestion

Before starting the tests, please watch the video tutorial presented here.

Test 1: Identify problematic segmentations

• Candidates are given a QC from 25 subjects and are asked to report the subjects that require manual correction. There should be a good balance (ie: only the very problematic segmentations should be corrected).
• Candidates will create a YML file named surname_name.yml that lists the problematic subjects as instructed in the documentation. (Only send qc_fail.yml renamed)
• Candidates will email the YML to the lead team including the time it took to perform the selection and generate the YML file.
• Lead team will make a script to compare the ground truth YML with the submitted YML. E.g.: compute number of matched subjects.

Test 2: Manual correction

• Lead team performs manual correction on 5 subjects (different from the subjects in Test 1).
• Lead team packages the 5 subjects + automatic segmentations from the "/data_processed" folder + YML file.
• Lead team will upload the data on a temporary cloud (eg Dropbox). Applicants will be asked to not share the data.
• Candidates are asked to:
  • Follow the instructions for manual correction. Make sure to install the required dependencies (see documentation).
  • Correct the segmentation on the 5 subjects following the documentation.
  • Put the created "derivatives/" folder in a folder named surname_name/ and zip it. Upload it to your Gdrive or Dropbox and email the link to the lead team including the time it took to perform the manual corrections (ie: from the start to the end of the manual_correction.py script).

Questions?

If the instructions are unclear or if you experience an error/bug on your end, please open an issue.
If you have a simple question related to the test period, please post it here.