Refactor the BIA retrieval tool

[kostrykin]

There is a tool for downloading images from the Bioimage Archive:
https://imaging.usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/bgruening/bia_download/bia_download/0.1.0+galaxy0

---

The UI of this tool needs some love:

• The "storage mode" can only be nfs or fire, this should thus be a dropdown field. This has been removed in Biaftp update bgruening/galaxytools#1582.
• The help text of the field should say when to use which of the two options. Not applicable anymore.
• The "path of accession" field is unclear — what does a user need to put in here? Add a help text. Has been cleared up in Biaftp update bgruening/galaxytools#1582.
• Why are both fields is the accession field optional? Is that correct? Should be required.
• Is it possible to retrieve only part of a dataset? Not with the current API.

Optional:

• If it is possible to retrieve only part of a dataset, this should be implemented. See above.
• Migrate to the alpha version of the API? Stable API could be available by now (Beatriz should have further info). It is not.

[B0r1sD]

I'm looking into this task.

Tool source: tools/image_processing/bia-ftplinks

Is it helpful to add this tool to the IUC tool repository, so we also make use of their tests and best practices?

[B0r1sD]

Started tracking the progress in this draft PR: bgruening/galaxytools#1541

• Added EDAM ontology:

  • https://edamontology.github.io/edam-browser/#topic_3383
  • https://edamontology.github.io/edam-browser/#operation_2422
    As this is my first time using it, a double check from someone would be nice!

• This point is not clear to me yet:

• The "storage mode" can only be nfs or fire, this should thus be a dropdown field.

• The help text of the field should say when to use which of the two options.

but I have found info about it in the following places:

• https://www.ebi.ac.uk/about/news/technology-and-innovation/fire-archive-100-petabytes/
• https://www.ebi.ac.uk/about/teams/its/blog/2022/05/data-archiving-through-the-generations/

And some context:
FIRE stands for FIle REplication, EMBL-EBI’s very large-scale object data storage system. This provides long-term sustainable storage, operational redundancy, and backup to tape. Dataset level metadata are stored in a MongoDB database. The system backend is coded in Kotlin.

• They are essential, added.

• Why are both fields optional? Is that correct?

[kostrykin]

Thanks @B0r1sD!

• This point is not clear to me yet:

• The "storage mode" can only be nfs or fire, this should thus be a dropdown field.

Right now "storage mode" is a text field, but according to the help text of the field, only two values are accepted (either nfs or fire). In that case, this field should either be a dropdown field, where one of the two options can be selected? Since any other input value would be invalid and invalid input should be prevented by a good UI.

• The help text of the field should say when to use which of the two options.

To me and @beatrizserrano it wasn't immediately clear how to determine the correct value for this input. I see from your explanations what either of the two is, but still, how is the user supposed to determine the correct value for input here? Can we add a help text here to provide some guidance?

[B0r1sD]

• Mail sent to BIA to ask about nfs/fire, exert from reply:

To answer your question, NFS is our older storage system and FIRE is the new one. So, we have some files still in NFS while the newer ones are on FIRE. There is not a recommended mode for FTP download, but you will need to use the correct one for each dataset. Sorry if this was unclear.
You can find out what is the storage mode for a dataset using this command:

curl https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fbiostudies%2Fapi%2Fv1%2Fstudies%2FS-BIAD570%2Finfo&data=05%7C02%7Cboris.depoortere%40vib.be%7C210c86d23fcb4025e91e08dcfd8c3c39%7C2d714a65b97f41a98ff1ec2cdf7df5cd%7C0%7C0%7C638664025957330959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=sV4V1uORpINn7y1pDnhDOh7HylKpCknw0Epra8EWDCA%3D&reserved=0 -s | jq -r .ftpLink

• Other checklist points addressed and committed to the PR.
• Looked into downloading subsets via wget. It should be possible, but have to look into how to implement it. Presumably let the user input a list of subsets and then implementing the download script from BIA in the wrapper.
  .
  Example of how the downloaded script looks like:

#!/bin/bash
# Run this file in bash with this command:  ./filename
HOST=ftp.ebi.ac.uk
USER=anonymous
ftp -pinv $HOST <<EOF
user $USER
cd biostudies/fire/S-BIAD/458/S-BIAD1458/Files
binary
mget "Red blood cell differential image data/data/0-0.3/0(11).jpg"
mget "Red blood cell differential image data/data/0-0.3/0(2).jpg"
disconnect
bye
EOF

[kostrykin]

• Mail sent to BIA to ask about nfs/fire, exert from reply:

To answer your question, NFS is our older storage system and FIRE is the new one. So, we have some files still in NFS while the newer ones are on FIRE. There is not a recommended mode for FTP download, but you will need to use the correct one for each dataset. Sorry if this was unclear.
You can find out what is the storage mode for a dataset using this command:
>
curl https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fbiostudies%2Fapi%2Fv1%2Fstudies%2FS-BIAD570%2Finfo&data=05%7C02%7Cboris.depoortere%40vib.be%7C210c86d23fcb4025e91e08dcfd8c3c39%7C2d714a65b97f41a98ff1ec2cdf7df5cd%7C0%7C0%7C638664025957330959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=sV4V1uORpINn7y1pDnhDOh7HylKpCknw0Epra8EWDCA%3D&reserved=0 -s | jq -r .ftpLink

Cool can we use the curl command in the tool wrapper to determine the correct mode automatically?

[B0r1sD]

Yeah that would be ideal.

[kostrykin]

Yeah that would be ideal.

Let me know if you need any help!

[kostrykin]

@B0r1sD What's your current state? We need to report our state tomorrow. It would be ideal if you could tick the boxes! 🥳