Adding support for reading medical images

[steffenvogler]

There is a large need (and no solution) for structured sharing of medical data in context of ML development. Since I have a background in medical image analysis I would like to use Croissant specifically with medical images. Across the domain of cell biology, histo-pathology and radiology, there are may be 5 relevant file formats that need to be considered (each has a go-to Python lib already). Potential support for these file formats would make Croissant a very relevant library in the domain of life science.

(1) Is this generally of interest for the Croissant community?
(2) Should this go to a Croissant extension or part of core?

Also, I have mentioned Croissant as a solution in the proposal for this Arpa-h call about large scale exchange platform of imaging data (https://arpa-h.gov/explore-funding/programs/index). I.e. looking into this topic of medical data can be quite opportunistic.

[benjelloun]

Thanks for getting this effort started! IMO this is a very interesting use case for Croissant.

To start with, I would suggest examining whether any changes are needed to the Croissant spec to support this use case. This would determine whether a Croissant extension is needed or not.

Based on your description, it sounds like the main aspect is support for the 5 file formats used in this domain. If these have associated mime types, then it should be possible to describe the corresponding files in a Croissant description.

To make the MLCroissant python library work with those files, we probably need to integrate with the corresponding python libs for these formats.

Next, beyond describing resources as FileObject / FileSets, we need to map their contents to RecordSets. Is any of these formats structured? If so, is there a natural way to map their contents to Fields? If they are akin to columns, then just specifying the coressponding column ids / names is sufficient. If the data is hierarchical, then we need something like a path language.

[stefanches7]

@steffenvogler cool use case! We are also looking for the ways to integrate Bioimaging data with Croissant: #CroissantML > Publishing Bioimaging Datasets for AI, maybe there could be some synergies with medical imaging. And in my department we also extensively work with medical images, so happy to chat about this too :)

One thing that should be integrated in Croissant could be the RTSTRUCT annotation format - or maybe there are specialized Python libs that could facilitate the integration? As it is essentially a structured format, maybe we could map the annotations to RecordSets / Fields with appropriate semantic annotations.

[steffenvogler]

Thank you so much @benjelloun for rough direction of determining complexity of extending Croissant capabilities.

@stefanches7: Oh... I didn't see the MLCroissant Bioimaging thread on image.sc (I am rarely there). Since I am originally from bioimaging, I know both sides of the game and found the distinction between bio and medical imaging very unfortunate and suboptimal. Ideally we join forces, but without being spread thin between too many features.

Just technically, IMHO the file formats in bioimaging are way more advanced than in medical imaging (ZARR & OMERO vs DICOM & SVS) and may be we can help to connect the two domains. Also there is another dependency: there is a separate thread (only on email so far) around a concept of BioCroissant, where *omics data and lab measurements can be stored in a structured way. At the end, any image needs contextual data - otherwise it is a bit obsolete. I can provide more details in a call. I'll DM you on this.

[slobentanzer]

Hi all,
really cool to see this take off! We are working on prototypes for omics as well as genetics and prior knowledge (I am a PI at Helmholtz Munich and Open Targets). Would be happy to join the discussion. We already have a pipeline that dynamically ingests Open Targets data using its metadata descriptions (which will be Croissant ML in the next release). Would be awesome to connect also to the medical imaging domain (@stefanches7 thanks for making the link).

[stefanches7]

@slobentanzer maybe in a wider context the BioCroissant will be interesting too. They do work with omics, not sure about the prior knowledge - see https://docs.google.com/presentation/d/1uREePrWgJjYXySOJ-ylklGhPUHf_FLMcAWeDpHnZpPU/edit?usp=sharing for the intro.

There is apparently also the BioCroissant mailing list, but I could not find the link to it. Maybe you would still have it @steffenvogler ?

[steffenvogler]

@stefanches7 - the BioCroissant working group is just about to be officially created and so far there is only a lengthy email thread, that essentially tries to collect experts with potential interest. I will reply to that thread, put you in CC and do a little intro. You can follow up from there.

[joshmoore]

🤚🏽 I'd be interested in lurking, too.

[slobentanzer]

@steffenvogler @stefanches7 sorry for randomly bringing this up again, but I don't think I saw an email; can you invite me (possibly again)?

[St3V0Bay]

@slobentanzer @joshmoore @stefanches7 - Hi there, I went for a low hanging fruit: extending Croissant to DICOM using pydicom - link to PR above. This has a lot of potential for my peers in medical imaging and development of diagnostic tools for the clinic (no compromises with data lineage, extended metadata to detect annotator bias, intended use, responsible AI etc etc). There is no data transformation (that we discussed in a call a few months ago).

I also understand that bioimages and digital pathology are very different beasts to tame. But may be we can built some momentum here. We can do bioformats or openslide next, or even non-image formats. May be we have first topics for BioCroissant already.

[joshmoore]

Nice, @St3V0Bay! Looking at 9a95129 I definitely have some thoughts about how to do the same thing for bioimages. For pydicom, I'd suggest going with bioio. For the MIME Type we might have to get creative.

cc: @toloudis

[stefanches7]

Great thing, and definitely an inspiration for the Bioimaging world @St3V0Bay ! @joshmoore thanks for the bioio pointer. Another thing caught my attention that I thought is interesting for the round: #883 - FHIR record support for Croissant.

cc @nolden

[joshmoore]

Then I'll cc: @ericprud while we're at it ;)

[toloudis]

Nice, @St3V0Bay! Looking at 9a95129 I definitely have some thoughts about how to do the same thing for bioimages. For pydicom, I'd suggest going with bioio. For the MIME Type we might have to get creative.

cc: @toloudis

Yes, a DICOM reader as bioio package (wrapping pydicom or something) would be very interesting...