Standardize representation of tracking results in Galaxy

[kostrykin]

To keep the PR discussion cleaner… @xulman galaxyproject/tools-iuc#7893 (comment)

Galaxy developers would need to address how to represent the tracking

What we already have: The representation that we currently use in tools that we already have in Galaxy like https://github.com/BMCV/galaxy-image-analysis/blob/master/tools/points_association_nn/points_association_nn.xml is that we use one tabular file per track, and each tabular file has at least 3 columns. The first row of each tabular file contains the column names frame, pos_x, pos_y, where frame is the frame index (0-based indexing).

Trackastra-Galaxy PR: From looking at test-data/track_only/man_track.tsv, I assume that the representation that you currently use is a tabular file with 4 columns. It looks like column 1 is the frame index, and the other 3 columns are the track ID, x position, y position, I suppose? But in which order?

If I'm right, then changing the representation to something that we already have in Galaxy should be quite easy, and it would certainly facilitate tool interoperability and downstream analysis with tools that we already have:

(a) Add column names to the first row of the file.

(b) In addition to (a), adopt the column names frame, pos_x, pos_y.

(c) Use one tabular file with those columns per track, multiple files grouped together as a collection in Galaxy to represent the set of all tracks in an image sequence. The main motivation for this is, that otherwise the file gets quite large and difficult to inspect manually. Using one file per track eases the manual inspection.

What you think?

[bernt-matthias]

So the difference is if you have 1 table per object or 1 table containing the data for all objects?

I guess if they have standardized column names it does not really matter and they can easily be distinguished and transformed. It will also depend on the application.

From Galaxy's point of view the advantage of having all in one dataset is that only one dataset has to be stored 😅

[kostrykin]

Yes, having column names consistent would be very useful, but the question whether all tracks are stored in a single dataset is secondary, I agree.

[xulman]

Hi, let me join also this party :-)

Correction and explanation around of the CTC (Cell Tracking Challenge) format, aka Trackastra-Galaxy PR:

The output of tracking (according to the CTC format) is:

• time-lapse series of tiffs, each showing instance segmentation
• man_track.txt, a file that explains division relation-ship of the tracked objects

What's in the man_track.txt?
It is one file for the whole lineage. Each row is

• track_id (> 0!),
• starting_tp,
• ending_tp (tp = time point = frame index), starting <= ending,
• parent_track_id (can be 0 if this track_id hasn't been introduced via the cell division).

It especially carries no spatial information. From the txt file, one can only see how many tracks are there in total, the time span of each track - when it started to exist, how long it lasted, and when it disappeared, and who is the daughter of whom.

The track_id is not any random number. In fact, the tiff files complement the txt file by providing masks (pixels with values corresponding to track_id), each mask shows the position and extent of the tracked object (nucleus or cell usually).

For example, if a txt file contains a line 10 20 30 4 that means

• in every tiffs with indices between [20, 30] (inclusively) there has to be (in each file) at least one pixel with value 10,
• in tiff at index <20, there must be a pixel with value 4 (parent/mother cell).

Obviously, the tiffs, when carefully labeled, could easily tell the tracking - when a cell started to appear, where it was over the time lapse. The tiffs cannot tell the division, and that's why the txt has been introduced to the CTC format. The txt additionally has a little redundancy (because it requires a time span for each track) to offer (a very naive) mean for consistency checking.

Ooh, and the txt file cannot be supplemented with a header row (sadly!), as this file would not meet the CTC format specification. The CTC-ready tools are simply not ready for this.

[xulman]

A modern alternative to the CTC format could be the GEFF, src and paper. However, in both cases, they are a folder with content, a unit of information is the folder, the handle of the data is the folder. CTC is tiffs+txt, GEFF is zarr (folder structure with chunks-files and jsons). That said, the whole tracking is not represented with just one file :(

[xulman]

If Galaxy has its own established format (let's nickname it G), why not create nodes to convert CTC <-> G and GEFF <-> G? The nodes would fit to the workflows nicely, no? It doesn't feel difficult from what I read about G above.

Or are there many tools supporting G?
Wouldn't it be better to adopt these nodes/tools to work with either CTC or GEFF? (Here, I have very little knowledge.)

PS: Consider also pos_z in G.

[kostrykin]

Galaxy has no established format yet, just a few tools that use a specific format. Which emerged kind of ad-hoc. I saw that the format produced by your PR @xulman was quite similar at a first glance, and hence I hoped we could homogenize. But now I see that your format is actually CTC, I wasn't aware of that (and I think you are rather the tracking expert than me).

My concern with CTC is, that it builds upon segmentation, but, technically and strictly speaking, you can do tracking without segmentation, e.g., via detection. There have even been attempts to do tracking directly (without detection or segmentation). So, overall, I think, while CTC is an established format and hence nice to support, it is somewhat specific and not that general. How do alternatives like GEFF play in here?

For that reason I think it is nice to support CTC, but—meanwhile—keep the discussion going for how we maybe can commit to some more general, flexible format, that is not strictly coupled to segmentation masks, also keeping cross-domain applications in mind that might reach beyond bioimaging (and might need to cover not only division-events but also merge-events).

[kostrykin]

Would it maybe make sense to add the format of the man_track.txt as a specific data type to Galaxy? Something like this: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/tabular.py

[xulman]

Well, in that Trackastra PR and what had happened before it, I was examining GEFF prior to the CTC format. The GEFF worked nicely in my hands at first, but then it didn't when I came back to revisit the project -- I guess rebuilding the Python environment had just chosen a different set of versions, and something just didn't snap... anyway. So I went to the CTC -- which I think used to be among the most established in the field (perhaps I'm too enthusiastic (or nostalgic?) about it), and that worked well. And it is still folder-centric, too, so no gain or loss on that end.

Btw, CTC can represent detections. It's just a matter of how accurate the marker/pixel patch one saves into the TIFFs (detailed marker -> segmentation, simple small circle -> detection); and yes, I hear you, saving one-pixel larger markers (as detections) is super inefficient. It's interesting to read the exact same complaint about CTC in the GEFF paper :). Anyway, CTC cannot handle overlapping objects; GEFF can.

Today, I would say that CTC could be superseded by the GEFF (I would be the one to support this change!), which really has several advantages over it. GEFF can be, but does not need to be, coupled to segmentation masks:

``GEFF’s object specification supports a wide range of shapes, enabling its application across
diverse fields in the life sciences. The library integrates multiple representation formats, from
binary masks (2D/3D), commonly used in cell biology, developmental biology, and natural
image tracking, to geometric primitives (points, circles, ellipses, spheres, and ellipsoids), which
are essential in super-resolution microscopy, virology, and developmental studies. It also includes
polygons and meshes for detailed structural analysis in cell biology, microbiology, and complex
shape modeling, as well as pose-based representations for markerless tracking of anatomical
keypoints in multi-animal and behavioral research.''
(page 2, bottom, from the paper)

I think this already satisfies Leonid's wish/expectations for Galaxy std. format for tracking, no?

[xulman]

To make myself clear: I think Galaxy would do the best to focus on GEFF.

• GEFF is superior in what it can represent than CTC... shape-wise & space/storage-wise
• GEFF is more cross-domain, CTC was designed for nuclei and cells (not even for particles)
• GEFF library promises conversions from/to CTC (haven't tested myself)

We could do GEFF <-> G convertors, should not be difficult
(I've played with GEFF a little recently, and it is really easy... just graphs at the end of the day :-) )

[kostrykin]

Thanks for the insights!

For the record, here are the docs of the GEFF Python package: https://liveimagetrackingtools.org/geff/latest