Image features and important Metadata fields missing from parquets

[kate-bowers-broad]

Hi there! I have been trying out CytoTable for the first time, following the tutorial for turning CellProfiler analysis CSVs into Parquet files (https://cytomining.github.io/CytoTable/tutorials/cellprofiler_to_parquet.html).

My code:

from cytotable import convert
import pandas as pd
import pyarrow.parquet as pq

source_path = "s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/analysis/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/analysis/BR00147139-A01-1/"
convert(
    source_path=source_path,
    source_datatype="csv",
    dest_path="cytotable_kb3",
    dest_datatype="parquet",
    concat=True,
    compartments=("cells", "nuclei", "cytoplasm", "image"),
    preset="cellprofiler_csv",
    no_sign_request=True,
    join=True,
    parsl_config=None
)

When I compare the columns in this parquet file to the columns in the backends CSV made for this plate by pycytominer collate.py, I see that there are 1300+ columns missing from the parquet file. These include Image measurements (ie Image_Granularity measurements, Image_Texture measurements, etc), Metadata_Plate,Metadata_Well, Metadata_Site_Count,Metadata_Object_Count, and all the Counts like Metadata_Count_Cells.

The pycytominer-made backends CSV I compared to is here: s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/backend/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/BR00147139.csv

I looked through the Cytotable documentation, but I couldn't figure out how to get these metadata and image measurements columns in my Cytotable parquet files. Am I missing a setting or command here? Thanks very much!

[d33bs]

Hi @kate-bowers-broad - thanks for opening this issue! The presets used to join compartment and image table data are opinionated and historically have cut much of the image table data. That said, you can pick and choose what parts of the presets you use, including the join logic. You can customize the join using the joins parameter and a SQL string argument. Reference the presets.py module for examples of how the presets work (including the joins). Here's a quick example based on what you provided that includes all image table data: https://gist.github.com/d33bs/7fc4670cb974e47c32a8aba109c76f7b .

Hope this helps but let me know if I can clarify anything!

Even if this addresses the technical issue we probably still have work to do in terms of documentation because you didn't find what you were looking for. Would you have a suggestion of where we could put this kind of information? Thanks in advance for your thoughts!

[bethac07]

Hey Dave, thanks for the quick feedback! I'll let Kate decide what works best for her re: how to move forward here in particular, but can I hijack this to ask if there's a reason not to have a preset (cellprofiler_csv_full ?) that in practice just passes everything through? It feels like that would be a more "default" behavior than opinionated metadata stripping, and even if it wasn't the dev team's default, I suspect it would improve utility for us and a substantial fraction of the potential user base. Thanks much!

[kate-bowers-broad]

This is really helpful, thank you Dave! I tried out the example that you shared and it does add in a bunch of image measurements, which is great. I haven't written SQL before, so it is tough to come up with these joins myself! As someone new to SQL, it can be hard to tell what is being included or excluded.

I agree with Beth's idea - it would be helpful to have a preset which keeps the metadata/image measurements/etc. I could then start with all of the columns, know my options, then drop columns I don't want to keep with pandas (or whatever package one prefers).

For your documentation question - I see now that this probably falls under the "preset manual override" description in the overview: "Please note that these presets may not capture all possible outcomes. Use manual overrides within convert() as needed." I wasn't sure quite what this meant. It could be clearer to describe when manual overrides would be needed, ie "Adjust which columns are included by editing CONFIG_JOINS within convert() as needed."

I think it would help to add information about this to the tutorials (I followed the Cellprofiler-to-Parquet tutorial), as that is where beginner users will start. I think you could add a note at the top, or in the "Step 3: check that the outputs look right" step, saying that if you are expecting different columns you can do XYZ to change them.

Thanks again for your help Dave!

[gwaybio]

Love to see this discussion, quick resolution, and actionable feedback!

In addition to clarifying the docs, generating a new preset (maybe cellprofiler_full_metadata; I'm not sure if this should be specific to "csv"?) sounds good to me. I also think this would be relatively easy development. @kate-bowers-broad - perhaps the two of us could tag-team a review that @d33bs posts in the next few weeks?

[d33bs]

Hi @kate-bowers-broad - if you're interested and have a moment I'd love to get your thoughts on #426 and/or #427 . Thank you for your help in detailing the issues here and for any feedback on the changes.