idr0091-julou-lacinduction S-BIAD852

[will-moore]

idr0091-julou-lacinduction

[dominikl]

Issue with conversion:

(base) [dlindner@pilot-zarr2-dev idr0091]$ time /home/dlindner/bioformats2raw/bin/bioformats2raw --memo-directory ../memo /uod/idr/filesets/idr0091-julou-lacinduction/20200622-ftp/Julou_2020_lacInduction_RawImages/20170919/20170919_glyc_lac_1/20170919_glyc_lac_1_MMStack_metadata.txt 20170919_glyc_lac_1_MMStack.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp4590289654988610984/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2023-02-22 13:49:12,806 [main] ERROR loci.formats.Memoizer - deleting invalid memo file: ../memo/uod/idr/filesets/idr0091-julou-lacinduction/20200622-ftp/Julou_2020_lacInduction_RawImages/20170919/20170919_glyc_lac_1/.20170919_glyc_lac_1_MMStack_metadata.txt.bfmemo
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at ome.xml.model.Annotation.<init>(Annotation.java:123)
        at ome.xml.model.TextAnnotation.<init>(TextAnnotation.java:91)
        at ome.xml.model.XMLAnnotation.<init>(XMLAnnotation.java:97)

I even tried with export BF_MAX_MEM=56G But watching the process never got over 20G mem usage before crashing.

[sbesson]

Pretty sure that BF_MAX_MEM is specific to the Bio-Formats command-line utilities and will not be recognized by bioformats2raw. Have you tried JAVA_OPTS="-Xmx<NN>G" ?

[dominikl]

👍 It finally worked with export JAVA_OPTS="-Xmx50G" !

[sbesson]

50G definitely feels excessive. I recall some improvements were targeting at handling similar issues for large Micro-Manager metadata files in the past. One thing possibly worth testing independently is whether bioformats2raw 0.6.0 would handle the same data will lower memory requirements /cc @melissalinkert

Semi-related, I would expect this particular file format to work without issues with OMERO 5.6.6. What is our policy for these types of submissions of mixed file formats (probably only a handful of them)? Are we converting everything or only the minimal amount of data? /cc @jburel

[dominikl]

Oh, I should test a different image then. Didn't notice that this submission had different file formats.

[melissalinkert]

ome/bioformats#3229 is the last time we addressed memory issues in Micro-Manager, so I'd be surprised if bioformats2raw 0.6.0 helps. Based on the partial stack trace, I'd guess it's original metadata annotations that are causing the problem.

Comparing memory usage for showinf -nopix -omexml /uod/idr/filesets/idr0091-julou-lacinduction/20200622-ftp/Julou_2020_lacInduction_RawImages/20170919/20170919_glyc_lac_1/20170919_glyc_lac_1_MMStack_metadata.txt and showinf -nopix -omexml -no-sas /uod/idr/filesets/idr0091-julou-lacinduction/20200622-ftp/Julou_2020_lacInduction_RawImages/20170919/20170919_glyc_lac_1/20170919_glyc_lac_1_MMStack_metadata.txt should confirm whether that is indeed the issue.

[dominikl]

Also converted one of the pattern files, and re-imported. Worked fine. But the converted MMStack can't be re-imported, also memory issue:

2023-03-07 11:54:22,437 17151      [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /data/ngff/idr0091/20170920_glyc_lac_6h_1_MMStack.ome.zarr/OME/METADATA.ome.xml
java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded

[will-moore]

@dominikl - are you able to try the --no-sas option suggested by @melissalinkert above and see if that affects memory usage?

@melissalinkert If that is the case, does it suggest a workaround for bioformats2raw or is a fix still a much bigger issue?

A possible option is to use omero-cli-zarr to export since it's only 342 Images (according to IDR/idr-utils#56)

[melissalinkert]

bioformats2raw does not have a direct equivalent to bfconvert's -no-sas. The closest workaround at the moment is bioformats2raw --no-ome-meta-export, which entirely prevents OME/METADATA.ome.xml from being written; that's likely not what you want. I'm not opposed to adding an equivalent to -no-sas in bioformats2raw, but would like to know if that actually would solve the problem first.

[will-moore]

Going to start exporting with omero-cli-zarr since I can also do this on the idr-ftp machine which doesn't have the raw data mounted...

$ ssh -A idr-ftp.openmicroscopy.org
$ conda create -n omero_zarr_export -c ome python=3.9 zeroc-ice36-python
$ conda activate omero_zarr_export
$ conda install -c conda-forge omero-py
$ pip install git+https://github.com/will-moore/omero-cli-zarr.git@name_option
...
omero-cli-zarr-0.1.dev452+ge882a62

cd /data/ngff/
mkdir idr0091 && cd idr0091

Export 100 images

omero login
for id in 10648046 10648047 10648048 10648049 10648050 10648051 10648052 10648053 10648054 10648055 10648056 10648057 10648058 10648059 10648060 10648061 10648062 10648063 10648064 10648065 10648066 10648067 10648068 10648069 10648070 10648071 10648072 10648073 10648074 10648075 10648076 10648077 10648078 10648079 10648080 10648081 10648082 10648083 10648084 10648085 10648086 10648087 10648088 10648089 10648090 10648091 10648092 10648093 10648094 10648095 10648096 10648097 10648098 10648099 10648100 10648101 10648102 10648103 10648104 10648317 10648318 10648319 10648320 10648321 10648322 10648323 10648324 10648325 10648326 10648327 10648328 10648329 10648330 10648331 10648332 10648333 10648334 10648335 10648336 10648337 10648338 10648339 10648340 10648341 10648342 10648343 10648344 10648345 10648346 10648347 10648196 10648197 10648198 10648199 10648200 10648201 10648202 10648203 10648204 10648205; do
  echo $id;
  omero zarr export Image:$id --name_by name;
done

[will-moore]

After about 17 hours we have 50 images... (about 3 an hour):

(base) [wmoore@idrftp-ftp ~]$ ls -alh /data/ngff/idr0091
...
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 21:50 20151218_switch8h_pos2_GL02.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 22:16 20151218_switch8h_pos2_GL04.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 22:44 20151218_switch8h_pos2_GL05.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 23:07 20151218_switch8h_pos5_GL03.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 23:32 20151218_switch8h_pos5_GL05.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 11 23:52 20151218_switch8h_pos5_GL06.pattern.ome.zarr
drwxrwxr-x.  6 wmoore wmoore  100 Jul 12 00:18 20151218_switch8h_pos5_GL08.pattern.ome.zarr
drwxrwxr-x.  3 wmoore wmoore   42 Jul 12 00:18 20151218_switch8h_pos5_GL09.pattern.ome.zarr