idr0004-thorpe-rad52 S-BIAD867

[will-moore]

idr0004-thorpe-rad52

[dominikl]

Fails with:

2023-02-27 11:54:05,356 [main] WARN  loci.formats.FormatHandler - Ignoring extra series for well #95
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/dlindner/bioformats2raw-0.7.0-SNAPSHOT/lib/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-02-27 11:54:35,469 [main] ERROR c.g.bioformats2raw.Converter - Error while writing series 91
java.lang.IllegalArgumentException: Invalid series: 91  index=91
        at loci.formats.FormatReader.seriesToCoreIndex(FormatReader.java:1267)
        at loci.formats.FormatReader.setSeries(FormatReader.java:928)
        at loci.formats.ReaderWrapper.setSeries(ReaderWrapper.java:375)
        at loci.formats.ReaderWrapper.setSeries(ReaderWrapper.java:375)
        at loci.formats.ReaderWrapper.setSeries(ReaderWrapper.java:375)
        at com.glencoesoftware.bioformats2raw.Converter.lambda$write$1(Converte

[sbesson]

The error rings a bell as I recall there were historical issues with some data from this historical submission.
Does this happen for all plates? If not, do you have a path to a representative sample?

[will-moore]

Going to use idr-ftp.openmicroscopy.org for conversion...

$ ssh -A idr-ftp.openmicroscopy.org
$ cd /data
$ sudo mkdir ngff && sudo chown wmoore ngff && cd ngff

$ conda create -n bioformats2raw python=3.9
$ conda activate bioformats2raw
$ conda install -c ome bioformats2raw

$ wget https://github.com/IDR/bioformats2raw/releases/download/v0.6.0-24/bioformats2raw-0.6.0-24.zip
$ unzip bioformats2raw-0.6.0-24.zip

$ sudo -Es git clone git@github.com:IDR/idr-metadata.git

$ mkdir idr0004
$ screen -S idr0004_ngff
$ conda activate bioformats2raw

(bioformats2raw) [wmoore@idrftp-ftp ngff]$ ./bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ./memo idr-metadata/idr0004-thorpe-rad52/screens/P101.screen idr0004/P101.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp5470699920951085994/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@16150369): java.io.FileNotFoundException: /uod/idr/filesets/idr0004-thorpe-rad52/Rad52_old/Rad52/P101/a2 (No such file or directory)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2192)
Caused by: java.io.FileNotFoundException: /uod/idr/filesets/idr0004-thorpe-rad52/Rad52_old/Rad52/P101/a2 (No such file or directory)
        at java.base/java.io.RandomAccessFile.open0(Native Method)
        at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)

Of course - idr-ftp doesn't have /uod/idr/filesets/ mounted!

[will-moore]

Size estimate of data...
uint16 (2 bytes per pixel), z:10 c:2, single field, 96-wells (some have missing wells), 47 plates

2 * 672 * 510 * 10 * 2 * 96 * 47 / 1000000000

61.8 GB

[joshmoore]

cc: @dgault in case it rings a bell for him as well.

[joshmoore]

@melissalinkert and I did some dredging of old synapses re: rad52 during the formats meeting:

• This is one of the original JCB datasets and used the original ScreenReader functionality (pre-IDR).
• We assume that we tested the IDR-ScreenReader functionality against it and that's what it should be using today. (@dominikl: you do have the IDR version, right?)
• If that's all the case, then it's unlikely that we want to try to touch the underlying format to fix this and additionally, the format won't have any metadata to speak of.
• ergo: candidate for omero-cli-zarr?

[dominikl]

I just used bioformats2raw, I didn't use any specific IDR reader version. Yes, probably easiest to use cli-zarr export. Let me try...

[dominikl]

Unfortunately doesn't work either:

Exporting to P101.ome.zarr (0.4)
Traceback (most recent call last):
  File "/home/dlindner/miniconda3/envs/myenv/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero_zarr/cli.py", line 125, in _wrapper
    return func(self, *args, **kwargs)
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero_zarr/cli.py", line 345, in export
    plate_to_zarr(plate, args)
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/omero_zarr/raw_pixels.py", line 311, in plate_to_zarr
    write_plate_metadata(
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/ome_zarr/writer.py", line 378, in write_plate_metadata
    "wells": _validate_plate_wells(wells, rows, columns, fmt=fmt),
  File "/home/dlindner/miniconda3/envs/myenv/lib/python3.9/site-packages/ome_zarr/writer.py", line 157, in _validate_plate_wells
    raise ValueError("Empty wells list")
ValueError: Empty wells list