idr0125 NGFF import

[will-moore]

On idr0125-pilot, going to test import of METADATA.ome.xml and then use mkngff to replace the fileset with full NGFF plate mounted on s3.

sudo cp idr0125/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr/OME/METADATA.ome.xml /opt/omero/server/
import as omero-server user...

omero import --transfer=ln_s --skip=all METADATA.ome.xml --file /tmp/idr0125_xml.log  --errs /tmp/idr0125_xml.err
$ tail -f /tmp/idr0125_xml.err
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-09-04 16:00:07,994 15499      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-09-04 16:00:31,170 38675      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-09-04 16:00:31,188 38693      [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /opt/omero/server/idr0125/METADATA.ome.xml...
2023-09-04 16:00:31,412 38917      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,219 39724      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,391 39896      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-09-04 16:00:32,673 40178      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 52201342

Other imported objects:
Fileset:5287519

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3447 images imported, 0 errors in 0:59:35.161

Running mkngff to replace OME.METADATA.ome.xml single-file Fileset with NGFF plate...

$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287519 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr > 5287519.sql

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04 // 16-00-19.149 for fileset 5287519
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr


$ psql -U omero -d idr -h $DBHOST -f 5287519.sql 
UPDATE 3447
BEGIN
 mkngff_fileset 
----------------
        5287569
(1 row)

COMMIT

[will-moore]

Updated bulk import yaml to import METADATA.ome.xml for every plate...

I went run bulk import on idr0125-pilot...

omero import --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20230918.log  --errs /tmp/idr0125_20230918.err

but found that this failed because every Plate.zarr directory is empty.

ls /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/

Could this be due to the update of symlinks in ManagedRepo? Commands are at https://github.com/IDR/idr0125-way-cellpainting/blob/d883ec3a88178b3aaf207acbf9612ef0e5e3f840/scripts/symlinks.bash

E.g.

cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-83/2022-12/14/10-51-00.924
rm -rf SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
ln -s /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

But this shouldn't affect what's under /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/ ??

[will-moore]

Instead, lets simply download and import the OME/METADATA.ome.xml files...

NB: this loop repeats for every - in each yaml row as well as the actual plate names!

for r in $(cat /uod/idr/metadata/idr0125-way-cellpainting/docs/_data/plates.yml); do
  cd
  cd idr0125
  echo $r
  mkdir -p $r
  cd $r
  wget https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$r/OME/METADATA.ome.xml
done

[will-moore]

Needed to rename each dir to remove .ome.zarr because trying to import e.g. SQ00015117__2016-04-13T18_12_45-Measurement1.ome.zarr/METADATA.ome.xml failed looking for .zgroup etc.

for i in $(ls ./); do
  mv $i ${i/.ome.zarr/}
done

for i in $(ls ./); do
  omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log  --errs /tmp/$i.err
done

[will-moore]

$ tail -f /tmp/SQ00014812__2016-05-23T20_44_31-Measurement1.err

2023-09-18 19:01:25,306 3563454    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 52313944
2023-09-18 19:01:26,455 3564603    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1/METADATA.ome.xml
Other imported objects:
Fileset:5287672

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 0:59:19.730

[will-moore]

Seems like import got stopped at some point (maybe due to server restart)...
Last log files are:

-rw-r--r--.  1 omero-server omero-server 4.3K Sep 19 12:33 SQ00015050__2016-05-24T06_38_27-Measurement1.err
-rw-r--r--.  1 omero-server omero-server    0 Sep 19 11:54 SQ00015050__2016-05-24T06_38_27-Measurement1.log

Moved the imported dirs out of idr0125/ into idr00125_imported/

ls idr0125_imported/
SQ00014812__2016-05-23T20_44_31-Measurement1  SQ00014816__2016-05-23T14_07_55-Measurement1  SQ00014820__2016-05-25T21_16_25-Measurement1  SQ00015044__2016-05-20T05_34_55-Measurement1	SQ00015048__2016-05-25T22_53_31-Measurement1
SQ00014813__2016-05-23T19_03_28-Measurement1  SQ00014817__2016-05-23T12_08_18-Measurement1  SQ00015041__2016-05-25T19_39_25-Measurement1  SQ00015045__2016-05-20T03_57_42-Measurement1	SQ00015049__2016-05-24T08_17_26-Measurement1
SQ00014814__2016-05-23T17_24_56-Measurement1  SQ00014818__2016-05-23T10_30_25-Measurement1  SQ00015042__2016-05-25T18_02_08-Measurement1  SQ00015046__2016-05-20T02_20_03-Measurement1	SQ00015050__2016-05-24T06_38_27-Measurement1
SQ00014815__2016-05-23T15_46_08-Measurement1  SQ00014819__2016-05-23T08_51_51-Measurement1  SQ00015043__2016-05-25T16_25_21-Measurement1  SQ00015047__2016-05-20T00_41_45-Measurement1

and restarted import in for loop as above, from idr0125/

[will-moore]

Moved SQ00015110__2016-05-05T04_50_06-Measurement1 and other Plates before it into idr0125_imported and restarted import from idr0125 as above...

for i in $(ls ./); do   omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log  --errs /tmp/$i.err

[will-moore]

For running mkngff on idr0125-pilot, we have data mounted at

$ ls /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/

Lets run mkngff on this Plate/Filesets... as regular wmoore user on idr0125-pilot...

(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ omero mkngff sql --secret=$SECRET 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr > idr0125_sql/5287672.sql
Using session for public@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024 for fileset: 5287672
...

$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287672.sql 
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5288197
(1 row)
COMMIT

as omero-server...

(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Viewing... http://localhost:1040/webclient/?show=well-2339987

EDIT: after 3 days this image(s) are still not viewable. Probably just takes too long for ZarrReader to read a BIG plate with all the data being remote in USA.

[will-moore]

All METADATA.ome.xml imported above.
Last batch - first plate SQ00015111__2016-05-05T03_10_03-Measurement1 created at 2023-09-28 15:02:03
Last plate imported (created) at 2023-10-02 10:23:45.

94 Plates imported in about 90 hours. 1 hour per plate!

[will-moore]

Since running mkngff above, and attempting to view image at http://localhost:1040/webclient/?show=well-2339987 (which still isn't visible) we now have ZarrReader update on idr0125-pilot.

Try another plate... 2nd plate: SQ00014813__2016-05-23T19_03_28-Measurement1

Fileset ID: 5287673

As wmoore user on idr0125-pilot...

omero mkngff sql --secret=$SECRET 5287673 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr > idr0125_sql/5287673.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585 for fileset: 5287673
...

(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287673.sql
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5288270
(1 row)

COMMIT

As omero-server user...

(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287673 "/cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr

View image at http://localhost:1040/webclient/?show=image-14847759 ...

[will-moore]

This image is also still not viewable. I think we have to conclude that this approach isn't going to work. There's probably just too many files to load remotely (from the US).

[will-moore]

Since the size of a "metadata-only" plate appears to be moderate, we can look again at the workflow tested earlier:

• download metadata-only copies of all plates
• import without chunks
• symlinks script to create symlinks (after rendering an image from each plate first)

(base) [wmoore@pilot-idr0125-omeroreadwrite idr0125]$ du -sh ./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/
82M	./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/