Add ome-zarr+zarr support & drop py < 3.10

Author: TorecLuik

.github/workflows/python-package.yml

@@ -11,11 +11,12 @@ on:
 jobs:
   build:
 
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        os: [ubuntu-latest, windows-latest]
+        python-version: ["3.10", "3.12"]
 
     steps:
     - uses: actions/checkout@v4
@@ -26,19 +27,18 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8
         python -m pip install --editable .[test]
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        flake8 biomero tests --count --select=E9,F63,F7,F82 --show-source --statistics
         # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+        flake8 biomero tests --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Test with pytest
       run: |
-        python -m pytest --cov=biomero --cov-report=xml
+        python -m pytest tests/unit/ --cov=biomero --cov-report=xml
     - name: Coveralls GitHub Action
-      if: matrix.python-version == '3.11'
+      if: matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest'
       uses: coverallsapp/github-action@v2.3.0
 
 

README.md

@@ -90,6 +90,37 @@ For a quick overview of what this library can do for you, we can install an exam
 
 # Prerequisites & Getting Started with BIOMERO
 
+## Installation
+
+### Using pip (Cross-platform)
+
+BIOMERO can be installed via pip with different dependency sets:
+
+```bash
+# Basic library only (core BIOMERO functionality)
+python3 -m pip install biomero
+
+# With BIOMERO.scripts requirements (includes BIOMERO.scripts dependencies)
+python3 -m pip install biomero[full]
+
+# Latest development version with BIOMERO.scripts requirements
+python3 -m pip install 'git+https://github.com/NL-BioImaging/biomero[full]'
+
+# With test dependencies (for local testing/development)
+python3 -m pip install biomero[test]
+
+# With both test and BIOMERO.scripts requirements
+python3 -m pip install biomero[test,full]
+```
+
+**Dependency sets explained:**
+- **Default (no extras)**: Core BIOMERO library for basic functionality
+- **`[test]`**: Adds pytest, coverage tools for local testing and development
+- **`[full]`**: Adds BIOMERO.scripts requirements (`ezomero`, `tifffile`, `omero-metadata`, `omero-cli-zarr`) which require complex dependencies like `zeroc-ice` and `omero-py`
+
+**Note**: The `[full]` dependencies require complex packages like `zeroc-ice` and `omero-py` that need system libraries. For OMERO integration, you may need to install these separately or use conda.
+
+
 ## Slurm Requirements
 Note: This library has only been tested on Slurm versions 21.08.6 and 22.05.09 !
 
@@ -106,12 +137,11 @@ Your Slurm cluster/login node needs to have:
 Your OMERO _processing_ node needs to have:
 1. SSH client and access to the Slurm cluster (w/ private key / headless)
 2. SCP access to the Slurm cluster
-3. Python3.8+
+3. Python3.10+
 4. This library installed 
-    - Latest release on PyPI `python3 -m pip install biomero`
-    - or latest Github version `python3 -m pip install 'git+https://github.com/NL-BioImaging/biomero'`
-5. Configuration setup at `/etc/slurm-.ini`
-6. Requirements for some scripts: `python3 -m pip install ezomero==1.1.1 tifffile==2020.9.3` and the [OMERO CLI Zarr plugin](https://github.com/ome/omero-cli-zarr).
+    - **With BIOMERO.scripts dependencies**: `python3 -m pip install biomero[full]`
+    - **Latest development**: `python3 -m pip install 'git+https://github.com/NL-BioImaging/biomero[full]'`
+5. Configuration setup at `/etc/slurm-config.ini`
 
 Your OMERO _server_ node needs to have:
 1. Some OMERO example scripts installed to interact with this library:
@@ -151,11 +181,12 @@ To connect an OMERO processor to a Slurm cluster using the `biomero` library, us
 
 !!*NOTE*: Do not install [Example Minimal Slurm Script](https://github.com/NL-BioImaging/biomero-scripts/blob/master/Example_Minimal_Slurm_Script.py) if you do not trust your users with your Slurm cluster. It has literal Command Injection for the SSH user as a **FEATURE**. 
 
-5. Install [BIOMERO.scripts](https://github.com/NL-BioImaging/biomero-scripts/) requirements, e.g.
-    - `python3 -m pip install ezomero==1.1.1 tifffile==2020.9.3` 
-    - the [OMERO CLI Zarr plugin](https://github.com/ome/omero-cli-zarr), e.g. 
-    `python3 -m pip install omero-cli-zarr==0.5.3` && `yum install -y blosc-devel`
-    - the [bioformats2raw-0.7.0](https://github.com/glencoesoftware/bioformats2raw/releases/download/v0.7.0/bioformats2raw-0.7.0.zip), e.g. `unzip -d /opt bioformats2raw-0.7.0.zip && export PATH="$PATH:/opt/bioformats2raw-0.7.0/bin"`
+5. 4. Install [BIOMERO.scripts](https://github.com/NL-BioImaging/biomero-scripts/) requirements:
+    - **If you installed `biomero[full]`**: All required dependencies (`ezomero`, `tifffile`, `omero-metadata`, `omero-cli-zarr`) are already included
+    - **Manual installation** (if using basic `biomero` without `[full]` extras): 
+      - `python3 -m pip install ezomero>=1.1.1 tifffile>=2020.9.3 omero-metadata>=0.12.0 omero-cli-zarr>=0.5.5` 
+      - Additional system dependencies may be required: `yum install -y blosc-devel`
+      - Install [bioformats2raw-0.7.0](https://github.com/glencoesoftware/bioformats2raw/releases/download/v0.7.0/bioformats2raw-0.7.0.zip): `unzip -d /opt bioformats2raw-0.7.0.zip && export PATH="$PATH:/opt/bioformats2raw-0.7.0/bin"`
 
 6. To finish setting up your `SlurmClient` and Slurm server, run it once with `init_slurm=True`. This is provided in an OMERO script form at [init/Slurm Init environment](https://github.com/NL-BioImaging/biomero-scripts/blob/master/admin/SLURM_Init_environment.py) , which you just installed in previous step.
     - You could provide the configfile location explicitly if it is not a default one defined earlier, but very likely you can omit that field. 

biomero/slurm_client.py

@@ -1606,8 +1606,19 @@ def run_conversion_workflow_job(self,
         data_path = f"{self.slurm_data_path}/{folder_name}"
         conversion_cmd, sbatch_env, chosen_converter, version = self.get_conversion_command(
             data_path, config_file, source_format, target_format)
+        
+        # Handle both .zarr and .ome.zarr extensions for backward compatibility
+        if source_format == 'zarr':
+            find_cmd = (f"find \"{data_path}/data/in\" -name \"*.zarr\" "
+                        f"-o -name \"*.ome.zarr\" | "
+                        f"awk '{{print NR, $0}}' > \"{config_file}\"")
+        else:
+            find_cmd = (f"find \"{data_path}/data/in\" "
+                        f"-name \"*.{source_format}\" | "
+                        f"awk '{{print NR, $0}}' > \"{config_file}\"")
+
         commands = [
-            f"find \"{data_path}/data/in\" -name \"*.{source_format}\" | awk '{{print NR, $0}}' > \"{config_file}\"",
+            find_cmd,
             f"N=$(wc -l < \"{config_file}\")",
             f"echo \"Number of .{source_format} files: $N\"",
             conversion_cmd
@@ -2284,7 +2295,7 @@ def get_logfile_from_slurm(self,
         return local_tmp_storage, export_file, result
 
     def get_unzip_command(self, zipfile: str,
-                          filter_filetypes: str = "*.zarr *.tiff *.tif"
+                          filter_filetypes: str = "*.zarr *.ome.zarr *.tiff *.tif"
                           ) -> str:
         """
         Generate a command string for unzipping a data archive and creating
@@ -2295,7 +2306,7 @@ def get_unzip_command(self, zipfile: str,
                 Without extension.
             filter_filetypes (str, optional): A space-separated string
                 containing the file extensions to extract from the zip file.
-                E.g. defaults to "*.zarr *.tiff *.tif".
+                E.g. defaults to "*.zarr *.ome.zarr *.tiff *.tif".
                 Setting this argument to `None` will omit the file
                 filter and extract all files.
 

pyproject.toml

@@ -12,13 +12,11 @@ authors = [
 ]
 description = "A python library for easy connecting between OMERO (jobs) and a Slurm cluster"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 keywords = ["omero", "slurm", "high-performance-computing", "fair", "image-analysis", "bioimaging", "high-throughput-screening", "high-content-screening", "cytomine", "biomero", "biaflows"]
 license = { file = "LICENSE" }
 classifiers = [
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
@@ -50,8 +48,15 @@ test = [
     "pytest",
     "pytest-cov",
     "mock",
+    "flake8",
     "psycopg2-binary"
 ]
+full = [
+    "ezomero>=1.1.1",
+    "tifffile>=2020.9.3",
+    "omero-metadata>=0.12.0",
+    "omero-cli-zarr>=0.6.1"
+]
 
 [project.urls]
 "Homepage" = "https://github.com/NL-BioImaging/biomero"

resources/slurm-config.ini

@@ -90,7 +90,7 @@ slurm_script_repo=
 # Please pin it to a specific version to reduce unforeseen errors.
 #
 # Key should be the types "X_to_Y" and value should be the docker image
-#zarr_to_tiff=cellularimagingcf/convert_zarr_to_tiff:1.14.0
+#zarr_to_tiff=cellularimagingcf/convert_zarr_to_tiff:2.0.0
 
 
 [MODELS]

tests/unit/test_slurm_client.py

@@ -148,7 +148,7 @@ def test_get_unzip_command(slurm_client):
     slurm_data_path = "/path/to/slurm/data"
     slurm_client.slurm_data_path = slurm_data_path
     zipfile = "example"
-    filter_filetypes = "*.zarr *.tiff *.tif"
+    filter_filetypes = "*.zarr *.ome.zarr *.tiff *.tif"
     expected_command = (
         f"mkdir \"{slurm_data_path}/{zipfile}\" \
                     \"{slurm_data_path}/{zipfile}/data\" \
@@ -422,8 +422,19 @@ def test_run_conversion_workflow_job(
         expected_sbatch_env["CONVERSION_PARTITION"] = f'"{conversion_partition}"'
 
     expected_conversion_cmd = 'sbatch --job-name=conversion --export=ALL,CONFIG_PATH="$PWD/$CONFIG_FILE" --array=1-$N "$SCRIPT_PATH/convert_job_array.sh"'
+    
+    # Handle special case for zarr format (.zarr and .ome.zarr)
+    if source_format == 'zarr':
+        find_cmd = (f'find "{expected_data_path}/data/in" -name "*.zarr" '
+                    f'-o -name "*.ome.zarr" | awk \'{{print NR, $0}}\' '
+                    f'> "{expected_config_file}"')
+    else:
+        find_cmd = (f'find "{expected_data_path}/data/in" '
+                    f'-name "*.{source_format}" | awk \'{{print NR, $0}}\' '
+                    f'> "{expected_config_file}"')
+
     expected_commands = [
-        f'find "{expected_data_path}/data/in" -name "*.{source_format}" | awk \'{{print NR, $0}}\' > "{expected_config_file}"',
+        find_cmd,
         f'N=$(wc -l < "{expected_config_file}")',
         f'echo "Number of .{source_format} files: $N"',
         expected_conversion_cmd,