Skip to content

Commit 52df404

Browse files
authored
Fix links and enhance Slurm setup instructions
Updated links and clarified instructions for Slurm setup and BIOMERO web interface.
1 parent 6409f3b commit 52df404

1 file changed

Lines changed: 96 additions & 22 deletions

File tree

README.md

Lines changed: 96 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -132,8 +132,8 @@ To connect an OMERO processor to a Slurm cluster using the `biomero` library, us
132132
`python3 -m pip install omero-cli-zarr==0.5.3` && `yum install -y blosc-devel`
133133
- the [bioformats2raw-0.7.0](https://github.com/glencoesoftware/bioformats2raw/releases/download/v0.7.0/bioformats2raw-0.7.0.zip), e.g. `unzip -d /opt bioformats2raw-0.7.0.zip && export PATH="$PATH:/opt/bioformats2raw-0.7.0/bin"`
134134

135-
6. To finish setting up your `SlurmClient` and Slurm server, run it once with `init_slurm=True`. This is provided in a OMERO script form at [init/Slurm Init environment](https://github.com/NL-BioImaging/biomero-scripts/blob/master/init/SLURM_Init_environment.py) , which you just installed in previous step.
136-
- Provide the configfile location explicitly if it is not a default one defined earlier, otherwise you can omit that field.
135+
6. To finish setting up your `SlurmClient` and Slurm server, run it once with `init_slurm=True`. This is provided in an OMERO script form at [init/Slurm Init environment](https://github.com/NL-BioImaging/biomero-scripts/blob/master/admin/SLURM_Init_environment.py) , which you just installed in previous step.
136+
- You could provide the configfile location explicitly if it is not a default one defined earlier, but very likely you can omit that field.
137137
- Please note the requirements for your Slurm cluster. We do not install Singularity / 7zip on your cluster for you (at the time of writing).
138138
- This operation will make it create the directories you provided in the `slurm-config.ini`, pull any described Singularity images to the server (note: might take a while), and generate (or clone from Git) any job scripts for these workflows:
139139

@@ -147,50 +147,124 @@ With the configuration files in place, you can utilize the `SlurmClient` class f
147147

148148
# BIOMERO scripts
149149

150-
The easiest interaction from OMERO with this library currently is through our BIOMERO scripts, which are just a set of OMERO scripts using this library for all the steps one needs to run a image analysis workflow from OMERO on Slurm and retrieve the results back into OMERO.
150+
The basic interaction from OMERO with this library currently is through our BIOMERO scripts, which are just a set of OMERO scripts using this library for all the steps one needs to run a image analysis workflow from OMERO on Slurm and retrieve the results back into OMERO.
151151

152-
!!*NOTE*: Do not install [Example Minimal Slurm Script](https://github.com/NL-BioImaging/biomero-scripts/blob/master/Example_Minimal_Slurm_Script.py) if you do not trust your users with your Slurm cluster. It has literal Command Injection for the SSH user as a **FEATURE**.
152+
!!*NOTE*: Do not install [Example Minimal Slurm Script](https://github.com/NL-BioImaging/biomero-scripts/blob/master/Example_Minimal_Slurm_Script.py) if you do not trust your users with your Slurm cluster. It has literal Command Injection for the SSH user as a **FEATURE**. I have removed it from the latest releases, just to be clear.
153153

154154
We have provided the BIOMERO scripts at https://github.com/NL-BioImaging/biomero-scripts (hopefully installed in a previous step).
155155

156-
For example, [workflows/Slurm Run Workflow](https://github.com/NL-BioImaging/biomero-scripts/blob/master/workflows/SLURM_Run_Workflow.py) should provide an easy way to send data to Slurm, run the configured and chosen workflow, poll Slurm until jobs are done (or errors) and retrieve the results when the job is done. This workflow script uses some of the other scripts, like
156+
For example, [workflows/Slurm Run Workflow](https://github.com/NL-BioImaging/biomero-scripts/blob/master/__workflows/SLURM_Run_Workflow.py) should provide an easy way to send data to Slurm, run the configured and chosen workflow, poll Slurm until jobs are done (or errors) and retrieve the results when the job is done. This workflow script uses some of the other scripts, like
157157

158-
- [`data/Slurm Image Transfer`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/data/_SLURM_Image_Transfer.py): to export your selected images / dataset / screen as TIFF files to a Slurm dir.
159-
- [`data/Slurm Get Results`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/data/SLURM_Get_Results.py): to import your Slurm job results back into OMERO as a zip, dataset or attachment.
158+
- [`data/Slurm Image Transfer`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/_data/_SLURM_Image_Transfer.py): to export your selected images / dataset / screen as TIFF files to a Slurm dir.
159+
- [`data/Slurm Get Results`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/_data/SLURM_Get_Results.py): to import your Slurm job results back into OMERO as a zip, dataset or attachment.
160160

161161
Other example OMERO scripts are:
162-
- [`data/Slurm Get Update`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/data/SLURM_Get_Update.py): to run while you are waiting on a job to finish on Slurm; it will try to get a `%` progress from your job's logfile. Depends on your job/workflow logging a `%` of course.
162+
- [`data/Slurm Get Update`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/_data/SLURM_Get_Update.py): to run while you are waiting on a job to finish on Slurm; it will try to get a `%` progress from your job's logfile. Depends on your job/workflow logging a `%` of course.
163163

164-
- [`workflows/Slurm Run Workflow Batched`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/workflows/SLURM_Run_Workflow_Batched.py): This will allow you to run several `workflows/Slurm Run Workflow` in parallel, by batching your input images into smaller chunks (e.g. turn 64 images into 2 batches of 32 images each). It will then poll all these jobs.
164+
- [`workflows/Slurm Run Workflow Batched`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/__workflows/SLURM_Run_Workflow_Batched.py): This will allow you to run several `workflows/Slurm Run Workflow` in parallel, by batching your input images into smaller chunks (e.g. turn 64 images into 2 batches of 32 images each). It will then poll all these jobs.
165165

166-
- [`workflows/Slurm CellPose Segmentation`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/workflows/SLURM_CellPose_Segmentation.py): This is a more primitive script that only runs the actual workflow `CellPose` (if correctly configured). You will need to manually transfer data first (with `Slurm Image Transfer`) and manually retrieve data afterward (with `Slurm Get Results`).
166+
- [`workflows/Slurm CellPose Segmentation`](https://github.com/NL-BioImaging/biomero-scripts/blob/master/__workflows/SLURM_CellPose_Segmentation.py): This is a more primitive script that only runs the actual workflow `CellPose` (if correctly configured). You will need to manually transfer data first (with `Slurm Image Transfer`) and manually retrieve data afterward (with `Slurm Get Results`).
167167

168168
You are encouraged to create your own custom scripts. Do note the copy-left license enforced by OME.
169169

170+
# BIOMERO Web Interface
171+
172+
In addition to the BIOMERO scripts, BIOMERO 2.0 introduces a modern web-based user interface through the [OMERO.biomero](https://github.com/Cellular-Imaging-Amsterdam-UMC/OMERO.biomero) web plugin. This plugin provides a more intuitive and user-friendly way to interact with BIOMERO workflows directly from the OMERO.web interface.
173+
174+
## Features
175+
176+
The OMERO.biomero plugin offers:
177+
178+
- **Interactive Workflow Management**: Browse and launch available workflows with a modern web interface
179+
- **Real-time Progress Tracking**: Monitor job progress with live updates and detailed status information
180+
- **Workflow History**: View past workflow executions with full tracking and metadata
181+
- **Enhanced Parameter Configuration**: Configure workflow parameters through an intuitive form interface
182+
- **Data Import/Export**: Streamlined data transfer to and from the HPC cluster
183+
- **Dashboard Overview**: Get an overview of all your workflows and their status at a glance
184+
185+
## Installation
186+
187+
The OMERO.biomero plugin is included in the [NL-BIOMERO](https://github.com/Cellular-Imaging-Amsterdam-UMC/NL-BIOMERO) deployment stack and comes pre-configured in our Docker containers.
188+
189+
For manual installation:
190+
191+
```bash
192+
pip install omero-biomero
193+
```
194+
195+
Then add `omero_biomero` to your OMERO.web apps configuration:
196+
197+
```bash
198+
omero config append omero.web.apps '"omero_biomero"'
199+
```
200+
201+
## Usage
202+
203+
1. **Access the Interface**: Navigate to the "BIOMERO" tab in OMERO.web after selecting your images or datasets
204+
2. **Choose a Workflow**: Browse available workflows and select the one that fits your analysis needs
205+
3. **Configure Parameters**: Set workflow parameters through the intuitive web forms
206+
4. **Launch Job**: Submit your job to the HPC cluster with a single click
207+
5. **Monitor Progress**: Track your job status in real-time through the dashboard
208+
6. **Retrieve Results**: Import results back into OMERO when the workflow completes
209+
210+
## Database Integration
211+
212+
The web plugin integrates with BIOMERO's eventsourcing database to provide:
213+
214+
- Persistent workflow tracking across sessions
215+
- Detailed execution logs and metadata
216+
- Job progress visualization
217+
- Historical workflow analysis
218+
219+
This integration ensures that all workflow executions are properly tracked and that results remain linked to their source data and parameters, enhancing the FAIR principles of your research data.
220+
221+
## Comparison with BIOMERO Scripts
222+
223+
While the traditional [BIOMERO scripts](#biomero-scripts) remain available and fully supported, the web interface offers several advantages:
224+
225+
| Feature | BIOMERO Scripts | OMERO.biomero Plugin |
226+
|---------|----------------|---------------------|
227+
| **User Interface** | OMERO script dialog | Modern web interface |
228+
| **Progress Tracking** | Manual scripts calls needed | Real-time updates |
229+
| **Workflow History** | Only in the logs | Full database tracking |
230+
| **Parameter Configuration** | Basic script forms | Rich web forms |
231+
| **Job Management** | Single job focus | Dashboard overview |
232+
| **Batching** | Supported via script | Not supported (yet) |
233+
234+
For new users, we recommend starting with the web interface for its enhanced usability and features. Advanced users who need custom scripting capabilities can continue using the traditional BIOMERO scripts.
235+
236+
For more information about the new Admin interfaces, see [NL-BIOMERO documentation](https://cellular-imaging-amsterdam-umc.github.io/NL-BIOMERO/sysadmin/omero-biomero-admin.html)
237+
170238
# (Docker) containers
171-
We host BIOMERO container dockerfiles at [NL-BIOMERO](https://github.com/Cellular-Imaging-Amsterdam-UMC/NL-BIOMERO), which publishes container images to our public dockerhub [cellularimagingcf](https://hub.docker.com/repositories/cellularimagingcf). Specifically the [cellularimagingcf/biomero](https://hub.docker.com/repository/docker/cellularimagingcf/biomero/general) image is an OMERO processor container with BIOMERO library installed. When we release a new version of BIOMERO, we will also release a new version of these containers (because we deploy these locally at our Core Facility - Cellular Imaging).
239+
We host BIOMERO container dockerfiles at [NL-BIOMERO](https://github.com/Cellular-Imaging-Amsterdam-UMC/NL-BIOMERO), which publishes container images to our public dockerhub [cellularimagingcf](https://hub.docker.com/u/cellularimagingcf). Specifically the [cellularimagingcf/biomero](https://hub.docker.com/r/cellularimagingcf/biomero) image is an OMERO processor container with BIOMERO library installed. When we release a new version of BIOMERO, we will also release a new version of these containers (because we deploy these locally at our Core Facility - Cellular Imaging in Amsterdam).
172240

173241
You can mount your specific configurations over those in the container, for example:
174242

175243
```
176244
# Run the biomero container
177245
echo "Starting BIOMERO..."
178-
podman run -d --rm --name biomero \
179-
-e CONFIG_omero_master_host=omeroserver \
180-
-e OMERO_WORKER_NAME=biomero \
181-
-e CONFIG_omero_logging_level=10 \
182-
--network omero \
183-
--volume /mnt/datadisk/omero:/OMERO \
184-
--volume /mnt/data:/data \
185-
--volume /my/slurm-config.ini:/etc/slurm-config.ini \
246+
podman run -d --rm --name ${CONTAINER_BIOMERO} \
247+
-e CONFIG_omero_master_host="${CONFIG_OMERO_MASTER_HOST}" \
248+
-e OMERO_WORKER_NAME="${CONTAINER_BIOMERO}" \
249+
-e CONFIG_omero_logging_level="${CONFIG_OMERO_LOGGING_LEVEL}" \
250+
-e CONFIG_omero_upgrades_url="${CONFIG_OMERO_UPGRADES_URL}" \
251+
-e CONFIG_omero_scripts_timeout="${CONFIG_OMERO_SCRIPTS_TIMEOUT}" \
252+
-e PERSISTENCE_MODULE="${PERSISTENCE_MODULE}" \
253+
-e SQLALCHEMY_URL="${BIOMERO_SQLALCHEMY_URL}" \
254+
--network ${OMERO_NETWORK_NAME} \
255+
--volume "${OMERO_DATA_PATH}":/OMERO:z \
256+
--volume "${IMPORT_DATA_PATH}":/data \
257+
--volume /opt/omero/slurm-config.ini:/etc/slurm-config.ini:Z \
258+
--volume "${BIOMERO_SHARED_CONFIG_FILE}":/opt/omero/server/slurm-config.ini:z \
259+
--volume "$(pwd)/logs/${CONTAINER_BIOMERO}:/opt/omero/server/OMERO.server/var/log:Z" \
186260
--secret ssh-config,target=/tmp/.ssh/config --secret ssh-key,target=/tmp/.ssh/id_rsa --secret ssh-pubkey,target=/tmp/.ssh/id_rsa.pub --secret ssh-known_hosts,target=/tmp/.ssh/known_hosts \
187-
--userns=keep-id:uid=1000,gid=997 \
188-
cellularimagingcf/biomero:0.2.3
261+
--userns=keep-id:uid=${OMERO_SERVER_UID},gid=${OMERO_SERVER_GID} \
262+
cellularimagingcf/biomero:"$NL_BIOMERO_VERSION"
189263
```
190264

191265
This will spin up the docker container (in Podman) with omero config (`-e CONFIG_omero_..`), mounting the required data drives (`--volume /mnt/...`) and adding a new slurm config (`--volume /my/slurm-config.ini:/etc/slurm-config.ini`) and the required SSH settings (`--secret ...,target=/tmp/.ssh/...`) to access the remote HPC.
192266

193-
Note: the [BIOMERO scripts](https://github.com/NL-BioImaging/biomero-scripts) are installed on the [main server](https://hub.docker.com/repository/docker/cellularimagingcf/omeroserver/general), not on the BIOMERO processor.
267+
Note: the [BIOMERO scripts](https://github.com/NL-BioImaging/biomero-scripts) are installed on the [main server](https://hub.docker.com/r/cellularimagingcf/omeroserver), not on the BIOMERO processor.
194268

195269
Note2: We will also update these containers with our own desired changes, so they will likely not be 1:1 copy with basic omero containers. Especially when we start making a nicer UI for BIOMERO. We will keep up-to-date with the OMERO releases when possible.
196270

0 commit comments

Comments
 (0)