wip

Author: calvinchai

docker-compose.yaml

@@ -10,7 +10,7 @@ services:
     expose:
       - 5432
     volumes: 
-      - db:/var/lib/postgresql/data
+      - ./db:/var/lib/postgresql
     profiles: ["server"]
 
   ### MinIO for flow storage
@@ -76,7 +76,6 @@ services:
 
 volumes:
   prefect:
-  db:
   minio:
 networks:
   default:

docs/developer/oct_events_and_state.md

@@ -82,7 +82,7 @@ config block (`PSOCTScanConfig`); the payload may override individual keys
 (see `PROCESS_MOSAIC_FLOW_KWARGS_KEYS` in `opticstream.flows.psoct.utils`), same
 idea as LSM strip flows.
 
-Emitters include `mosaic_ident` (e.g. `state_management_flow`, `tile_batch_update_flow` via `emit_mosaic_psoct_event`).
+Emitters include `mosaic_ident` (e.g. `state_management_flow`, `check_mosaic_ready_hook` via `emit_mosaic_psoct_event`).
 
 ### Event payload structure
 

docs/getting-started/quickstart.md

@@ -1,5 +1,5 @@
 ---
-title: Quickstart
+title: Quickstart - LSM
 ---
 
 # Quickstart
@@ -34,12 +34,36 @@ opticstream --help      # or: ops --help
 
 ## 2. Configure an LSM project
 
-Create and save an `LSMScanConfig` block (see the configuration page for details),
-ensuring `project_base_path`, `info_file`, and `output_path` are set.
+```bash
+ops lsm setup --help
+```
+
+```bash
+ops lsm setup testscan
+```
+
+you should be able to see a block in prefect ui named testscan-lsm-config
+
+in there you will need to complete the detailed setup, each field will have doc in there
+
+in one window, start 
+```bash
+ops lsm serve 3
+```
+which will start 3 local workers to process lsm flows
+
+in another windows, start watcher
+```bash
+ops lsm watch testscan E:/temp/testscan
+```
+
+## 3. somethings goes wrong and restart a slice
+in most case, the watcher doesn't need to be restarted if the configuration is going to stay the same
+make sure the flow run for this slice are done/stopped , we dont want two flow run for the same strip running at the same time
+```bash
+ops lsm state reset testscan --slice SLICE_TO_RESET
+```
 
-## 3. Run a strip processing flow
+the watcher will emit events for this slice again and reprocess them
 
-Once your scan configuration block is saved, you can trigger the
-`process_strip_event` flow via Prefect by emitting the appropriate event
-payload (see the LSM workflow user guide for a full example).
 

docs/getting-started/quickstart_oct.md

@@ -0,0 +1,8 @@
+
+1. start the venv
+
+
+2. 
+```bash
+ops oct setup 
+```

docs/getting-started/setup.md

@@ -0,0 +1,168 @@
+---
+title: Setup
+---
+
+# Prefect Server Setup
+
+This guide walks through setting up the Prefect server for the first time on a
+host that already has Docker and Docker Compose installed.
+
+## Prerequisites
+
+- Docker Engine and **Docker Compose v2** installed on the host machine.
+- The OpticStream source repository cloned on the host (see [Installation](installation.md)).
+- Network access to the host from any machine that will use the Prefect UI or
+  submit flows (if running remotely).
+
+## 1. Configure the server URL
+
+The `docker-compose.yaml` in the repository root uses a `HOST_IP` variable to
+set the Prefect API and UI URLs. By default it falls back to `127.0.0.1`, which
+only allows access from the Docker host itself.
+
+If you need to reach the Prefect UI or API from **other machines on the
+network** (e.g. workers or browsers on different hosts), set `HOST_IP` to the
+host's LAN IP address or resolvable hostname.
+
+### Option A — create a `.env` file (recommended)
+
+Copy the provided example and edit the IP:
+
+```bash
+cd opticstream            # repository root
+cp .env.example .env
+```
+
+Then open `.env` and replace the default value:
+
+```dotenv
+# Use the host's LAN IP so other machines can reach the Prefect server.
+# Use 127.0.0.1 if you only access the server from the Docker host.
+HOST_IP=192.168.1.100
+```
+
+Docker Compose automatically loads `.env` from the same directory as the
+compose file, so no extra flags are needed.
+
+### Option B — export the variable in your shell
+
+```bash
+export HOST_IP=192.168.1.100
+```
+
+This is useful for one-off launches or CI environments where you don't want a
+persisted `.env` file.
+
+### What the variable controls
+
+Inside `docker-compose.yaml`, the `server` service references `HOST_IP` in two
+environment variables:
+
+```yaml
+- PREFECT_UI_URL=http://${HOST_IP:-127.0.0.1}:4200/api
+- PREFECT_API_URL=http://${HOST_IP:-127.0.0.1}:4200/api
+```
+
+`PREFECT_API_URL` tells Prefect clients where to send API requests, and
+`PREFECT_UI_URL` tells the browser-based dashboard where to find the API
+backend. Both must be reachable from whatever machine is connecting.
+
+## 2. Start the Prefect server
+
+From the repository root, bring up the server profile (which starts the
+PostgreSQL database and the Prefect server):
+
+```bash
+docker compose --profile server up -d
+```
+
+This pulls the required images on first run (`postgres:alpine` and
+`prefecthq/prefect:3-python3.12`), creates the `prefect-network` Docker
+network, and starts both containers in detached mode.
+
+### Verify the services are running
+
+```bash
+docker compose --profile server ps
+```
+
+You should see two containers (`database` and `server`) with status **Up**.
+
+Check the server logs to confirm a healthy start:
+
+```bash
+docker compose --profile server logs server
+```
+
+Look for a line similar to:
+
+```
+Configure Prefect to communicate with the server with:
+
+    prefect config set PREFECT_API_URL=http://…:4200/api
+```
+
+## 3. Open the Prefect dashboard
+
+In a browser, navigate to:
+
+```
+http://<HOST_IP>:4200
+```
+
+Replace `<HOST_IP>` with the value you configured (or `127.0.0.1` / `localhost`
+if accessing from the Docker host). You should see the Prefect UI dashboard.
+
+## 4. Point local clients at the server
+
+On any machine that will submit or monitor flows, tell Prefect where the server
+lives:
+
+```bash
+prefect config set PREFECT_API_URL=http://<HOST_IP>:4200/api
+```
+
+Or set it as an environment variable in your shell:
+
+```bash
+export PREFECT_API_URL=http://<HOST_IP>:4200/api
+```
+
+You can verify the connection with:
+
+```bash
+prefect version   # prints client version
+prefect server health  # should return healthy
+```
+
+## 5. Stopping and restarting
+
+Stop the server (containers are removed but volumes persist):
+
+```bash
+docker compose --profile server down
+```
+
+Restart later with the same command used to start:
+
+```bash
+docker compose --profile server up -d
+```
+
+All flow-run history and configuration is stored in the `db` Docker volume and
+survives restarts.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| Browser cannot reach `http://<HOST_IP>:4200` | `HOST_IP` is wrong or firewall blocks port 4200 | Verify `HOST_IP` matches the host's actual LAN IP; open port 4200 in the firewall |
+| `prefect server health` fails from another machine | `PREFECT_API_URL` on the client doesn't match `HOST_IP` | Re-run `prefect config set PREFECT_API_URL=…` with the correct address |
+| Database connection errors in server logs | The `database` container isn't running | Run `docker compose --profile server up -d` and check `docker compose --profile server ps` |
+| Containers exit immediately | Port 4200 or 5432 already in use on the host | Stop the conflicting service or change the port mapping in `docker-compose.yaml` |
+
+## Next steps
+
+- Continue with the [Quickstart](quickstart.md) to configure a scan and run
+  your first flow.
+- See [Configuration](configuration.md) for scan and project settings.

docs/infrastructure/acquistion_host.md

@@ -0,0 +1,55 @@
+---
+title: LSM Acquisition Host
+---
+
+# LSM acquisition host
+
+The LSM acquisition host is the machine that runs both the LSM microscope acquisition software and the OpticStream real-time processing pipeline simultaneously. Because the acquisition software is latency-sensitive — a backlogged acquisition process will drop frames — the host must be tuned to ensure sufficient CPU headroom is reserved exclusively for acquisition.
+
+## Host responsibilities
+
+- Runs the vendor acquisition software, which writes raw strip folders to a local or network path.
+- Runs the OpticStream LSM watcher (`opticstream lsm watch`) and Prefect worker, which detect new strips and trigger processing flows in real time.
+- Performs CPU-intensive work such as Zarr compression, MIP generation, and optional strip archival while acquisition is ongoing.
+
+## Performance optimisations
+
+### 1. Disable hyperthreading (BIOS)
+
+Hyperthreading should be disabled in the server BIOS. With hyperthreading enabled, two logical cores share one physical core's execution resources, which causes unpredictable latency spikes under the mixed real-time / batch load typical of an acquisition session. Disabling it gives each logical core exclusive access to its physical core and makes CPU scheduling more deterministic.
+
+This setting is found under the processor or advanced CPU configuration section of the BIOS/UEFI, often labelled **Hyper-Threading**, **Simultaneous Multithreading (SMT)**, or similar. It must be set before the OS boots; no OS-level change is needed.
+
+### 2. Limit CPU affinity for processing workers
+
+Even with hyperthreading disabled, the processing flows must not be allowed to compete with the acquisition software for CPU time. OpticStream enforces this by pinning its Prefect worker processes to a subset of CPU cores using the `cpu_affinity` field in the `LSMScanConfig` block.
+
+**How it works:** the `cpu_affinity` setting is passed directly to the underlying worker processes (e.g. via `os.sched_setaffinity` or equivalent), restricting them to the listed cores. The acquisition software runs on the remaining cores and is never preempted by processing work.
+
+**Example — host Willow (28 physical cores, 0-indexed):**
+
+Cores 0–13 are reserved for the acquisition software. The processing workers are pinned to cores 14–27:
+
+```yaml
+cpu_affinity: [14, 28]
+```
+
+Set this in the project's `LSMScanConfig` Prefect block via the Prefect UI:
+
+1. Open the Prefect dashboard at `http://<HOST_IP>:4200`.
+2. Navigate to **Blocks** and open the `LSMScanConfig` block for the project.
+3. Set the `cpu_affinity` field to the list of core indices the processing workers should use, e.g. `[14, 28]`.
+4. Save the block.
+
+As a rule of thumb, give the acquisition software roughly half the physical cores and let the processing pipeline use the other half. Adjust the split based on observed acquisition CPU usage — if the acquisition process starts queueing or dropping frames, expand the reserved range.
+
+## Monitoring
+
+Use **Windows Task Manager** to verify CPU load is distributed as expected during an active acquisition session:
+
+1. Open Task Manager (`Ctrl+Shift+Esc`) and go to the **Performance** tab.
+2. Click **CPU**, then right-click the CPU graph and select **Change graph to → Logical processors**. Each core is shown as a separate tile.
+3. Confirm that the acquisition software cores (0–13 on Willow) stay at low-to-moderate utilisation, while the processing cores (14–27 on Willow) are free to spike to 100 % during Zarr compression without affecting acquisition.
+
+If frame drops are observed despite correct affinity settings, check whether any other system process (e.g. Windows Update, antivirus, background services) is also running on the cores reserved for acquisition and reschedule or disable those tasks if possible.
+8

docs/infrastructure/storage_data_flow.md

@@ -0,0 +1,63 @@
+# Storage and data flow
+
+Acquisition hosts **Willow** and **Cedar**, and processing host **Zircon**, connect to the storage system over **SAS** (host HBAs to the storage server or expander fabric).
+
+```
+                         ┌──────────────────────────┐
+                         │      Storage server       │
+                         └───────────┬──────────────┘
+                                     │
+                        SAS (HBAs / multipath as deployed)
+                                     │
+              ┌──────────────────────┼──────────────────────┐
+              │                      │                      │
+              │ SAS                  │ SAS                  │ SAS
+              ▼                      ▼                      ▼
+     ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
+     │ Willow          │   │ Cedar           │   │ Zircon          │
+     │ (acquisition)   │   │ (acquisition)   │   │ (processing)    │
+     └─────────────────┘   └─────────────────┘   └─────────────────┘
+```
+
+## Host networking (25G vSwitch on Zircon, 10Gb wall)
+
+**Willow**, **Cedar**, **Zircon**, and **…** (fourth host, such as Maple) all **connect to Zircon** for east–west traffic: a **25 Gbps virtual switch** runs **on Zircon**, and every host has a **25 Gbps** leg into that vSwitch (Zircon’s own participation is on-box). Separately, **each host** has its own **10 Gbps** link to the building **wall** (patch / upstream).
+
+```
+                         ┌──────────────────────────────────┐
+                         │   10Gb wall (building uplink)   │
+                         └───┬────────┬────────┬────────┬───┘
+                             │        │        │        │
+                          10Gb    10Gb    10Gb    10Gb
+                             │        │        │        │
+                          Willow   Cedar   Zircon    …
+                             │        │        │        │
+                             └──25G───┴──25G───┴──25G───┴──25G──┘
+                                         │
+                                ┌────────┴────────┐
+                                │ Zircon          │
+                                │ vSwitch 25 Gbps │
+                                └─────────────────┘
+```
+
+East–west traffic between hosts uses the **25 Gbps vSwitch on Zircon** (hub: all hosts → Zircon). North–south: **10 Gbps** to the **wall** per host, independent of the vSwitch path (exact VLANs or ports per your network plan).
+
+## SMB file sharing (Zircon reads data on other hosts)
+
+**Zircon** reaches filesystems on the other hosts **over the same east–west path** (the **25 Gbps** fabric to Zircon’s vSwitch, not over SAS). Hosts that hold data **publish folders as SMB shares** (Windows file sharing: SMB/CIFS). **Zircon** mounts or accesses those shares as an **SMB client** so pipelines and services on Zircon can read (or write, if permitted) files as if they were local or UNC paths, without moving bulk data off the acquisition machines first.
+
+Typical pattern: **Willow** / **Cedar** (and other data hosts as needed) run an **SMB server** and share the relevant disk or directory; **Zircon** connects with credentials and uses Windows UNC paths (for example `\\Willow\Acquisition`) or Linux `//Willow/Acquisition` mounts (`cifs`, `smbclient`) over **25 Gbps**, which keeps latency and throughput suitable for large sequential reads compared to routing that traffic out the **10 Gbps wall** uplink.
+
+Operational notes: align **firewall** rules and **SMB signing** policy with your site; for Linux servers use **Samba** for SMB shares; ensure **service accounts** and **share permissions** match what processing jobs on Zircon expect.
+
+## Shared volume on acquisition and processing (dual attach, storage-mapped RO on Zircon)
+
+A less common option is to **present the same volume** (same LUN or equivalent) to **both** an acquisition host (**Willow** / **Cedar**) and **Zircon**. In many environments that is **discouraged or disallowed** when two hosts could **both write** the same filesystem.
+
+Here the **storage server** defines two **mappings** for that volume: the **acquisition host** sees it **read–write** so capture can write **NTFS** (or the host OS) normally. **Zircon** is given a **read-only** mapping to the **same** volume—enforced **below the OS** by the **array / presentation** (SCSI permissions, LUN mask, or equivalent), so **Zircon cannot write** even if a client tried. Only **one** host mutates data; **Zircon** is a **reader**.
+
+**Upside:** Reads on **Zircon** go over **SAS/block** with **lower overhead** and often **better throughput** than **SMB** over the 25 Gbps network.
+
+**Downside:** **Zircon** does **not** see changes in **real time** the way a network share can; metadata and cache may lag. Operators sometimes need to **unmount and remount** on **Zircon** (or refresh the mount) after new data lands on the acquisition side.
+
+**Protect acquisition:** Even read-heavy workloads on **Zircon** can **contend** with the array or paths the acquisition host relies on. **Throttle** sequential readers, **limit parallelism**, and **cap IOPS/bandwidth** on the processing host so **Zircon** does not **read fast enough to interfere** with ongoing capture (latency spikes, dropped frames, or storage saturation). Treat this as a **shared resource budget**, not “read as fast as possible.”

docs/infrastructure/storage_pool.md

@@ -0,0 +1 @@
+on windows we setup storage pool for the ssd, so it survive single drive failure, 

docs/user-guides/troubleshooting.md

@@ -16,7 +16,6 @@ Symptoms:
 
 Checks:
 
-- run `opticstream init blocks`
 - run `opticstream lsm setup PROJECT_NAME` or `opticstream oct setup PROJECT_NAME`
 - verify block names follow project naming conventions used by setup commands
 

opticstream/cli/__init__.py

@@ -14,6 +14,7 @@
 from .oct import oct_cli
 from .lsm import lsm_cli  # noqa: F401
 from .utils import utils_cli  # noqa: F401
+from .process_priority import process_priority_cli  # noqa: F401
 
 
 def main() -> None: