Merge branch 'main' into support_sharding

Author: LucaMarconato

.claude/skills/memray/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: memray
+description: Profile the memory usage of a Python script using memray and visualize a temporal flamegraph in the browser. Use when the user wants to investigate memory consumption, find leaks, or understand allocation patterns.
+compatibility: Requires the pixi profiling environment (pixi run -e profiling). Supports Linux and macOS.
+allowed-tools: Bash(pixi run -e profiling memray-run:*) Bash(pixi run -e profiling memray-flame:*) Bash(open:*) Bash(xdg-open:*) Bash(python -m webbrowser:*)
+---
+
+## Steps
+
+1. Ask the user which script to profile (full or relative path).
+
+2. Run the script under memray:
+
+    ```bash
+    pixi run -e profiling memray-run script.py
+    ```
+
+    This produces a binary file named `memray-script.py.<pid>.bin` in the current directory.
+
+3. Generate the flamegraph HTML report from the `.bin` file:
+
+    ```bash
+    pixi run -e profiling memray-flame memray-script.py.<pid>.bin
+    ```
+
+    Replace `<pid>` with the actual PID shown in the filename. This writes `memray-flamegraph-script.py.<pid>.html`.
+
+4. Open the report in the browser:
+    - macOS: `open memray-flamegraph-script.py.<pid>.html`
+    - Linux: `xdg-open memray-flamegraph-script.py.<pid>.html`
+    - Either: `python -m webbrowser memray-flamegraph-script.py.<pid>.html`
+
+## Notes
+
+- The `--temporal` flag (included in `memray-flame`) shows memory over time, not just peak — use this to spot leaks and allocation bursts.
+- To find the `.bin` file if unsure of the name: `ls memray-*.bin`
+- To compare runs, save the previous report: `cp memray-flamegraph-script.py.<pid>.html memray-flamegraph-before.html`

.claude/skills/profimp/SKILL.md

@@ -0,0 +1,30 @@
+---
+name: profimp
+description: Profile Python import time using profimp and open a waterfall HTML report. Use when investigating slow startup or wanting to identify which imports are most expensive.
+compatibility: Requires profimp (available as a pixi dependency). macOS or Linux.
+allowed-tools: Bash(profimp:*) Bash(open:*) Bash(xdg-open:*) Bash(python -m webbrowser:*)
+---
+
+## Steps
+
+1. Ask what to profile. Suggest common patterns for this repo:
+    - `import spatialdata`
+    - `from spatialdata import SpatialData`
+    - `from spatialdata_io import xenium`
+
+2. Run:
+
+```bash
+profimp --html "<import_stmt>" > /tmp/profimp.html
+```
+
+3. Open the report:
+    - macOS: `open /tmp/profimp.html`
+    - Linux: `xdg-open /tmp/profimp.html`
+    - Either: `python -m webbrowser /tmp/profimp.html`
+
+## Notes
+
+- The report is a waterfall chart showing every sub-import and its timing.
+- To compare before/after: `cp /tmp/profimp.html /tmp/profimp-before.html` before re-running.
+- `pixi run python -m profimp` also works if `profimp` is not on PATH.

.claude/skills/pyspy/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: pyspy
+description: Profile the execution time of a Python script using py-spy and visualize the result with speedscope. Use when the user wants to benchmark performance, find slow code paths, or profile CPU time.
+compatibility: Requires the pixi profiling environment (pixi run -e profiling). Speedscope must be installed separately (npm install -g speedscope). sudo is required on macOS.
+allowed-tools: Bash(pixi run -e profiling pyspy:*) Bash(pixi run -e profiling speedscope:*) Bash(sudo pixi run -e profiling pyspy:*)
+---
+
+## Steps
+
+1. Ask the user which script to profile (full or relative path).
+
+2. Run py-spy to record the profile. The output is always written to `profile.speedscope.json` in the current directory.
+
+    **Linux** (no sudo needed):
+
+    ```bash
+    pixi run -e profiling pyspy script.py
+    ```
+
+    **macOS** (sudo required — py-spy needs to attach to the process):
+
+    ```bash
+    sudo pixi run -e profiling pyspy script.py
+    ```
+
+    If sudo fails to find the pixi environment, use absolute paths:
+
+    ```bash
+    sudo /path/to/.pixi/envs/profiling/bin/py-spy record --gil \
+      -o profile.speedscope.json --format speedscope \
+      -- /path/to/.pixi/envs/profiling/bin/python script.py
+    ```
+
+3. Open the result in speedscope:
+    ```bash
+    pixi run -e profiling speedscope
+    ```
+    This opens `profile.speedscope.json` in the browser via the local speedscope CLI.
+
+## Notes
+
+- If the speedscope view is blank, switch threads using the thread selector in the top-right corner.
+- To save a profile before overwriting: `cp profile.speedscope.json profile-before.speedscope.json`
+- `--gil` records only time when the GIL is held (Python-level CPU time). Drop it to include C extension time.
+- speedscope must be installed globally: `npm install -g speedscope`

.github/workflows/test.yaml

@@ -19,12 +19,12 @@ jobs:
             fail-fast: false
             matrix:
                 include:
-                    - {os: windows-latest, python: "3.11", dask-version: "2026.3.0", name: "Dask 2026.3.0"}
-                    - {os: windows-latest, python: "3.13", dask-version: "latest", name: "Dask latest"}
-                    - {os: ubuntu-latest, python: "3.11", dask-version: "latest", name: "Dask latest"}
-                    - {os: ubuntu-latest, python: "3.13", dask-version: "latest", name: "Dask latest"}
-                    - {os: macos-latest, python: "3.11", dask-version: "latest", name: "Dask latest"}
-                    - {os: macos-latest, python: "3.13", prerelease: "allow", name: "Python 3.13 (pre-release)"}
+                    - {os: windows-latest, python: "3.11", dask-version: "2026.3.0", name: "min dask"}
+                    - {os: windows-latest, python: "3.14", dask-version: "latest"}
+                    - {os: ubuntu-latest, python: "3.11", dask-version: "latest"}
+                    - {os: ubuntu-latest, python: "3.14", dask-version: "latest"}
+                    - {os: macos-latest, python: "3.11", dask-version: "latest"}
+                    - {os: macos-latest, python: "3.14", prerelease: "allow", name: "prerelease"}
         env:
             OS: ${{ matrix.os }}
             PYTHON: ${{ matrix.python }}
@@ -59,7 +59,7 @@ jobs:
                   PLATFORM: ${{ matrix.os }}
                   DISPLAY: :42
               run: |
-                  uv run pytest --cov --color=yes --cov-report=xml
+                  uv run pytest --cov --color=yes --cov-report=xml -n auto --dist worksteal
             - name: Upload coverage to Codecov
               uses: codecov/codecov-action@v5
               with:

.gitignore

@@ -46,11 +46,18 @@ spatialdata-sandbox
 # version file
 _version.py
 
-# other
-node_modules/
+# agents configurations
+.claude/settings.local.json
 
+# benchmarking and profiling
 .asv/
+profile.speedscope.json
+
+# other
+node_modules/
 
 .mypy_cache
 .ruff_cache
 uv.lock
+pixi.lock
+

.pre-commit-config.yaml

@@ -9,18 +9,18 @@ ci:
     skip: []
 repos:
     - repo: https://github.com/rbubley/mirrors-prettier
-      rev: v3.8.1
+      rev: v3.8.3
       hooks:
           - id: prettier
             exclude: ^.github/workflows/test.yaml
     - repo: https://github.com/pre-commit/mirrors-mypy
-      rev: v1.19.1
+      rev: v2.0.0
       hooks:
           - id: mypy
             additional_dependencies: [numpy, types-requests]
             exclude: tests/|docs/
     - repo: https://github.com/astral-sh/ruff-pre-commit
-      rev: v0.15.2
+      rev: v0.15.12
       hooks:
           - id: ruff
             args: [--fix, --exit-non-zero-on-fix]