Complete reworking of client/server architecture to setup inference with the Raspberry Pi (#15)

This pull request introduces a simplified structure to the code base.
There is an improved web app for the client.
Web socket connections between the client's server and the inference server.
Things are logged properly now.
Better configurations through YAML files.
Moved away from this whole memory buffer architecture to go for a simpler approach.
Now we're abstracting away with a job manager to handle how jobs are called, how they are triggered, and how they log their output back to the client.
There is also HTTPS enabled so that the Raspberry Pi's webserver can let the user use their own camera to do inference on.
In general, code in place was simplified, to simpler architecture and configuration to avoid debugging hassle.

The following is a detailed view of the commits that made this possible!

* Adapting Client: Add inference results display to the client

Implements a WebSocket endpoint on the client to receive and display inference results from the server.

This includes:
- A new "/results" WebSocket route in the client web app
- Updates to the client-side JavaScript to connect to the new endpoint and display results in a dedicated section
- Modification of the StreamingClient to use a callback to store latest inference results.
- Updates the default ports of client and server to avoid collision.

* Tweaks to improve server architecture

* Small linter changes

* Client Refactor: Improved logging calls

Uses the logger instead of print
Additionally, the client frontend should reflect the logs correctly

* Better logging management server-side

Logging is now more streamlined, and allows for better configuration of these outputs to the client.
Also encompasses better logging like GPU metrics and more.

* Implement flexible job system with frame collection and video inference

Added a multi-job architecture to support frame buffering and batch inference:

- there is job configuration, to allow per job to configure settings.
- A Job factory is used to instantiate a new job in a type-safe way.
- Implemented WIP FrameCollectionJob and VideoInferenceJob
- Added a Job control API to start, delete and retrieve on-going jobs
- Also decoupled the websocket job creation to allow for any jobs.
- Polymorphic implementation to handle responses from jobs

* Centralized configuration with YAML files

Introduced a centralized YAML configuration for both serving and training settings.

Using YAML to configure, and Pydantic to load, removing the previous `src/iris/server/config.py` as it is replaced by the new system.

Also updated frame collection job defaults for 1 FPS streaming, adjusting trigger conditions and frame skipping.

Finally, added memory buffer initialization in the server, enabling it if configured.

* Fixes on the client, for better display, and camera control

* Optimizations to run on pi

* typos and tiny fixes to pyproject.toml

* HTTPS self cert

* Implement SSH tunneling from the client

* Fixes fo UI tunnel connection, and server shutdown

* Remove setup for memory buffer, not pursuing that anymore

* Simplified Job classes and more flexible trigger system for launching them

TriggerConfig.should_trigger() encapsulates trigger logic.
Jobs manage their own triggers (more responsive than centralized polling).

Unified FrameCollectionJob and VideoInferenceJob into one VideoJob class

Also input some logging changes

* Improved logging for job manager, and API triggering of jobs

* Simplified Job creation and configuration

Also documentation in the README.
This commit simplifies the codebase and the amount of variables to configure for job creation

* Small tweaks to improve how the server runs

* Fixes to client logging

* Optimizations to the config of settings

* torch_dtype is deprecated so using dtype

* Fix circular imports for configurations in server

* Improved results from inference in client

* Dataset prep files

* Fixes to let client and server run together better

* More fixes hopefully for the inference results to display

* Even more fixes, this is tiring

Author: animarcus

.gitignore

@@ -1,9 +1,11 @@
-data/**/*.mp4
-data/**/*.json
-data/**/*.txt
-data/**/*.csv
+data/**/*
 models/**/*.pth
 !**/.gitkeep
+uv.lock
+
+benchmark*.json
+
+.claude/
 
 # Created by https://www.toptal.com/developers/gitignore/api/vim,latex,linux,macos,synology,jetbrains+all,visualstudiocode,python,jupyternotebooks
 # Edit at https://www.toptal.com/developers/gitignore?templates=vim,latex,linux,macos,synology,jetbrains+all,visualstudiocode,python,jupyternotebooks

.vscode/settings.json

@@ -31,17 +31,20 @@
         "reportUndefinedVariable": "error",
         "reportMissingImports": "warning"
     },
+    // Disable old linting system
+    "python.linting.enabled": false,
+    "python.linting.pylintEnabled": false,
     // Python Formatting with Ruff
     "[python]": {
         "editor.formatOnSave": true,
         "editor.defaultFormatter": "charliermarsh.ruff",
         "editor.codeActionsOnSave": {
-            "source.fixAll": "explicit",
-            "source.organizeImports": "explicit"
+            "source.fixAll.ruff": "explicit",
+            "source.organizeImports.ruff": "explicit"
         }
     },
     // Ruff Settings
     "ruff.nativeServer": "on",
     // Other Extensions
-    "evenBetterToml.schema.enabled": false,
+    "evenBetterToml.schema.enabled": false
 }

README.md

@@ -3,3 +3,249 @@
 A research project done with the AI Team by Myriam Benlamri (Data Science MSc 2nd year) and Marcus Hamelink (Computer Science BSc 3rd year) as a collaborative research semester project.
 
 More info at [https://epflaiteam.ch/projects/iris](https://epflaiteam.ch/projects/iris)
+
+
+## Set up
+
+### Client
+
+On Your Raspberry Pi, generate the self-signed certificate:
+```bash
+mkdir -p ~/iris-certs
+cd ~/iris-certs
+openssl req -x509 -newkey rsa:4096 -nodes \
+  -keyout key.pem \
+  -out cert.pem \
+  -days 365 \
+  -subj "/C=US/ST=State/L=City/O=Organization/CN=$(hostname -I | awk '{print $1}')"
+```
+
+To use HTTPS, make sure to specify `use_ssl=true` under `client` in `config.yaml`. In that case, make sure to connect to the HTTPS IP.
+
+Run `uv run iris-client` to start a client instance.
+
+
+### Server
+
+```bash
+uv sync
+uv sync --group server
+
+```
+
+#### For training (unsloth)
+
+```bash
+uv pip install unsloth
+```
+
+#### Running the pipeline
+
+Run `uv run iris-server` to start a server instance.
+
+
+## Job System
+
+IRIS uses a flexible job system for managing inference tasks. Jobs can be started via API and triggered in multiple ways.
+
+### Job Types
+
+**1. SingleFrameJob**
+- Processes each incoming frame individually with VLM
+- Useful for: real-time inference, testing, continuous monitoring
+- Trigger: Automatic on every frame (or every Nth frame with `frame_skip`)
+
+**2. VideoJob**
+- Collects frames in buffer, then processes batch with video-aware VLM
+- Useful for: temporal understanding, action recognition, video summarization
+- Supports three trigger modes: periodic (automatic), manual (API), and disabled (job-to-job)
+
+### Trigger Modes
+
+VideoJob supports three triggering modes via the `trigger_mode` parameter:
+
+**PERIODIC (Automatic)**
+- Automatically triggers inference when buffer reaches `buffer_size` frames
+- After inference, keeps last `overlap_frames` for temporal continuity
+- Configuration example:
+```python
+{
+    "job_type": "video",
+    "trigger_mode": "periodic",
+    "buffer_size": 8,
+    "overlap_frames": 4
+}
+```
+**Use case:** Continuous video analysis (e.g., Qwen2.5-VL logging)
+
+**MANUAL (API-triggered)**
+- Buffers frames but only triggers via API call: `POST /jobs/{job_id}/trigger`
+- No overlap - buffer clears after each trigger
+- Configuration example:
+```python
+{
+    "job_type": "video",
+    "trigger_mode": "manual",
+    "buffer_size": 1
+}
+```
+**Use case:** On-demand analysis (e.g., colony counter when user clicks)
+
+**DISABLED (Buffering Only)**
+- Accepts and buffers frames but never processes them
+- For future use or conditional triggering
+- Configuration example:
+```python
+{
+    "job_type": "video",
+    "trigger_mode": "disabled",
+    "buffer_size": 8
+}
+```
+**Use case:** Placeholder for future YOLO integration
+
+### Auto-Started VideoJob
+
+When a client connects to `/ws/stream`, a VideoJob is automatically created for that connection:
+- Job ID: Unique per connection (e.g., `video_job_a3f7b2c1`)
+- Mode: PERIODIC
+- Buffer: 8 frames (configurable in `config.yaml`)
+- Overlap: 4 frames - 50% overlap for temporal continuity
+- Cleanup: Automatically stopped and removed when WebSocket disconnects
+
+**No manual job creation needed - just start streaming!**
+
+You can configure defaults in `config.yaml`:
+```yaml
+jobs:
+  video:
+    trigger_mode: "periodic"
+    buffer_size: 8
+    overlap_frames: 4
+```
+
+### API Endpoints
+
+**Start a job:**
+```bash
+POST /jobs/start
+Content-Type: application/json
+
+{
+    "job_type": "video",
+    "prompt": "Describe what you see in the video.",
+    "trigger_mode": "periodic",
+    "buffer_size": 8,
+    "overlap_frames": 4
+}
+```
+
+**Manually trigger inference:**
+```bash
+POST /jobs/{job_id}/trigger
+```
+
+**Get job status:**
+```bash
+GET /jobs/{job_id}/status
+```
+
+**List active jobs:**
+```bash
+GET /jobs/active
+```
+
+**Stop a job:**
+```bash
+POST /jobs/{job_id}/stop
+```
+
+### WebSocket Logging
+
+Jobs send progress logs via WebSocket (`/ws/stream`):
+
+```json
+{
+    "type": "log",
+    "job_id": "video-abc123",
+    "message": "Buffered frame 3/5",
+    "timestamp": 1234567890.123
+}
+```
+
+Results are also sent via WebSocket:
+
+```json
+{
+    "type": "result",
+    "job_id": "video-abc123",
+    "job_type": "video",
+    "status": "completed",
+    "result": "..."
+}
+```
+
+### Job Orchestration
+
+Jobs can launch other jobs during execution, enabling conditional workflows:
+
+```python
+class YOLOVideoJob(VideoJob):
+    async def _run_inference(self):
+        # Run YOLO detection
+        detections = await self._run_yolo(self.frame_buffer)
+
+        # If object detected, launch VLM job
+        if detections["confidence"] > 0.5:
+            vlm_config = VideoJobConfig(
+                prompt="Describe what the detected object is doing.",
+                trigger=TriggerConfig(mode=TriggerMode.DISABLED)
+            )
+            vlm_job = self.job_factory.create_job(vlm_config, ...)
+            await self.queue.submit(vlm_job)
+```
+
+### Multi-GPU Support
+
+Set `server.num_workers` in `config.yaml` to utilize multiple GPUs:
+
+```yaml
+server:
+  num_workers: 2  # Uses 2 GPUs in round-robin
+```
+
+Workers are automatically assigned to GPUs: `worker_id % device_count`.
+
+### Video Inference Notes
+
+**TODO:** The current VideoJob implementation processes only the first frame as a placeholder. Proper video inference requires exploring Qwen2.5-VL's video prompt template, which may support native video input with special tokens for temporal understanding.
+
+See `src/iris/vlm/inference/queue/jobs.py:VideoJob._sync_inference()` for implementation details.
+
+## Workflow with Izar
+
+This supposee
+
+### On Izar
+```
+cd /path/to/IRIS
+Sinteract -t 00:20:00 -g gpu:1 -m 32G -q team-ai
+hostname
+./run_iris.sh
+```
+
+### On Personal machine
+
+**Terminal 1**
+```
+uv run iris-client
+```
+
+**Terminal 2**
+```
+ssh -N -L 8005:[RUN hostname ON NODE TO SEE]:8001 EPFL-USERNAME@izar.hpc.epfl.ch
+```
+
+Then go to http://localhost:8006
+
+Important, make sure to modify the hostname

config.yaml

@@ -0,0 +1,43 @@
+# IRIS Configuration
+# Override via environment variables: IRIS_SERVER__PORT=8002
+
+server:
+  model_id: "unsloth/Qwen2-VL-7B-Instruct-bnb-4bit"  # Direct model selection (qwen2.5-7b)
+  vlm_hardware: null      # Optional: Hardware profile (v100, mac, etc.)
+  max_queue_size: 10
+  num_workers: 1
+  host: "0.0.0.0"
+  port: 8001
+  graceful_shutdown_timeout: 30.0
+  enable_log_streaming: true
+  log_streaming_min_level: "INFO"
+  enable_metrics: true
+
+jobs:
+  video:
+    trigger_mode: "periodic"  # "periodic" | "manual" | "disabled"
+    buffer_size: 8
+    overlap_frames: 4
+
+client:
+  video:
+    width: 640
+    height: 480
+    fps: 10
+    jpeg_quality: 80
+    camera_index: 0
+  server:
+    host: "localhost"
+    port: 8001
+    use_ssl: false
+  web:
+    host: "0.0.0.0"
+    port: 8006
+    use_ssl: true
+    cert_dir: "~/iris-certs"  # Directory containing key.pem and cert.pem
+  ssh_tunnel:
+    enabled: false  # Toggle on for IZAR HPC
+    ssh_host: "izar.hpc.epfl.ch"
+    ssh_user: "mhamelin"
+    ssh_key_path: "~/.ssh/id_rsa"
+    remote_host: ""  # Set via UI or config (IZAR compute node hostname)

configs/vlm/hardware/mac.yaml

@@ -0,0 +1,13 @@
+# Apple Silicon (M1/M2/M3) optimization
+# Uses Metal Performance Shaders (MPS) for GPU acceleration
+
+model:
+  dtype: "float16"  # MPS works well with float16
+  attn_implementation: "sdpa"
+  low_cpu_mem_usage: true
+
+quantization:
+  load_in_8bit: false
+  load_in_4bit: false
+
+device: "mps"  # Metal Performance Shaders for Apple Silicon

configs/vlm/hardware/v100.yaml

@@ -1,25 +1,13 @@
-training:
-#   batch_size: 8
-#   gradient_accumulation_steps: 4  # Effective batch = 32
+# V100 GPU optimization (16GB VRAM)
+# For inference use on IZAR cluster
 
-# model:
-#   torch_dtype: "float16"
+model:
+  dtype: "float16"  # V100 doesn't support bfloat16
+  attn_implementation: "sdpa"  # V100 doesn't support flash_attention_2
+  low_cpu_mem_usage: true
 
-# accelerate:
-#   use_accelerate: true
-#   mixed_precision: "fp16"  # V100 supports fp16, NOT bf16
-#   gradient_checkpointing: true
+quantization:
+  load_in_8bit: false  # V100 has enough VRAM for float16
+  load_in_4bit: false
 
-# peft:
-#   use_peft: true
-#   peft_method: "lora"
-#   r: 8
-#   alpha: 16
-#   dropout: 0.1
-
-# quantization:
-#   load_in_4bit: true
-#   bnb_4bit_quant_type: "nf4"
-#   bnb_4bit_compute_dtype: "float16"
-
-# device: "cuda"
+device: "auto"