Merge branch 'main' into add/startup-wizard

Author: aliasaria

.agents/skills/transformerlab-cli/SKILL.md

@@ -87,7 +87,9 @@ lab task list
 # 3. Queue a task on a compute provider
 #    NOTE: --no-interactive silently picks the DEFAULT provider (Local).
 #    To pick a specific provider, run interactively (see "Selecting a provider" below).
-lab task queue TASK_ID --no-interactive
+#    ALWAYS pass --description/-m with a markdown note describing the iteration
+#    (see "Always write a run description" below).
+lab task queue TASK_ID --no-interactive -m "Testing lr=3e-5 after loss plateaued at 2.1"
 
 # 4. Monitor the job (three log streams — see "Job logs: three real commands" below)
 lab job list --running
@@ -226,6 +228,37 @@ lab task add ./hello-world-task --no-interactive
 10. **Never use the deprecated `lab job logs`** — see the "Job logs: three real commands" section below.
 11. **After queuing a task, ASK the user if they'd like you to watch the logs.** Don't start streaming or polling automatically — jobs can take minutes to hours, and `--follow` blocks. Report the Job ID and ask: "Want me to watch the logs and report back?"
 12. **Never create API keys programmatically** — if auth fails, ask the user to provide an API key from the web UI
+13. **Always pass `--description/-m` when queuing a task. Generate it yourself — never ask the user.** See "Always write a run description" below.
+
+### Always write a run description
+
+Every `lab task queue` call MUST include `--description/-m "..."`. The description is markdown stored on the job and shown in `lab job info`. Its audience is a future researcher reading `lab job list` weeks later — they have git and the task code, but NOT this chat. The description is the only bridge.
+
+**Treat it like a short PR description for this run.** Draft 1–5 lines (bullets for multi-point notes) covering:
+
+1. **What changed vs the prior run / baseline** — the concrete diff (hyperparameters, code, model, data, infra). If nothing changed, say so and link the prior job.
+2. **What hypothesis you're testing** — why this run is worth doing.
+3. **What a future reader should remember** — gotchas, prior surprises, things to check in the output.
+
+Pull these from the conversation and recent git diff / edited files. If the note has newlines or shell-awkward characters, pipe it: `printf '%s' "$DESC" | lab task queue abc123 -m -`.
+
+```bash
+# Good: diff + hypothesis + watch
+printf '%s' "- Bumped lr 1e-5 → 3e-5, warmup 100 → 500 steps.
+- Testing whether higher lr clears the eval/loss=2.1 plateau seen in job 7f21 around step 2k.
+- Watch: earlier runs with lr≥5e-5 diverged by step 500." | lab task queue abc123 --no-interactive -m -
+
+# Good: small change — one line is enough
+lab task queue abc123 --no-interactive -m "Rerun on H100 (was A100) to confirm throughput regression from #1850."
+
+# Good: nothing changed
+lab task queue abc123 --no-interactive -m "Rerun of job 7f21, no code or config changes (network flake on first attempt)."
+
+# Bad: generic filler that tells the reader nothing
+lab task queue abc123 -m "train model"
+```
+
+Don't restate the task name, full hyperparameter dict, or file paths — those are already on the job record. Don't copy the user's last message verbatim — synthesize. If the conversation is truly empty of signal, fall back to `"Rerun of <id>, no changes"`.
 
 ### Selecting a provider when queuing a task
 
@@ -308,7 +341,7 @@ This applies to launching jobs, fetching logs, checking cluster status, and ever
 | `lab task init` | Scaffold `task.yaml` + `main.py` in the current directory (`--interactive` to prompt) | No |
 | `lab task add [dir]` | Add task from directory or `--from-git` URL (`--no-interactive`, `--dry-run`) | Yes |
 | `lab task delete <id>` | Delete a task (`--no-interactive` to skip confirmation) | Yes |
-| `lab task queue <id>` | Queue task on compute provider | Yes |
+| `lab task queue <id>` | Queue task on compute provider (`-m/--description` for a markdown run note; required for agents, see "Always write a run description") | Yes |
 | `lab task gallery` | Browse/import from task gallery | Yes |
 | `lab job list` | List jobs (`--running` for active only) | Yes |
 | `lab job info <id>` | Get detailed job information | Yes |

.agents/skills/transformerlab-cli/references/commands.md

@@ -131,6 +131,7 @@ Queue a task on a compute provider.
 | Option | Description |
 |---|---|
 | `--no-interactive` | Skip prompts. Uses the task's configured provider or first available. Parameters use defaults. **Always use this in automated workflows.** |
+| `-m`, `--description <text>` | Markdown note describing what this run is trying to accomplish (stored on the job, shown in `lab job info`). Pass `-` to read from stdin. **Agents: required per SKILL.md rule 13.** |
 
 **JSON output:** Returns the created job object with `id` and `status`.
 

.gitignore

@@ -227,3 +227,4 @@ test-results/
 **/.secrets
 
 docs/superpowers/*
+.superpowers/

api/api.py

@@ -55,7 +55,6 @@
 
 
 from transformerlab.routers.experiment import experiment  # noqa: E402
-from transformerlab.routers.experiment import jobs  # noqa: E402
 from transformerlab.shared import shared  # noqa: E402
 from transformerlab.shared import dirs  # noqa: E402
 from lab.dirs import set_organization_id as lab_set_org_id  # noqa: E402
@@ -306,7 +305,6 @@ async def validation_exception_handler(request, exc):
 app.include_router(serverinfo.router, dependencies=[Depends(get_user_and_team)])
 app.include_router(data.router, dependencies=[Depends(get_user_and_team)])
 app.include_router(experiment.router, dependencies=[Depends(get_user_and_team)])
-app.include_router(jobs.router, dependencies=[Depends(get_user_and_team)])
 app.include_router(config.router, dependencies=[Depends(get_user_and_team)])
 app.include_router(teams.router, dependencies=[Depends(get_user_and_team)])
 app.include_router(compute_provider.router)

api/localprovider_pyproject.toml

@@ -32,7 +32,7 @@ dependencies = [
   "soundfile==0.13.1",
   "tensorboardX==2.6.2.2",
   "timm==1.0.15",
-  "transformerlab==0.1.22",
+  "transformerlab==0.1.24",
   "transformerlab-inference==0.2.52",
   "transformers==4.57.1",
   "wandb==0.23.1",

api/pyproject.toml

@@ -24,7 +24,7 @@ dependencies = [
   "python-dotenv==1.1.0",
   "python-multipart==0.0.20",
   "sqlalchemy[asyncio]==2.0.40",
-  "transformerlab==0.1.23",
+  "transformerlab==0.1.24",
   "transformerlab-inference==0.2.52",
   "uvicorn==0.35.0",
   "watchfiles==1.0.5",

api/test/api/test_storage_probe_service.py

@@ -0,0 +1,71 @@
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+
+@pytest.mark.asyncio
+async def test_check_storage_probe_found():
+    """check_storage_probe returns found=True when sentinel file exists."""
+    with patch("transformerlab.services.compute_provider.storage_probe_service.set_organization_id") as mock_set_org:
+        with patch(
+            "transformerlab.services.compute_provider.storage_probe_service.get_workspace_dir",
+            new=AsyncMock(return_value="/mnt/nfs/orgs/team1/workspace"),
+        ):
+            with patch("transformerlab.services.compute_provider.storage_probe_service.storage") as mock_storage:
+                mock_storage.join = lambda *parts: "/".join(parts)
+                mock_storage.exists = AsyncMock(return_value=True)
+
+                from transformerlab.services.compute_provider.storage_probe_service import check_storage_probe
+
+                result = await check_storage_probe(job_id="job-123", team_id="team1")
+
+    assert result["found"] is True
+    assert "storage-probe-job-123.txt" in result["path"]
+    mock_set_org.assert_any_call("team1")
+    mock_set_org.assert_any_call(None)
+
+
+@pytest.mark.asyncio
+async def test_check_storage_probe_not_found():
+    """check_storage_probe returns found=False when sentinel file is missing."""
+    with patch("transformerlab.services.compute_provider.storage_probe_service.set_organization_id"):
+        with patch(
+            "transformerlab.services.compute_provider.storage_probe_service.get_workspace_dir",
+            new=AsyncMock(return_value="/mnt/nfs/orgs/team1/workspace"),
+        ):
+            with patch("transformerlab.services.compute_provider.storage_probe_service.storage") as mock_storage:
+                mock_storage.join = lambda *parts: "/".join(parts)
+                mock_storage.exists = AsyncMock(return_value=False)
+
+                from transformerlab.services.compute_provider.storage_probe_service import check_storage_probe
+
+                result = await check_storage_probe(job_id="job-999", team_id="team1")
+
+    assert result["found"] is False
+
+
+@pytest.mark.asyncio
+async def test_launch_storage_probe_returns_job_id():
+    """launch_storage_probe calls launch_template_on_provider and returns job_id."""
+    mock_launch = AsyncMock(return_value={"job_id": 42, "status": "success"})
+
+    with patch(
+        "transformerlab.services.compute_provider.storage_probe_service.launch_template_on_provider",
+        new=mock_launch,
+    ):
+        from transformerlab.services.compute_provider.storage_probe_service import launch_storage_probe
+
+        user_and_team = {"team_id": "team1", "user": MagicMock(id="user1", first_name="", last_name="", email="")}
+        result = await launch_storage_probe(
+            provider_id="prov-1",
+            user_and_team=user_and_team,
+            session=MagicMock(),
+        )
+
+    assert result["job_id"] == 42
+    assert result["experiment_id"] == "__storage_probe__"
+    call_args = mock_launch.call_args
+    request = call_args.kwargs["request"]
+    assert request.run == "python -m lab.probe"
+    assert request.experiment_id == "__storage_probe__"
+    assert request.cpus == "1"
+    assert request.accelerators is None

api/transformerlab/galleries/channels/stable/latest/interactive-gallery.json

@@ -4,8 +4,8 @@
     "interactive_type": "vscode",
     "name": "VS Code",
     "description": "Remote VS Code development environment via tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-vscode",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-vscode",
     "env_parameters": [],
     "icon": "https://lab.cloud/img/icons/vscode.png",
     "supported_accelerators": [
@@ -46,8 +46,8 @@
     "interactive_type": "jupyter",
     "name": "Jupyter Notebook",
     "description": "Jupyter Lab notebook environment via ngrok tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-jupyter",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-jupyter",
     "env_parameters": [
       {
         "field_name": "ngrok Auth Token",
@@ -98,8 +98,8 @@
     "interactive_type": "vllm",
     "name": "vLLM Server",
     "description": "vLLM OpenAI-compatible API server via ngrok tunnel (plus Open WebUI)",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-vllm",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-vllm",
     "env_parameters": [
       {
         "field_name": "Model Name",
@@ -191,8 +191,8 @@
     "interactive_type": "ollama",
     "name": "Ollama + Open WebUI",
     "description": "Ollama model server with Open WebUI chat interface via ngrok tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-ollama-openwebui",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-ollama-openwebui",
     "env_parameters": [
       {
         "field_name": "Model Name",
@@ -268,8 +268,8 @@
     "id": "ollama_gradio",
     "name": "Ollama + Gradio",
     "description": "Ollama model server with Gradio chat UI via ngrok tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "inference-ollama-gradio-chat",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/inference-ollama-gradio-chat",
     "env_parameters": [
       {
         "field_name": "Model Name",
@@ -323,8 +323,8 @@
     "id": "comfy_ui",
     "name": "ComfyUI",
     "description": "ComfyUI server via ngrok tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-comfyui",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-comfyui",
     "env_parameters": [
       {
         "field_name": "ngrok Auth Token",
@@ -374,8 +374,8 @@
     "interactive_type": "ssh",
     "name": "SSH",
     "description": "SSH access via ngrok TCP tunnel",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "interactive-ssh",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/interactive-ssh",
     "env_parameters": [
       {
         "field_name": "ngrok Auth Token",
@@ -442,8 +442,8 @@
     "id": "mlx_gradio",
     "name": "MLX LM + Gradio",
     "description": "MLX LM inference server with Gradio chat UI for Apple Silicon",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "inference-mlx-gradio-chat",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/inference-mlx-gradio-chat",
     "env_parameters": [
       {
         "field_name": "Model Name",
@@ -494,8 +494,8 @@
     "id": "mlx_audio_tts",
     "name": "MLX Audio TTS + Gradio",
     "description": "Text-to-Speech using MLX Audio with Gradio UI for Apple Silicon",
-    "github_repo_url": "https://github.com/transformerlab/transformerlab-examples",
-    "github_repo_dir": "inference-mlx-audio-tts",
+    "github_repo_url": "https://github.com/transformerlab/transformerlab-app",
+    "github_repo_dir": "api/transformerlab/galleries/examples/inference-mlx-audio-tts",
     "env_parameters": [
       {
         "field_name": "Model Name",

api/transformerlab/galleries/channels/stable/latest/manifest.json

@@ -1,11 +1,10 @@
 {
-  "bundle_version": "20260415184435",
+  "bundle_version": "20260421193056",
   "channel": "stable",
-  "released_at": "2026-04-15T18:44:35.364579+00:00",
+  "released_at": "2026-04-21T19:30:56.442725+00:00",
   "files": {
     "task-gallery.json": 25,
     "interactive-gallery.json": 9,
     "announcement-gallery.json": 2
-  },
-  "min_supported_app_version": "0.0.0"
+  }
 }