Merge branch 'main' into fix/edit-task-modal-load

Author: deep1401

.agents/skills/transformerlab-cli/references/autoresearch.md

@@ -219,6 +219,11 @@ The agent loops autonomously. Never ask "should I continue?" — keep going unti
      --param lr=3e-5 --param warmup_steps=500 \
      -m "Bumped lr 1e-5→3e-5, warmup 100→500. Testing whether higher lr clears the eval/loss=2.1 plateau seen in baseline."
    ```
+   Immediately after queueing, add a structured log line to experiment notes so intent survives even if the session resets before `finalize`:
+   ```bash
+   lab notes append "- $(date +%F): queued <JOB_ID> | intent: <what changed> | hypothesis: <expected metric behavior> | reason: <why this is the next best idea> | expected-risk: <what might break>"
+   ```
+   Get `<JOB_ID>` from the queue command output or from `lab --format json job list | jq '.[-1].id'` if needed.
 6. **Advance.** What this means depends on parallelism:
    - **N = 1 (sequential):** wait for the just-queued job to finish, then go to step 7. Poll with `lab --format json job list --running`.
    - **N ≥ 2 (parallel / fire-and-advance):** **do not wait.** Return to step 0 and pick the next idea immediately, until `RUNNING >= N`. Only block when the queue is full. This is what enables grid-style search; waiting after every queue collapses parallelism back to 1.
@@ -234,6 +239,10 @@ The agent loops autonomously. Never ask "should I continue?" — keep going unti
    # If COMPLETE but primary metric did NOT improve over current best → lab job discard <JOB_ID>.
    # If COMPLETE and improved → leave kept. Jobs are kept by default.
    ```
+   After deciding keep/discard, append a one-line outcome note for each processed job:
+   ```bash
+   lab notes append "- $(date +%F): result <JOB_ID> | <kept|discarded|failed> | <primary_metric>=<value> | delta-vs-best: <+/-> | takeaway: <what to repeat/avoid next> | next: <follow-up idea or 'none'>"
+   ```
 8. **Loop.**
 
 The agent stops the loop only when:
@@ -351,9 +360,9 @@ Do **not** delete the experiment. Do **not** delete jobs. The session record is
 
 ---
 
-## Run descriptions are the durable record
+## Run descriptions + notes are the durable record
 
-Because Transformer Lab persists per-job descriptions, **`-m/--description` is the per-run log** — there is no separate `autoresearch.jsonl` to maintain. Treat each description as a mini commit message for that iteration:
+Because Transformer Lab persists per-job descriptions, **`-m/--description` is the per-run log** — and experiment notes are the cross-run narrative. There is no separate `autoresearch.jsonl` to maintain. Treat each description as a mini commit message for that iteration:
 
 1. **What changed vs the prior best run** (params, code, infra). If nothing changed, say so.
 2. **What hypothesis is being tested** (why this run is worth doing).
@@ -368,14 +377,41 @@ printf '%s' "- Switched optimizer AdamW→Lion (β1=0.95, β2=0.98).
 
 Bad descriptions (`"train model"`, `"another run"`) defeat the entire point — the agent two iterations from now has no idea what was tried.
 
+Also mirror each queue/result in `lab notes append` during the loop (not only at finalize). Descriptions are attached to one job; notes capture session-level reasoning and make "why this order of experiments?" visible when reading the full session timeline.
+
+### What to capture in `lab notes` beyond the job description
+
+Use descriptions for per-job details and notes for cross-job reasoning. A good rule: if the information helps choose the *next* idea, it belongs in notes even when it is also present in `-m`.
+
+- **Decision context:** Why this run was chosen over 1–2 alternatives from Backlog.
+- **Risk + guardrails:** What failure mode is expected and which signal confirms it.
+- **Comparative result:** Delta vs baseline and delta vs current best (not just raw metric).
+- **Actionability:** Explicit next step (`repeat`, `widen`, `revert`, `pivot`) and why.
+- **Session-level blockers:** Provider instability, flaky evals, or data issues affecting multiple runs.
+
+Compact append examples:
+
+```bash
+lab notes append "- $(date +%F): decision | picked JOB <JOB_ID> idea (lr+warmup) over scheduler swap because recent runs suggest under-training, not optimizer instability."
+lab notes append "- $(date +%F): result <JOB_ID> | kept | eval/loss=1.84 | delta-vs-baseline=-0.26 | delta-vs-best=-0.07 | next: widen around lr=3e-5."
+lab notes append "- $(date +%F): blocker | provider cold-start >20m on 2/3 attempts; avoid wide parallel bursts until stable."
+```
+
+When notes get long, keep append-only in the hot loop, then periodically consolidate into:
+
+1. **Key wins** (what consistently improves the primary metric)
+2. **Dead ends** (what reliably regresses and should not be retried)
+3. **Open hypotheses** (highest-value next experiments)
+4. **Operational caveats** (infra/eval quirks that affect interpretation)
+
 ---
 
 ## Loop discipline
 
 - **Never stop the loop unless the user interrupts.** The user expects autonomous work.
 - **One iteration per running job slot.** Don't queue past the parallelism budget.
 - **Annotate failures heavily** via `-m`. Discarded ≠ deleted, and the description survives.
-- **Keep the experiment notes honest.** Update **What's been tried** every ~5 runs via `lab notes append`. Re-read with `lab notes show --raw` at the start of every iteration — never skip the rehydrate.
+- **Keep the experiment notes honest.** For every queued job, append intent/hypothesis/reason; for every processed completion, append outcome/takeaway. Continue periodic consolidation of **What's been tried** every ~5 runs. Re-read with `lab notes show --raw` at the start of every iteration — never skip the rehydrate.
 - **Respect provider capacity.** Local providers serialize anyway; SkyPilot/RunPod cost real money — never raise parallelism without explicit user approval.
 - **No `curl` workarounds.** If something seems missing from the CLI, run `lab <cmd> --help`, re-read this file, and tell the user — don't fall back to the REST API. (See the parent skill's "Do NOT call the REST API as a CLI workaround" section.)
 

api/alembic/versions/46378c10f132_add_user_experiment_access_table.py

@@ -0,0 +1,55 @@
+"""add_user_experiment_access_table
+
+Revision ID: 46378c10f132
+Revises: 6ccd4a4d9ca1
+Create Date: 2026-05-04 13:23:27.122716
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from transformerlab.db.migration_utils import table_exists
+
+
+# revision identifiers, used by Alembic.
+revision: str = "46378c10f132"
+down_revision: Union[str, Sequence[str], None] = "6ccd4a4d9ca1"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    connection = op.get_bind()
+    if not table_exists(connection, "user_experiment_access"):
+        op.create_table(
+            "user_experiment_access",
+            sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
+            sa.Column("user_id", sa.String(), nullable=False),
+            sa.Column("team_id", sa.String(), nullable=False),
+            sa.Column("experiment_id", sa.String(), nullable=False),
+            sa.Column(
+                "last_opened_at",
+                sa.DateTime(),
+                server_default=sa.text("(CURRENT_TIMESTAMP)"),
+                nullable=False,
+            ),
+            sa.PrimaryKeyConstraint("id"),
+            sa.UniqueConstraint(
+                "user_id",
+                "team_id",
+                "experiment_id",
+                name="uq_user_experiment_access",
+            ),
+        )
+        op.create_index(
+            "idx_user_experiment_access_user_team",
+            "user_experiment_access",
+            ["user_id", "team_id"],
+        )
+
+
+def downgrade() -> None:
+    op.drop_index("idx_user_experiment_access_user_team", table_name="user_experiment_access", if_exists=True)
+    op.drop_table("user_experiment_access", if_exists=True)

api/test/api/test_experiment_service.py

@@ -46,6 +46,16 @@ async def test_missing_experiment_returns_none(tmp_experiments_dir):
     assert await experiment_service.experiment_get("no_such_experiment") is None
 
 
+@pytest.mark.asyncio
+async def test_duplicate_experiment_create_raises_file_exists(tmp_experiments_dir):
+    _ = tmp_experiments_dir
+    name = f"duplicate_exp_{uuid.uuid4().hex[:8]}"
+    await experiment_service.experiment_create(name, {"a": 1})
+
+    with pytest.raises(FileExistsError):
+        await experiment_service.experiment_create(name, {"a": 2})
+
+
 # Added test to hit the new FileNotFoundError except-clauses in experiment_service
 @pytest.mark.asyncio
 async def test_missing_experiment_operations_handle_FileNotFound(tmp_experiments_dir):

api/test/api/test_task_provider_resolution.py

@@ -0,0 +1,49 @@
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from fastapi import HTTPException
+
+from transformerlab.routers.experiment import task as task_router
+
+
+@pytest.mark.asyncio
+async def test_resolve_provider_rejects_unknown_provider_name(monkeypatch):
+    providers = [
+        SimpleNamespace(id="prov-default", name="AWS Default", is_default=True),
+        SimpleNamespace(id="prov-skypilot", name="SkypilotNew", is_default=False),
+    ]
+    monkeypatch.setattr(task_router, "list_team_providers", AsyncMock(return_value=providers))
+
+    task_data = {"provider_name": "Skypilot New"}
+
+    with pytest.raises(HTTPException) as exc:
+        await task_router._resolve_provider(
+            task_data=task_data,
+            user_and_team={"team_id": "team-1"},
+            session=MagicMock(),
+        )
+
+    assert exc.value.status_code == 400
+    assert "Unknown compute provider 'Skypilot New'" in str(exc.value.detail)
+    assert "SkypilotNew" in str(exc.value.detail)
+    assert "AWS Default" in str(exc.value.detail)
+
+
+@pytest.mark.asyncio
+async def test_resolve_provider_uses_default_when_provider_name_missing(monkeypatch):
+    providers = [
+        SimpleNamespace(id="prov-default", name="AWS Default", is_default=True),
+        SimpleNamespace(id="prov-other", name="Other Provider", is_default=False),
+    ]
+    monkeypatch.setattr(task_router, "list_team_providers", AsyncMock(return_value=providers))
+
+    task_data: dict = {}
+    await task_router._resolve_provider(
+        task_data=task_data,
+        user_and_team={"team_id": "team-1"},
+        session=MagicMock(),
+    )
+
+    assert task_data["provider_id"] == "prov-default"
+    assert task_data["provider_name"] == "AWS Default"

api/test/test_experiment_access_service.py

@@ -0,0 +1,58 @@
+from unittest.mock import AsyncMock, MagicMock
+
+from sqlalchemy.exc import IntegrityError
+
+import transformerlab.services.experiment_access_service as svc
+
+
+async def test_touch_experiment_upserts_record():
+    mock_session = AsyncMock()
+    mock_session.add = MagicMock()
+    mock_result = MagicMock(rowcount=0)
+    mock_session.execute.return_value = mock_result
+
+    await svc.touch_experiment(mock_session, "user1", "team1", "exp1")
+
+    mock_session.add.assert_called_once()
+    mock_session.commit.assert_called_once()
+
+
+async def test_touch_experiment_updates_existing_record():
+    mock_session = AsyncMock()
+    mock_session.add = MagicMock()
+    mock_result = MagicMock(rowcount=1)
+    mock_session.execute.return_value = mock_result
+
+    await svc.touch_experiment(mock_session, "user1", "team1", "exp1")
+
+    mock_session.add.assert_not_called()
+    mock_session.commit.assert_called_once()
+
+
+async def test_touch_experiment_handles_insert_race_integrity_error():
+    mock_session = AsyncMock()
+    mock_session.add = MagicMock()
+    mock_result = MagicMock(rowcount=0)
+    mock_session.execute.return_value = mock_result
+    mock_session.commit.side_effect = [
+        IntegrityError("stmt", "params", Exception("duplicate key")),
+    ]
+
+    await svc.touch_experiment(mock_session, "user1", "team1", "exp1")
+
+    mock_session.rollback.assert_called_once()
+
+
+async def test_get_recent_experiment_ids_returns_ordered_list():
+    mock_session = AsyncMock()
+    record1 = MagicMock()
+    record1.experiment_id = "exp_b"
+    record2 = MagicMock()
+    record2.experiment_id = "exp_a"
+    mock_result = MagicMock()
+    mock_result.scalars.return_value.all.return_value = [record1, record2]
+    mock_session.execute.return_value = mock_result
+
+    result = await svc.get_recent_experiment_ids(mock_session, "user1", "team1", limit=3)
+
+    assert result == ["exp_b", "exp_a"]

api/test/test_notes_helpers.py

@@ -148,3 +148,30 @@ async def _run():
         mock_storage.rm.assert_not_called()
 
     asyncio.run(_run())
+
+
+def test_resolve_unique_asset_filename_returns_original_when_available():
+    async def _run():
+        from transformerlab.routers.experiment.notes import resolve_unique_asset_filename
+
+        with patch("transformerlab.routers.experiment.notes.storage") as mock_storage:
+            mock_storage.join.side_effect = lambda *parts: "/".join(parts)
+            mock_storage.exists = AsyncMock(return_value=False)
+            result = await resolve_unique_asset_filename("/experiments/exp1/notes/assets", "image.png")
+        assert result == "image.png"
+
+    asyncio.run(_run())
+
+
+def test_resolve_unique_asset_filename_adds_incrementing_suffix_on_collision():
+    async def _run():
+        from transformerlab.routers.experiment.notes import resolve_unique_asset_filename
+
+        # image.png exists, image-1.png exists, image-2.png is available.
+        with patch("transformerlab.routers.experiment.notes.storage") as mock_storage:
+            mock_storage.join.side_effect = lambda *parts: "/".join(parts)
+            mock_storage.exists = AsyncMock(side_effect=[True, True, False])
+            result = await resolve_unique_asset_filename("/experiments/exp1/notes/assets", "image.png")
+        assert result == "image-2.png"
+
+    asyncio.run(_run())