Merge branch 'main' into add/juicefs-mounting

Author: deep1401

.agents/skills/transformerlab-cli/SKILL.md

@@ -86,6 +86,40 @@ lab --format json task info 42
 lab job list --format json
 ```
 
+## Profiles: talking to multiple servers in parallel
+
+A **profile** bundles one server's URL, team, experiment, and API key. The `default` profile lives at the legacy location (`~/.lab/config.json`, `~/.lab/credentials`); named profiles live under `~/.lab/profiles/<name>/`. This lets two `lab` commands hit two different servers (with different teams / API keys) **at the same time** without clobbering each other.
+
+Selection is **per-process** — there is no stored "current profile" to switch (so nothing one process can yank out from under another). Precedence (highest first):
+
+1. `--profile <name>` — a root-level flag; like `--format` it MUST come before the subcommand: `lab --profile prod job list`
+2. `LAB_PROFILE` environment variable
+3. `default`
+
+```bash
+# Create/authenticate a profile. `login` is the profile-creating command;
+# combine the global --profile flag with --server:
+lab --profile prod    login --server https://prod.example.com:8338
+lab --profile staging login --server https://staging.example.com:8338
+
+# Run against two servers in parallel — fully isolated, safe:
+LAB_PROFILE=prod    lab job list &
+LAB_PROFILE=staging lab job list &
+
+# One-off override without exporting the env var:
+lab --profile prod job list
+```
+
+Manage profiles:
+
+```bash
+lab profile list           # all profiles; marks the active one + which have saved credentials
+lab profile show [name]    # server/team/user/experiment for a profile (defaults to the active one)
+lab profile delete <name>  # remove a named profile ('default' cannot be deleted)
+```
+
+**For agents:** to run workstreams against different *servers* concurrently, give each its own profile and set `LAB_PROFILE` per process (or pass `--profile` on every call). This is the server/team/credentials analogue of the per-command `-e/--experiment` override — profiles scope which server+team+key you talk to, `-e` scopes which experiment *within* that server (see "Scoping a single command to an experiment"). Profiles do **not** scope the experiment; keep using `-e` for that.
+
 ## Core Workflow
 
 The standard pattern for working with Transformer Lab:
@@ -308,7 +342,7 @@ lab notes append "..." -e exp-a
 
 - **Pass `-e <exp>` on every `task` / `job` / `notes` command** for that workstream. Treat it as mandatory, not optional — a single omitted `-e` leaks into the global default.
 - **Never call `lab experiment set-default` (or `lab config set current_experiment`) mid-flight** while other experiments are in flight — it yanks the default out from under any command that forgot its `-e`.
-- There is **no environment variable** (e.g. `LAB_EXPERIMENT`) and **no per-session config file** to scope this once for a whole session — the override is strictly per-command. Plan for passing `-e` on each call.
+- There is **no environment variable** (e.g. `LAB_EXPERIMENT`) and **no per-session config file** to scope the *experiment* once for a whole session — the `-e` override is strictly per-command. Plan for passing `-e` on each call. (The *server/team/credentials* dimension is different: that **can** be scoped per-process via a profile + `LAB_PROFILE` — see "Profiles: talking to multiple servers in parallel". A profile does not pin the experiment, so even within a profile you still pass `-e` per command.)
 
 For a single, sequential workflow it's fine to set the default once with `set-default` and omit `-e`. The `-e` discipline only becomes load-bearing when more than one experiment is active concurrently.
 
@@ -596,6 +630,13 @@ lab --format json provider info PROVIDER_ID
 # Health check (verifies the CLI can reach the provider's backend)
 lab --format json provider check PROVIDER_ID
 
+# Show the GPUs available on a provider (accepts an id OR a name, like `delete`).
+# Reports live availability where the backend can report it (Slurm, SkyPilot,
+# RunPod, Lambda, Vast, Local), otherwise the provider's catalog of launchable
+# GPU types (AWS, GCP, Azure, Nebius). Each row is a GPU name + count; an empty
+# result ("No GPU information available") is normal for dstack and CPU-only hosts.
+lab --format json provider gpus PROVIDER_ID_OR_NAME
+
 # Toggle availability without deleting
 lab provider enable PROVIDER_ID
 lab provider disable PROVIDER_ID
@@ -952,6 +993,9 @@ This applies to launching jobs, fetching logs, checking cluster status, and ever
 | `lab logout` | Remove stored API key | No |
 | `lab whoami` | Show current user and team | No |
 | `lab version` | Show CLI version | No |
+| `lab profile list` | List CLI profiles (server+team+credentials sets); marks the active one and which have credentials | No |
+| `lab profile show [name]` | Show a profile's server/team/user/experiment (defaults to the active profile) | No |
+| `lab profile delete <name>` | Delete a named profile (`'default'` cannot be deleted; `--yes` to skip prompt) | No |
 | `lab experiment list` | List all experiments (current default marked with `*`) | No |
 | `lab experiment create <name>` | Create a new experiment (`--set-default` to also switch to it) | No |
 | `lab experiment delete <id>` | Delete an experiment (`--no-interactive` to skip prompt) | No |
@@ -990,6 +1034,7 @@ This applies to launching jobs, fetching logs, checking cluster status, and ever
 | `lab provider update <id>` | Update provider config | No |
 | `lab provider delete <id>` | Delete a provider (`--no-interactive` to skip prompt) | No |
 | `lab provider check <id>` | Check provider health | No |
+| `lab provider gpus <id_or_name>` | Show available GPUs on a provider (live where supported, else catalog) | No |
 | `lab provider verify-lifecycle <id>` | Verify provider lifecycle via a storage probe (`--no-wait` to launch only; see `--help` for polling options) | No |
 | `lab provider enable <id>` | Enable a provider | No |
 | `lab provider disable <id>` | Disable a provider | No |

.agents/skills/transformerlab-cli/references/commands.md

@@ -6,12 +6,14 @@ Full reference for every CLI command. All commands are invoked via:
 lab <command> [subcommand] [options]
 ```
 
-Global option available on all commands:
-- `--format pretty|json` — Output format (default: `pretty`). **Must come before the subcommand.**
+Global options available on all commands (**must come before the subcommand**):
+- `--format pretty|json` — Output format (default: `pretty`).
+- `--profile <name>` — Select the profile (server + team + credentials) for this invocation. Overrides the `LAB_PROFILE` env var; defaults to `default`. See [`profile`](#profile) below.
 
 ```bash
 # Correct
 lab --format json task list
+lab --profile prod job list
 
 # Wrong — flag will be ignored
 lab task list --format json
@@ -61,9 +63,15 @@ Authenticate with the server. Also configures `server`, `user_email`, `team_id`,
 | `--api-key <key>` | API key (prompted if omitted) |
 | `--server <url>` | Server URL (prompted if omitted) |
 
+To log into a **named profile**, combine the global `--profile` flag with `--server` — the credentials and config are written under that profile instead of `default`:
+
+```bash
+lab --profile prod login --server https://prod.example.com:8338
+```
+
 ### `logout`
 
-Remove stored API key.
+Remove the stored API key for the active profile (selected via `--profile` / `LAB_PROFILE`; defaults to `default`).
 
 ### `whoami`
 
@@ -74,6 +82,31 @@ Show current user, team, and server.
 {"email": "user@example.com", "team_id": "...", "team_name": "...", "server": "http://localhost:8338"}
 ```
 
+### `profile`
+
+Manage CLI **profiles** — independent `(server + team + experiment + API key)` sets, so two `lab` commands can talk to two different servers in parallel. The `default` profile lives at the legacy root (`~/.lab/config.json`, `~/.lab/credentials`); named profiles live under `~/.lab/profiles/<name>/`.
+
+Selection is **per-process** (there is no `use` subcommand / stored "current profile"). Precedence: `--profile <name>` (root-level flag, before the subcommand) > `LAB_PROFILE` env var > `default`.
+
+```bash
+lab profile list                 # list all profiles; marks active + which have credentials
+lab profile show [name]          # show a profile's config (defaults to the active profile)
+lab profile delete <name> --yes  # delete a named profile ('default' cannot be deleted)
+
+# Create/authenticate a profile via login (see `login` above):
+lab --profile prod login --server https://prod.example.com:8338
+
+# Run two servers in parallel:
+LAB_PROFILE=prod    lab job list &
+LAB_PROFILE=staging lab job list &
+```
+
+| Subcommand | Description |
+|---|---|
+| `list` | List all profiles. JSON: array of `{name, active, has_credentials}`. |
+| `show [name]` | Show `name`/`server`/`team_id`/`team_name`/`user_email`/`current_experiment`/`has_credentials` for a profile (defaults to the active one). |
+| `delete <name>` | Delete a named profile directory. Refuses `default`. `--yes`/`-y` skips the confirm prompt. |
+
 ---
 
 ## Experiment Commands
@@ -498,6 +531,28 @@ Delete a compute provider.
 
 Check connectivity and health of a provider.
 
+### `provider gpus <provider_id_or_name>`
+
+Show the GPUs available on a provider. The argument accepts either a provider id
+or a name (resolved the same way as `provider delete`). Output is a `GPU | Count`
+table, or `--format json` returns `{provider_id, provider_type, gpus: [{gpu, count}]}`.
+
+Semantics: **live availability where the backend can report it** — Slurm (free
+GPUs per node), SkyPilot (catalog across enabled clouds), RunPod, Lambda
+(regions with capacity), Vast.ai (rentable offers), Local (detected GPUs) —
+otherwise the provider's **catalog of launchable GPU types** (AWS, GCP, Azure,
+Nebius, and the fallback for the live providers). `count` is the available
+quantity for live sources, or the max launchable count per node for catalog
+sources; there is no live-vs-catalog flag in the output. An empty list
+(`No GPU information available`) is expected for dstack (no enumeration endpoint)
+and CPU-only local hosts. The command never errors on backend failures — it
+degrades to the catalog or an empty list.
+
+```bash
+lab provider gpus my-skypilot
+lab --format json provider gpus PROVIDER_ID
+```
+
 ### `provider enable <provider_id>`
 
 Enable a disabled provider.

.agents/skills/transformerlab-cli/references/troubleshooting.md

@@ -53,6 +53,8 @@ lab config set server http://correct-host:8338
 lab status
 ```
 
+If you regularly switch between servers, `lab config set server` (which mutates the shared `default` profile) is the wrong tool — use **profiles** instead so each server keeps its own URL + team + key, and select per command with `--profile <name>` / `LAB_PROFILE`. See "Profiles: talking to multiple servers in parallel" in `SKILL.md`.
+
 ---
 
 ## Authentication Errors (401/403)

api/test/api/test_request_logs.py

@@ -61,6 +61,9 @@ def list_jobs(self, cluster_name):
     def check(self):
         return True
 
+    def show_gpus(self):
+        return []
+
 
 def _make_job_dict(
     job_id: str = "test-1",

api/test/test_remote_workdir_cd.py

@@ -0,0 +1,77 @@
+"""Tests for prepend_remote_workdir_cd.
+
+Regression coverage for the `//main.py` quirk: on remote providers the run command
+executes from the provider's default cwd (e.g. `/` for Lambda cloud-init), not the
+directory where lab.copy_file_mounts() synced the task files. prepend_remote_workdir_cd
+prefixes a `cd` into the recorded workdir so a bare `python main.py` resolves.
+"""
+
+from transformerlab.services.compute_provider.launch_credentials import WORKDIR_FILE_PATH
+from transformerlab.services.compute_provider.launch_template import prepend_remote_workdir_cd
+from transformerlab.shared.models.models import ProviderType
+
+EXPECTED_PREFIX = f'cd "$(cat {WORKDIR_FILE_PATH} 2>/dev/null || echo "$HOME")" && '
+
+
+def test_prefixes_cd_for_remote_provider_with_file_mounts():
+    for provider_type in (
+        ProviderType.LAMBDA.value,
+        ProviderType.RUNPOD.value,
+        ProviderType.SKYPILOT.value,
+        ProviderType.VASTAI.value,
+        ProviderType.DSTACK.value,
+    ):
+        out = prepend_remote_workdir_cd(
+            "python main.py",
+            task_id="task-123",
+            file_mounts=True,
+            provider_type=provider_type,
+        )
+        assert out == EXPECTED_PREFIX + "python main.py", provider_type
+
+
+def test_local_provider_is_not_prefixed():
+    # Local already forces cwd == $HOME == the file-drop dir.
+    out = prepend_remote_workdir_cd(
+        "python main.py",
+        task_id="task-123",
+        file_mounts=True,
+        provider_type=ProviderType.LOCAL.value,
+    )
+    assert out == "python main.py"
+
+
+def test_no_prefix_when_file_mounts_not_enabled():
+    # No synced files (inlined run, or dict-form file_mounts handled natively):
+    # copy_file_mounts isn't injected, so there's no workdir file and nothing to cd into.
+    for file_mounts in (False, None, {"/remote": "/local"}):
+        out = prepend_remote_workdir_cd(
+            "python main.py",
+            task_id="task-123",
+            file_mounts=file_mounts,
+            provider_type=ProviderType.LAMBDA.value,
+        )
+        assert out == "python main.py", file_mounts
+
+
+def test_no_prefix_without_task_id():
+    out = prepend_remote_workdir_cd(
+        "python main.py",
+        task_id=None,
+        file_mounts=True,
+        provider_type=ProviderType.LAMBDA.value,
+    )
+    assert out == "python main.py"
+
+
+def test_preserves_shell_operators_in_command():
+    # The whole command is later shlex.quoted as one payload to tfl-remote-trap, so
+    # operators must be carried through verbatim after the cd prefix.
+    cmd = "python main.py --epochs 3 && echo done"
+    out = prepend_remote_workdir_cd(
+        cmd,
+        task_id="task-123",
+        file_mounts=True,
+        provider_type=ProviderType.RUNPOD.value,
+    )
+    assert out == EXPECTED_PREFIX + cmd